Evaluating reliability of individual classifications with local methods

In this thesis we take upon different approaches for estimating reliability of individual classification predictions made by classifiers based on supervised learning. The general definition of the term reliability is the ability to perform required functions under stated conditions. In machine learning, we refer to accuracy, as in the ability to provide accurate predictions. We face the problem that measures of reliability are not quantitatively defined. We can therefore only conceive estimates. Reliability estimates of individual predictions provide valuable information that could be beneficial in individual predictions assessment of use. For the needs of our thesis we develop several methods for reliability estimation based on existing approaches of local methods and the variance of a bagged model. We test our methods on various available real-life and artificial datasets and compare our methods with those based on inverse transduction. Methods were tested on 20 different datasets on 7 classification models and the estimates were calculated using 11 measures of similarity. We applied three statistical methods to our results. We came to a conclusion that these tests do not give clear results, as Q-Q plots only vaguely support calculated correlation. Correlation tests show potential of estimates LCV and BAGV as they demonstrated best on average performance. Second-comers with good result were estimates TRANS1 and CNK, while other estimates failed to excel

Reliability estimation of individual predictions in supervised learning

The thesis discuses reliability estimation of individual predictions in the supervised learning framework. This research field is rather new and currently attracts little attention from experts and users alike. The main indicators of success of machine learning models are consequently still measures of average performance, for example classification accuracy or root mean square error. However, averaged statistics are not able to provide a full view of models' performance and an increasing number of today's users of machine learning have interest for additional information that can help them to better understand the models' results. This additional information is even more important in cases where wrong predictions may lead to serious financial losses or medical complications. It happens that experts become reluctant to use prediction systems if their predictions are not backed up by their reliability assessments. The prevalent methods for model assessment are insufficient among fields where decision support is of crucial importance or where the average performance is not of paramount importance, information on the reliability of single predictions may prove very beneficial. The greatest concern with use of machine learning algorithms is whether the chosen model represents the data well and if the predicted values conform to the dataset or has the model learned a wrong concept or even over-fitted to noise in the data. As we want to take into account all possible machine learning models, we have to deal with them as with black-boxes, which means we only have access to their input (the training examples) and their output (their predictions). This work presents a complete overview of reliability estimators for supervised learning. This framework consists of classification and regression, due to their inherent differences. We also distinguish between point-wise and interval estimators, but interestingly, point-wise estimators can be applied both to classification and regression, whereas interval estimators are defined only for regression. The first contribution of this thesis is a new comparative study of the usefulness of point-wise estimators in the classification setting. The analysis and comparison with a reference function shows that this kind of reliability estimation is rarely useful on real-world datasets. But in cases when we have to deal with a suboptimal model and the point-wise estimators conform with the data, they can prove to improve the results and provide additional information. Regarding interval estimation, the thesis contributes a novel, unifying view of reliability estimation enabling their comparison, which was not possible before. Our analysis shows the dual nature of the two families of approaches: methods based on bootstrap and maximum likelihood estimation provide valid prediction intervals and methods based on local neighborhoods provide optimal prediction intervals. Based on this finding, we present a combined approach that merges the properties of the two groups. Results of this method are favorable, indicating that the combined prediction intervals are more robust. Existing statistics that provide information merely on the models' average accuracy are not truly informative, while on the other hand, appropriate graphic visualizations are known to be very useful for developing users' intuition and understanding of the models behavior. After demonstrating an existing visualization tool for comparing prediction intervals we present a new visualization technique that enables model comparison and has potential for knowledge discovery. The final contribution is a model aggregation procedure based on a combined statistic for robust selection and merging of regression predictions. This new evaluation statistic and aggregation procedure provides confirmatory and consequently more reliable predictions

Modifying the brake drum geometry to avoid selfexcited vibrations and noise

A squealing noise of 50 dB was measured on a vehicle homologation test at around 900 Hz on the existing brake drum design, mounted on the rear axle of the mid-sized passenger automobile. Therefore, analysis of eigenfrequencies of the original drum design was performed using the impact hammer test and numerical analysis. It was established that a critical mode shape 0/2 exists at around 900 Hz, exactly where the squeal noise was recorded at the brake road noise evaluation vehicle test. The analysis was carried out with the intention to eliminate the possibility of the squealing noise by increasing the critical mode above 900 Hz. The relation between different brake drum modifications parameters and the eigenfrequencies was determined and the best solution was obtained. The first eigenfrequency of the proposed drum design was increased by 58 Hz and the difference between the in-plane and out-of plane mode shape was sufficient. We can conclude that the modified drum design will not have squeal issues at 900 Hz as there are no eigenfrequencies of the brake drum in that range and therefore the problem of the loud brake is solved