Operating characteristics for the design and optimisation of classification systems

In statistical pattern recognition, problems involve distinguishing of various concepts or classes, based on the development of classifiers/discriminators. These exploit discriminatory information existing in measurements originating from objects. A trained classifier results in a partitioning in measurement space, providing some separation between the various classes. In the (typical) case of class overlap, this partitioning inherently results in a trade-off between the various possible classification errors that may occur. This partitioning can be modified to adjust these trade-offs. Given class abundances, a classifier can be evaluated at a given partitioning. However, variations in the abundances leads to an altered classifier performance. These fundamental aspects behind classifier design and evaluation can be studied within the framework of classifier operating characteristics, which is the topic of this dissertation. The contents consist of a number of published/accepted journal and conference papers, contextualised into a number of chapters representing various aspects of operating characteristic analysis. First the well-known two-class operating characteristic is considered, with two new analyses that are useful in certain circumstances. Next, the extension to the elusive multiclass case is considered, showing how standard 2-class operating characteristics analyses can be extended theoretically to the multiclass case. The challenge behind the multiclass extension is shown to be of a computational nature, with the calculation size increasing exponentially with the number of classes. The primary thesis contribution is then presented, consisting of a number of approaches and philosophies that can be used to overcome the computational challenges. Of primary importance is the finding that most practical problems are such that not all dimensions of the operating characteristic interact together significantly. Next it is shown how the operating characteristic approach can be used to design classifiers in ill-defined environments. In these problems some classes may be poorly represented, and the goal of the classifier design is to protect against these unforeseen conditions. Finally, it is shown that operating characteristics can be applied to a multi-stage classifier setup, allowing for a holistic design incorporating interactions between classes, and the classifier stages.Electrical Engineering, Mathematics and Computer Scienc

Efficient Multiclass ROC Approximation by Decomposition via Confusion Matrix Perturbation Analysis

Receiver operator characteristic (ROC) analysis has become a standard tool in the design and evaluation of two-class classification problems. It allows for an analysis that incorporates all possible priors, costs, and operating points, which is important in many real problems, where conditions are often nonideal. Extending this to the multiclass case is attractive, conferring the benefits of ROC analysis to a multitude of new problems. Even though theROCanalysis extends theoretically to the multiclass case, the exponential computational complexity as a function of the number of classes is restrictive. In this paper, we show that the multiclass ROC can often be simplified considerably because some ROC dimensions are independent of each other. We present an algorithm that analyzes interactions between various ROC dimensions, identifying independent classes, and groups of interacting classes, allowing the ROC to be decomposed. The resulting decomposed ROC hypersurface can be interrogated in a similar fashion to the ideal case, allowing for approaches such as cost sensitive and Neyman-Pearson optimization, as well as the volume under the ROC. An extensive bouquet of examples and experiments demonstrates the potential of this methodology.MediamaticsElectrical Engineering, Mathematics and Computer Scienc

Testing absolute plate reference frames and the implications for the generation of geodynamic mantle heterogeneity structure

International audienceAbsolute reference frames are a means of describing the motion of plates on the surface of the Earth over time, relative to a fixed point or “frame.” Multiple models of absolute plate motion have been proposed for the Cretaceous–Tertiary period, however, estimating the robustness and limitations of each model remains a significant limitation for refining both regional and global models of plate motion as well as fully integrated and time dependent geodynamic models. Here, we use a novel approach to compare five models of absolute plate motion in terms of their consequences for forward modelled deep mantle structure since at least 140 Ma. We show that the use of hotspots, either fixed or moving, or palaeomagnetics, with or without corrections for true-polar wander, leads to significant differences in palaeo-plate velocities of over 10 cm/yr as well as differences in the location of palaeo-plate boundaries of up to 30° in longitude and latitude. Furthermore, we suggest that first order differences in forward predicted mantle structure between the models are due mostly to differences in palaeo-plate velocities, whereas variation in the location of plate boundaries may contribute to smaller wavelength offsets. We present a global comparison of the absolute reference frames in terms of mantle structure, which we have tomographically filtered to reflect the resolution of the seismic tomography model S20RTS. At very long wavelengths hotspot models best reproduce the mantle structure. However, when geometry and the match of smaller-scale subducted slab volumes are compared, a hybrid model based on moving hotspots after 100 Ma and palaeomagnetic data before (with no corrections for true-polar wander), best reproduces the overall mantle structure of slab burial grounds, even though no single model fits best at all mantle depths. We find also that the published subduction reference frame tested here results in a modelled mantle structure that agrees well with S20RTS for depths > 2500 km, equivalent to subduction before the Cretaceous, but not for shallower depths. This indicates that a careful assimilation of hotspot, palaeomagnetic and seismic tomography data into future absolute plate motion models is required to derive a more robust subduction reference frame