ROC Optimisation of Safety Related Systems

1st Workshop on ROC Analysis in Artificial Intelligence (ROCAI 2004), part of the 16th European Conference on Artificial Intelligence, Valencia, Spain, 22-27 August 2004Many safety related and critical systems warn of potentially dangerous events; for example the Short Term Conflict Alert (STCA) system warns of airspace infractions between aircraft. Although installed with current technology such critical systems may become out of date due to changes in the circumstances in which they function, operational procedures and the regulatory environment. Current practice is to ‘tune’ by hand the many parameters governing the system in order to optimise the operating point in terms of the true positive and false positive rates, which are frequently associated with highly imbalanced costs. In this paper we cast the tuning of critical systems as a multiobjective optimisation problem. We show how a region of the optimal receiver operating characteristic (ROC) curve may be obtained, permitting the system operators to select the operating point. We apply this methodology to the STCA system, using a multi-objective (1 + 1)-evolution strategy, showing that we can improve upon the current hand-tuned operating point as well as providing the salient ROC curve describing the true-positive versus false-positive tradeoff

Multi-objective optimisation in the presence of uncertainty

2005 IEEE Congress on Evolutionary Computation, Edinburgh, Scotland, 2-5 September 2005The codebase for this paper is available at https://github.com/fieldsend/ieee_cec_2005_bayes_uncertainThere has been only limited discussion on the effect of uncertainty and noise in multi-objective optimisation problems and how to deal with it. Here we address this problem by assessing the probability of dominance and maintaining an archive of solutions which are, with some known probability, mutually non-dominating.We examine methods for estimating the probability of dominance. These depend crucially on estimating the effective noise variance and we introduce a novel method of learning the variance during optimisation.Probabilistic domination contours are presented as a method for conveying the confidence that may be placed in objectives that are optimised in the presence of uncertainty

The Rolling Tide Evolutionary Algorithm: A Multi-Objective Optimiser for Noisy Optimisation Problems

As the methods for evolutionary multiobjective optimization (EMO) mature and are applied to a greater number of real-world problems, there has been gathering interest in the effect of uncertainty and noise on multiobjective optimization, specifically how algorithms are affected by it, how to mitigate its effects, and whether some optimizers are better suited to dealing with it than others. Here we address the problem of uncertain evaluation, in which the uncertainty can be modeled as an additive noise in objective space. We develop a novel algorithm, the rolling tide evolutionary algorithm (RTEA), which progressively improves the accuracy of its estimated Pareto set, while simultaneously driving the front toward the true Pareto front. It can cope with noise whose characteristics change as a function of location (both design and objective), or which alter during the course of an optimization. Four state-of-the-art noise-tolerant EMO algorithms, as well as four widely used standard EMO algorithms, are compared to RTEA on 70 instances of ten continuous space test problems from the CEC'09 multiobjective optimization test suite. Different instances of these problems are generated by modifying them to exhibit different types and intensities of noise. RTEA seems to provide competitive performance across both the range of test problems used and noise types

Multi-objective optimisation of safety related systems: An application to Short Term Conflict Alert.

Copyright © 2006 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other users, including reprinting/ republishing this material for advertising or promotional purposes, creating new collective works for resale or redistribution to servers or lists, or reuse of any copyrighted components of this work in other works.Notes: In this paper multi-objective optimisation is used for the first time to adjust the 1500 parameters of Short-Term Conflict Alert systems to optimise the Receiver Operating Characteristic (ROC) by simultaneously reducing the false positive rate and increasing the true positive alert rate, something that previous work by other researchers had not succeeded in doing. Importantly for such safety-critical systems, the method also yields an assessment of the confidence that may be placed in the optimised ROC curves. The paper results from a collaboration with NATS and a current KTP project, also with NATS, is deploying the methods in air-traffic control centres nationwide.Many safety related and critical systems warn of potentially dangerous events; for example, the short term conflict alert (STCA) system warns of airspace infractions between aircraft. Although installed with current technology, such critical systems may become out of date due to changes in the circumstances in which they function, operational procedures, and the regulatory environment. Current practice is to "tune," by hand, the many parameters governing the system in order to optimize the operating point in terms of the true positive and false positive rates, which are frequently associated with highly imbalanced costs. We cast the tuning of critical systems as a multiobjective optimization problem. We show how a region of the optimal receiver operating characteristic (ROC) curve may be obtained, permitting the system operators to select the operating point. We apply this methodology to the STCA system, using a multiobjective (1+1) evolution strategy, showing that we can improve upon the current hand-tuned operating point, as well as providing the salient ROC curve describing the true positive versus false positive tradeoff. We also provide results for three-objective optimization of the alert response time in addition to the true and false positive rates. Additionally, we illustrate the use of bootstrapping for representing evaluation uncertainty on estimated Pareto fronts, where the evaluation of a system is based upon a finite set of representative data

On the efficient maintenance and updating of Pareto solutions when assigned objectives values may change

Copyright © 2013 University of ExeterNOTE: Version 1.1 is the more recent version (having some additional order of complexity analysis).Usually when undertaking a multi-objective optimisation problem it is assumed that on evaluation of a design, the assigned objectives are fixed. This allows, for instance, domination comparisons to be undertaken just once to decide on whether a design should be categorised as an elite (non-dominated) solution, with the designation only removed if a new location is found to be better at a later time step. However, there are some situations where the objective vector assigned to a design may change at a later time point. This may be due to some global change in the environment (a dynamic problem), which effects all designs proposed thus far, however it can also be a change of objectives of a single solution in isolation. This may be due to for instance due to resampling in a noisy domain updating an estimate of objective values, or via an increased resolution of the objective evaluation (e.g. a finer mesh on a finite element analysis of a design, or different data instances for a classifier being tuned). How to efficiently maintain an elite archive when the assigned objectives are susceptible to change has not been confronted until this point. Although a number of data structures exist for efficiently maintaining and querying solutions, they assume that a domination relationship between two designs at time t, will persist at all future time steps. When this no longer holds, it is necessary to track dominated as well as non-dominated solutions as the search progresses, if we want to guarantee access to the best estimate of the non-dominated subset of solutions visited by an optimiser at any time step. Here we discuss different storage and query approaches which guarantee this, and propose a novel data maintenance regime based on chaining single domination links between all solutions evaluated at any time point to rapidly discern the non-dominated subset as solutions are reevaluated, and new design locations proposed. We detail the computational complexity of this approach, and compare the empirical performance of three different link selection protocols on simulated behaviour of set updates, mimicking a converging optimiser and a optimiser performing a random search, and where locations are resampled at random, or resampling based on their estimated non-dominance.Computer Science, University of Exete

Formulation and comparison of multi-class ROC surfaces

2nd ROCML workshop, held within the 22nd International Conference on Machine Learning (ICML 2005), Bonn, Germany, 7-11 August 2005The Receiver Operating Characteristic (ROC) has become a standard tool for the analysis and comparison of classifiers when the costs of misclassification are unknown. There has been relatively little work, however, examining ROC for more than two classes. Here we define the ROC surface for the Q-class problem in terms of a multi-objective optimisation problem in which the goal is to simultaneously minimise the Q(Q − 1) mis-classification rates, when the misclassification costs and parameters governing the classifier’s behaviour are unknown. We present an evolutionary algorithm to locate the optimal trade-off surface between misclassifications of different types. The performance of the evolutionary algorithm is illustrated on a synthetic three class problem. In addition the use of the Pareto optimal surface to compare classifiers is discussed, and we present a straightforward multi-class analogue of the Gini coefficient. This is illustrated on synthetic and standard machine learning dat

Multi-class ROC analysis from a multi-objective optimisation perspective

Copyright © 2006 Elsevier. NOTICE: this is the author’s version of a work that was accepted for publication in Pattern Recognition Letters . Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. A definitive version was subsequently published in Pattern Recognition Letters, Vol. 27 Issue 8 (2006), DOI: 10.1016/j.patrec.2005.10.016Notes: Receiver operating characteristics (ROC) are traditionally used for assessing and tuning classifiers discriminating between two classes. This paper is the first to set ROC analysis in a multi-objective optimisation framework and thus generalise ROC curves to any number of classes, showing how multi-objective optimisation may be used to optimise classifier performance. An important new result is that the appropriate measure for assessing overall classifier quality is the Gini coefficient, rather than the volume under the ROC surface as previously thought. The method is currently being exploited in a KTP project with AI Corporation on detecting credit card fraud.The receiver operating characteristic (ROC) has become a standard tool for the analysis and comparison of classifiers when the costs of misclassification are unknown. There has been relatively little work, however, examining ROC for more than two classes. Here we discuss and present an extension to the standard two-class ROC for multi-class problems. We define the ROC surface for the Q-class problem in terms of a multi-objective optimisation problem in which the goal is to simultaneously minimise the Q(Q − 1) misclassification rates, when the misclassification costs and parameters governing the classifier’s behaviour are unknown. We present an evolutionary algorithm to locate the Pareto front—the optimal trade-off surface between misclassifications of different types. The use of the Pareto optimal surface to compare classifiers is discussed and we present a straightforward multi-class analogue of the Gini coefficient. The performance of the evolutionary algorithm is illustrated on a synthetic three class problem, for both k-nearest neighbour and multi-layer perceptron classifiers

Visualisation of multi-class ROC surfaces

2nd ROCML workshop, held within the 22nd International Conference on Machine Learning (ICML 2005), Bonn, Germany, 7-11 August 2005The Receiver Operating Characteristic (ROC) has become a standard tool for the analysis and comparison of binary classifiers when the costs of misclassification are un- known. Although there has been relatively little work in examining ROC for more than two classes – there has been growing interest in the area, and in recent studies we have formulated it in terms of misclassification rates. Although techniques exist for the numerical comparison of the fronts generated by these new methods, the useful visualisation of these fronts to aid the selection of a final operating point are still very much in their infancy. Methods exist for the visualisation of similar surfaces, Pareto fronts, which we discuss, however the particular properties of the ROC front that the practitioner is interested in may also direct us to new and more suitable visualisation methods. This paper briefly outlines what is currently in use, and what avenues may be of interest to examine in the future

On the efficient use of uncertainty when performing expensive ROC optimisation.

Copyright © 2008 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other users, including reprinting/ republishing this material for advertising or promotional purposes, creating new collective works for resale or redistribution to servers or lists, or reuse of any copyrighted components of this work in other works.IEEE Congress on Evolutionary Computation 2008 (CEC 2008). (IEEE World Congress on Computational Intelligence), Hong Kong, 1-6 June 2008When optimising receiver operating characteristic (ROC) curves there is an inherent degree of uncertainty associated with the operating point evaluation of a model parameterisation x. This is due to the finite amount of training data used to evaluate the true and false positive rates of x. The uncertainty associated with any particular x can be reduced, but only at the computation cost of evaluating more data. Here we explicitly represent this uncertainty through the use of probabilistically non-dominated archives, and show how expensive ROC optimisation problems may be tackled by only evaluating a small subset of the available data at each generation of an optimisation algorithm. Illustrative results are given on data sets from the well known UCI machine learning repository

Variable interactions and exploring parameter space in an expensive optimisation problem: Optimising Short Term Conflict Alert

Copyright © 2010 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other users, including reprinting/ republishing this material for advertising or promotional purposes, creating new collective works for resale or redistribution to servers or lists, or reuse of any copyrighted components of this work in other works.2010 IEEE Congress on Evolutionary Computation (CEC), Barcelona, Spain, 18-23 July 2010Short Term Conflict Alert (STCA) systems provide warnings to air traffic controllers if aircraft are in danger of becoming too close. They are complex software programs, with many inter-dependent parameters that must be adjusted to achieve the best trade-off between wanted and nuisance alerts. We describe a multi-archive evolutionary algorithm for optimising regional parameter subsets in parallel, reducing the number of evaluations required to generate an estimated Pareto optimal Receiver Operating Characteristic (ROC), showing that it provides superior results to traditional single-archived algorithms. A method of `aggressive' optimisation, designed to explore unknown parameter ranges in a `safe' manner, is shown to yield more extensive and better converged estimated Pareto fronts