824 research outputs found
Conformal Prediction: a Unified Review of Theory and New Challenges
In this work we provide a review of basic ideas and novel developments about
Conformal Prediction -- an innovative distribution-free, non-parametric
forecasting method, based on minimal assumptions -- that is able to yield in a
very straightforward way predictions sets that are valid in a statistical sense
also in in the finite sample case. The in-depth discussion provided in the
paper covers the theoretical underpinnings of Conformal Prediction, and then
proceeds to list the more advanced developments and adaptations of the original
idea.Comment: arXiv admin note: text overlap with arXiv:0706.3188,
arXiv:1604.04173, arXiv:1709.06233, arXiv:1203.5422 by other author
Criteria of efficiency for conformal prediction
We study optimal conformity measures for various criteria of efficiency of
classification in an idealised setting. This leads to an important class of
criteria of efficiency that we call probabilistic; it turns out that the most
standard criteria of efficiency used in literature on conformal prediction are
not probabilistic unless the problem of classification is binary. We consider
both unconditional and label-conditional conformal prediction.Comment: 31 page
TorchCP: A Library for Conformal Prediction based on PyTorch
TorchCP is a Python toolbox for conformal prediction research on deep
learning models. It contains various implementations for posthoc and training
methods for classification and regression tasks (including multi-dimension
output). TorchCP is built on PyTorch (Paszke et al., 2019) and leverages the
advantages of matrix computation to provide concise and efficient inference
implementations. The code is licensed under the LGPL license and is
open-sourced at
Using random forest for reliable classification and cost-sensitive learning for medical diagnosis
Background: Most machine-learning classifiers output label predictions for new instances without indicating how reliable the predictions are. The applicability of these classifiers is limited in critical domains where incorrect predictions have serious consequences, like medical diagnosis. Further, the default assumption of equal misclassification costs is most likely violated in medical diagnosis. Results: In this paper, we present a modified random forest classifier which is incorporated into the conformal predictor scheme. A conformal predictor is a transductive learning scheme, using Kolmogorov complexity to test the randomness of a particular sample with respect to the training sets. Our method show well-calibrated property that the performance can be set prior to classification and the accurate rate is exactly equal to the predefined confidence level. Further, to address the cost sensitive problem, we extend our method to a label-conditional predictor which takes into account different costs for misclassifications in different class and allows different confidence level to be specified for each class. Intensive experiments on benchmark datasets and real world applications show the resultant classifier is well-calibrated and able to control the specific risk of different class. Conclusion: The method of using RF outlier measure to design a nonconformity measure benefits the resultant predictor. Further, a label-conditional classifier is developed and turn to be an alternative approach to the cost sensitive learning problem that relies on label-wise predefined confidence level. The target of minimizing the risk of misclassification is achieved by specifying the different confidence level for different class
Group-Conditional Conformal Prediction via Quantile Regression Calibration for Crop and Weed Classification
As deep learning predictive models become an integral part of a large
spectrum of precision agricultural systems, a barrier to the adoption of such
automated solutions is the lack of user trust in these highly complex, opaque
and uncertain models. Indeed, deep neural networks are not equipped with any
explicit guarantees that can be used to certify the system's performance,
especially in highly varying uncontrolled environments such as the ones
typically faced in computer vision for agriculture.Fortunately, certain methods
developed in other communities can prove to be important for agricultural
applications. This article presents the conformal prediction framework that
provides valid statistical guarantees on the predictive performance of any
black box prediction machine, with almost no assumptions, applied to the
problem of deep visual classification of weeds and crops in real-world
conditions. The framework is exposed with a focus on its practical aspects and
special attention accorded to the Adaptive Prediction Sets (APS) approach that
delivers marginal guarantees on the model's coverage. Marginal results are then
shown to be insufficient to guarantee performance on all groups of individuals
in the population as characterized by their environmental and pedo-climatic
auxiliary data gathered during image acquisition.To tackle this shortcoming,
group-conditional conformal approaches are presented: the ''classical'' method
that consists of iteratively applying the APS procedure on all groups, and a
proposed elegant reformulation and implementation of the procedure using
quantile regression on group membership indicators. Empirical results showing
the validity of the proposed approach are presented and compared to the
marginal APS then discussed
Detecting adversarial manipulation using inductive Venn-ABERS predictors
Inductive Venn-ABERS predictors (IVAPs) are a type of probabilistic predictors with the theoretical guarantee that their predictions are perfectly calibrated. In this paper, we propose to exploit this calibration property for the detection of adversarial examples in binary classification tasks. By rejecting predictions if the uncertainty of the IVAP is too high, we obtain an algorithm that is both accurate on the original test set and resistant to adversarial examples. This robustness is observed on adversarials for the underlying model as well as adversarials that were generated by taking the IVAP into account. The method appears to offer competitive robustness compared to the state-of-the-art in adversarial defense yet it is computationally much more tractable
- …