375 research outputs found
Inductive conformal predictors in the batch mode
Conformal predictors are set predictors that are automatically valid in the sense of having coverage probability equal to or exceeding a given confidence level. Inductive conformal predictors are a computationally efficient version of conformal predictors satisfying the same property of validity. However, inductive conformal predictors have been only known to control unconditional coverage probability. This paper explores various versions of conditional validity and various ways to achieve them using inductive conformal predictors and their modifications
Conformal Prediction: a Unified Review of Theory and New Challenges
In this work we provide a review of basic ideas and novel developments about
Conformal Prediction -- an innovative distribution-free, non-parametric
forecasting method, based on minimal assumptions -- that is able to yield in a
very straightforward way predictions sets that are valid in a statistical sense
also in in the finite sample case. The in-depth discussion provided in the
paper covers the theoretical underpinnings of Conformal Prediction, and then
proceeds to list the more advanced developments and adaptations of the original
idea.Comment: arXiv admin note: text overlap with arXiv:0706.3188,
arXiv:1604.04173, arXiv:1709.06233, arXiv:1203.5422 by other author
Hedging predictions in machine learning
Recent advances in machine learning make it possible to design efficient
prediction algorithms for data sets with huge numbers of parameters. This paper
describes a new technique for "hedging" the predictions output by many such
algorithms, including support vector machines, kernel ridge regression, kernel
nearest neighbours, and by many other state-of-the-art methods. The hedged
predictions for the labels of new objects include quantitative measures of
their own accuracy and reliability. These measures are provably valid under the
assumption of randomness, traditional in machine learning: the objects and
their labels are assumed to be generated independently from the same
probability distribution. In particular, it becomes possible to control (up to
statistical fluctuations) the number of erroneous predictions by selecting a
suitable confidence level. Validity being achieved automatically, the remaining
goal of hedged prediction is efficiency: taking full account of the new
objects' features and other available information to produce as accurate
predictions as possible. This can be done successfully using the powerful
machinery of modern machine learning.Comment: 24 pages; 9 figures; 2 tables; a version of this paper (with
discussion and rejoinder) is to appear in "The Computer Journal
From conformal to probabilistic prediction
This paper proposes a new method of probabilistic prediction, which is based
on conformal prediction. The method is applied to the standard USPS data set
and gives encouraging results.Comment: 12 pages, 2 table
Detecting adversarial manipulation using inductive Venn-ABERS predictors
Inductive Venn-ABERS predictors (IVAPs) are a type of probabilistic predictors with the theoretical guarantee that their predictions are perfectly calibrated. In this paper, we propose to exploit this calibration property for the detection of adversarial examples in binary classification tasks. By rejecting predictions if the uncertainty of the IVAP is too high, we obtain an algorithm that is both accurate on the original test set and resistant to adversarial examples. This robustness is observed on adversarials for the underlying model as well as adversarials that were generated by taking the IVAP into account. The method appears to offer competitive robustness compared to the state-of-the-art in adversarial defense yet it is computationally much more tractable
On-line predictive linear regression
We consider the on-line predictive version of the standard problem of linear
regression; the goal is to predict each consecutive response given the
corresponding explanatory variables and all the previous observations. We are
mainly interested in prediction intervals rather than point predictions. The
standard treatment of prediction intervals in linear regression analysis has
two drawbacks: (1) the classical prediction intervals guarantee that the
probability of error is equal to the nominal significance level epsilon, but
this property per se does not imply that the long-run frequency of error is
close to epsilon; (2) it is not suitable for prediction of complex systems as
it assumes that the number of observations exceeds the number of parameters. We
state a general result showing that in the on-line protocol the frequency of
error for the classical prediction intervals does equal the nominal
significance level, up to statistical fluctuations. We also describe
alternative regression models in which informative prediction intervals can be
found before the number of observations exceeds the number of parameters. One
of these models, which only assumes that the observations are independent and
identically distributed, is popular in machine learning but greatly underused
in the statistical theory of regression.Comment: 34 pages; 6 figures; 1 table. arXiv admin note: substantial text
overlap with arXiv:0906.312
- …