1,182 research outputs found
A tutorial on conformal prediction
Conformal prediction uses past experience to determine precise levels of
confidence in new predictions. Given an error probability , together
with a method that makes a prediction of a label , it produces a
set of labels, typically containing , that also contains with
probability . Conformal prediction can be applied to any method for
producing : a nearest-neighbor method, a support-vector machine, ridge
regression, etc.
Conformal prediction is designed for an on-line setting in which labels are
predicted successively, each one being revealed before the next is predicted.
The most novel and valuable feature of conformal prediction is that if the
successive examples are sampled independently from the same distribution, then
the successive predictions will be right of the time, even though
they are based on an accumulating dataset rather than on independent datasets.
In addition to the model under which successive examples are sampled
independently, other on-line compression models can also use conformal
prediction. The widely used Gaussian linear model is one of these.
This tutorial presents a self-contained account of the theory of conformal
prediction and works through several numerical examples. A more comprehensive
treatment of the topic is provided in "Algorithmic Learning in a Random World",
by Vladimir Vovk, Alex Gammerman, and Glenn Shafer (Springer, 2005).Comment: 58 pages, 9 figure
Discretized conformal prediction for efficient distribution-free inference
In regression problems where there is no known true underlying model,
conformal prediction methods enable prediction intervals to be constructed
without any assumptions on the distribution of the underlying data, except that
the training and test data are assumed to be exchangeable. However, these
methods bear a heavy computational cost-and, to be carried out exactly, the
regression algorithm would need to be fitted infinitely many times. In
practice, the conformal prediction method is run by simply considering only a
finite grid of finely spaced values for the response variable. This paper
develops discretized conformal prediction algorithms that are guaranteed to
cover the target value with the desired probability, and that offer a tradeoff
between computational cost and prediction accuracy
Predictive Inference with Feature Conformal Prediction
Conformal prediction is a distribution-free technique for establishing valid
prediction intervals. Although conventionally people conduct conformal
prediction in the output space, this is not the only possibility. In this
paper, we propose feature conformal prediction, which extends the scope of
conformal prediction to semantic feature spaces by leveraging the inductive
bias of deep representation learning. From a theoretical perspective, we
demonstrate that feature conformal prediction provably outperforms regular
conformal prediction under mild assumptions. Our approach could be combined
with not only vanilla conformal prediction, but also other adaptive conformal
prediction methods. Apart from experiments on existing predictive inference
benchmarks, we also demonstrate the state-of-the-art performance of the
proposed methods on large-scale tasks such as ImageNet classification and
Cityscapes image segmentation.The code is available at
\url{https://github.com/AlvinWen428/FeatureCP}.Comment: Published as a conference paper at ICLR 202
Conformal Prediction: a Unified Review of Theory and New Challenges
In this work we provide a review of basic ideas and novel developments about
Conformal Prediction -- an innovative distribution-free, non-parametric
forecasting method, based on minimal assumptions -- that is able to yield in a
very straightforward way predictions sets that are valid in a statistical sense
also in in the finite sample case. The in-depth discussion provided in the
paper covers the theoretical underpinnings of Conformal Prediction, and then
proceeds to list the more advanced developments and adaptations of the original
idea.Comment: arXiv admin note: text overlap with arXiv:0706.3188,
arXiv:1604.04173, arXiv:1709.06233, arXiv:1203.5422 by other author
Criteria of efficiency for conformal prediction
We study optimal conformity measures for various criteria of efficiency of
classification in an idealised setting. This leads to an important class of
criteria of efficiency that we call probabilistic; it turns out that the most
standard criteria of efficiency used in literature on conformal prediction are
not probabilistic unless the problem of classification is binary. We consider
both unconditional and label-conditional conformal prediction.Comment: 31 page
- …