1,196 research outputs found
Conformal Prediction: a Unified Review of Theory and New Challenges
In this work we provide a review of basic ideas and novel developments about
Conformal Prediction -- an innovative distribution-free, non-parametric
forecasting method, based on minimal assumptions -- that is able to yield in a
very straightforward way predictions sets that are valid in a statistical sense
also in in the finite sample case. The in-depth discussion provided in the
paper covers the theoretical underpinnings of Conformal Prediction, and then
proceeds to list the more advanced developments and adaptations of the original
idea.Comment: arXiv admin note: text overlap with arXiv:0706.3188,
arXiv:1604.04173, arXiv:1709.06233, arXiv:1203.5422 by other author
Predictive Inference with Feature Conformal Prediction
Conformal prediction is a distribution-free technique for establishing valid
prediction intervals. Although conventionally people conduct conformal
prediction in the output space, this is not the only possibility. In this
paper, we propose feature conformal prediction, which extends the scope of
conformal prediction to semantic feature spaces by leveraging the inductive
bias of deep representation learning. From a theoretical perspective, we
demonstrate that feature conformal prediction provably outperforms regular
conformal prediction under mild assumptions. Our approach could be combined
with not only vanilla conformal prediction, but also other adaptive conformal
prediction methods. Apart from experiments on existing predictive inference
benchmarks, we also demonstrate the state-of-the-art performance of the
proposed methods on large-scale tasks such as ImageNet classification and
Cityscapes image segmentation.The code is available at
\url{https://github.com/AlvinWen428/FeatureCP}.Comment: Published as a conference paper at ICLR 202
An Exact and Robust Conformal Inference Method for Counterfactual and Synthetic Controls
We introduce new inference procedures for counterfactual and synthetic
control methods for policy evaluation. We recast the causal inference problem
as a counterfactual prediction and a structural breaks testing problem. This
allows us to exploit insights from conformal prediction and structural breaks
testing to develop permutation inference procedures that accommodate modern
high-dimensional estimators, are valid under weak and easy-to-verify
conditions, and are provably robust against misspecification. Our methods work
in conjunction with many different approaches for predicting counterfactual
mean outcomes in the absence of the policy intervention. Examples include
synthetic controls, difference-in-differences, factor and matrix completion
models, and (fused) time series panel data models. Our approach demonstrates an
excellent small-sample performance in simulations and is taken to a data
application where we re-evaluate the consequences of decriminalizing indoor
prostitution
Post-selection Inference for Conformal Prediction: Trading off Coverage for Precision
Conformal inference has played a pivotal role in providing uncertainty
quantification for black-box ML prediction algorithms with finite sample
guarantees. Traditionally, conformal prediction inference requires a
data-independent specification of miscoverage level. In practical applications,
one might want to update the miscoverage level after computing the prediction
set. For example, in the context of binary classification, the analyst might
start with a prediction sets and see that most prediction sets contain
all outcome classes. Prediction sets with both classes being undesirable, the
analyst might desire to consider, say prediction set. Construction of
prediction sets that guarantee coverage with data-dependent miscoverage level
can be considered as a post-selection inference problem. In this work, we
develop uniform conformal inference with finite sample prediction guarantee
with arbitrary data-dependent miscoverage levels using distribution-free
confidence bands for distribution functions. This allows practitioners to trade
freely coverage probability for the quality of the prediction set by any
criterion of their choice (say size of prediction set) while maintaining the
finite sample guarantees similar to traditional conformal inference
Robust Validation: Confident Predictions Even When Distributions Shift
While the traditional viewpoint in machine learning and statistics assumes
training and testing samples come from the same population, practice belies
this fiction. One strategy---coming from robust statistics and
optimization---is thus to build a model robust to distributional perturbations.
In this paper, we take a different approach to describe procedures for robust
predictive inference, where a model provides uncertainty estimates on its
predictions rather than point predictions. We present a method that produces
prediction sets (almost exactly) giving the right coverage level for any test
distribution in an -divergence ball around the training population. The
method, based on conformal inference, achieves (nearly) valid coverage in
finite samples, under only the condition that the training data be
exchangeable. An essential component of our methodology is to estimate the
amount of expected future data shift and build robustness to it; we develop
estimators and prove their consistency for protection and validity of
uncertainty estimates under shifts. By experimenting on several large-scale
benchmark datasets, including Recht et al.'s CIFAR-v4 and ImageNet-V2 datasets,
we provide complementary empirical results that highlight the importance of
robust predictive validity.Comment: 35 pages, 6 figure
Root-finding Approaches for Computing Conformal Prediction Set
Conformal prediction constructs a confidence set for an unobserved response
of a feature vector based on previous identically distributed and exchangeable
observations of responses and features. It has a coverage guarantee at any
nominal level without additional assumptions on their distribution. Its
computation deplorably requires a refitting procedure for all replacement
candidates of the target response. In regression settings, this corresponds to
an infinite number of model fit. Apart from relatively simple estimators that
can be written as pieces of linear function of the response, efficiently
computing such sets is difficult and is still considered as an open problem. We
exploit the fact that, \emph{often}, conformal prediction sets are intervals
whose boundaries can be efficiently approximated by classical root-finding
algorithm. We investigate how this approach can overcome many limitations of
formerly used strategies and we discuss its complexity and drawbacks
Conformalized matrix completion
Matrix completion aims to estimate missing entries in a data matrix, using
the assumption of a low-complexity structure (e.g., low rank) so that
imputation is possible. While many effective estimation algorithms exist in the
literature, uncertainty quantification for this problem has proved to be
challenging, and existing methods are extremely sensitive to model
misspecification. In this work, we propose a distribution-free method for
predictive inference in the matrix completion problem. Our method adapts the
framework of conformal prediction, which provides confidence intervals with
guaranteed distribution-free validity in the setting of regression, to the
problem of matrix completion. Our resulting method, conformalized matrix
completion (cmc), offers provable predictive coverage regardless of the
accuracy of the low-rank model. Empirical results on simulated and real data
demonstrate that cmc is robust to model misspecification while matching the
performance of existing model-based methods when the model is correct.Comment: accepted to 37th Conference on Neural Information Processing Systems
(NeurIPS 2023
- …