494 research outputs found
Conformal Drug Property Prediction with Density Estimation under Covariate Shift
In drug discovery, it is vital to confirm the predictions of pharmaceutical
properties from computational models using costly wet-lab experiments. Hence,
obtaining reliable uncertainty estimates is crucial for prioritizing drug
molecules for subsequent experimental validation. Conformal Prediction (CP) is
a promising tool for creating such prediction sets for molecular properties
with a coverage guarantee. However, the exchangeability assumption of CP is
often challenged with covariate shift in drug discovery tasks: Most datasets
contain limited labeled data, which may not be representative of the vast
chemical space from which molecules are drawn. To address this limitation, we
propose a method called CoDrug that employs an energy-based model leveraging
both training data and unlabelled data, and Kernel Density Estimation (KDE) to
assess the densities of a molecule set. The estimated densities are then used
to weigh the molecule samples while building prediction sets and rectifying for
distribution shift. In extensive experiments involving realistic distribution
drifts in various small-molecule drug discovery tasks, we demonstrate the
ability of CoDrug to provide valid prediction sets and its utility in
addressing the distribution shift arising from de novo drug design models. On
average, using CoDrug can reduce the coverage gap by over 35% when compared to
conformal prediction sets not adjusted for covariate shift.Comment: Accepted at NeurIPS 202
Bayesian Optimization with Conformal Prediction Sets
Bayesian optimization is a coherent, ubiquitous approach to decision-making
under uncertainty, with applications including multi-arm bandits, active
learning, and black-box optimization. Bayesian optimization selects decisions
(i.e. objective function queries) with maximal expected utility with respect to
the posterior distribution of a Bayesian model, which quantifies reducible,
epistemic uncertainty about query outcomes. In practice, subjectively
implausible outcomes can occur regularly for two reasons: 1) model
misspecification and 2) covariate shift. Conformal prediction is an uncertainty
quantification method with coverage guarantees even for misspecified models and
a simple mechanism to correct for covariate shift. We propose conformal
Bayesian optimization, which directs queries towards regions of search space
where the model predictions have guaranteed validity, and investigate its
behavior on a suite of black-box optimization tasks and tabular ranking tasks.
In many cases we find that query coverage can be significantly improved without
harming sample-efficiency.Comment: For code, see
https://www.github.com/samuelstanton/conformal-bayesopt.gi
Not all distributional shifts are equal: Fine-grained robust conformal inference
We introduce a fine-grained framework for uncertainty quantification of
predictive models under distributional shifts. This framework distinguishes the
shift in covariate distributions from that in the conditional relationship
between the outcome () and the covariates (). We propose to reweight the
training samples to adjust for an identifiable covariate shift while protecting
against worst-case conditional distribution shift bounded in an -divergence
ball. Based on ideas from conformal inference and distributionally robust
learning, we present an algorithm that outputs (approximately) valid and
efficient prediction intervals in the presence of distributional shifts. As a
use case, we apply the framework to sensitivity analysis of individual
treatment effects with hidden confounding. The proposed methods are evaluated
in simulation studies and three real data applications, demonstrating superior
robustness and efficiency compared with existing benchmarks.Comment: 25 pages, 5 figure
Conformal Prediction: a Unified Review of Theory and New Challenges
In this work we provide a review of basic ideas and novel developments about
Conformal Prediction -- an innovative distribution-free, non-parametric
forecasting method, based on minimal assumptions -- that is able to yield in a
very straightforward way predictions sets that are valid in a statistical sense
also in in the finite sample case. The in-depth discussion provided in the
paper covers the theoretical underpinnings of Conformal Prediction, and then
proceeds to list the more advanced developments and adaptations of the original
idea.Comment: arXiv admin note: text overlap with arXiv:0706.3188,
arXiv:1604.04173, arXiv:1709.06233, arXiv:1203.5422 by other author
Sensitivity Analysis of Individual Treatment Effects: A Robust Conformal Inference Approach
We propose a model-free framework for sensitivity analysis of individual
treatment effects (ITEs), building upon ideas from conformal inference. For any
unit, our procedure reports the -value, a number which quantifies the
minimum strength of confounding needed to explain away the evidence for ITE.
Our approach rests on the reliable predictive inference of counterfactuals and
ITEs in situations where the training data is confounded. Under the marginal
sensitivity model of Tan (2006), we characterize the shift between the
distribution of the observations and that of the counterfactuals. We first
develop a general method for predictive inference of test samples from a
shifted distribution; we then leverage this to construct covariate-dependent
prediction sets for counterfactuals. No matter the value of the shift, these
prediction sets (resp. approximately) achieve marginal coverage if the
propensity score is known exactly (resp. estimated). We describe a distinct
procedure also attaining coverage, however, conditional on the training data.
In the latter case, we prove a sharpness result showing that for certain
classes of prediction problems, the prediction intervals cannot possibly be
tightened. We verify the validity and performance of the new methods via
simulation studies and apply them to analyze real datasets
JAWS: Predictive Inference Under Covariate Shift
We propose \textbf{JAWS}, a series of wrapper methods for distribution-free
uncertainty quantification tasks under covariate shift, centered on our core
method \textbf{JAW}, the \textbf{JA}ckknife+ \textbf{W}eighted with
likelihood-ratio weights. JAWS also includes computationally efficient
\textbf{A}pproximations of JAW using higher-order influence functions:
\textbf{JAWA}. Theoretically, we show that JAW relaxes the jackknife+'s
assumption of data exchangeability to achieve the same finite-sample coverage
guarantee even under covariate shift. JAWA further approaches the JAW guarantee
in the limit of either the sample size or the influence function order under
mild assumptions. Moreover, we propose a general approach to repurposing any
distribution-free uncertainty quantification method and its guarantees to the
task of risk assessment: a task that generates the estimated probability that
the true label lies within a user-specified interval. We then propose
\textbf{JAW-R} and \textbf{JAWA-R} as the repurposed versions of proposed
methods for \textbf{R}isk assessment. Practically, JAWS outperform the
state-of-the-art predictive inference baselines in a variety of biased real
world data sets for both interval-generation and risk-assessment auditing
tasks
Group-Weighted Conformal Prediction
Conformal prediction (CP) is a method for constructing a prediction interval
around the output of a fitted model, whose validity does not rely on the model
being correct--the CP interval offers a coverage guarantee that is
distribution-free, but relies on the training data being drawn from the same
distribution as the test data. A recent variant, weighted conformal prediction
(WCP), reweights the method to allow for covariate shift between the training
and test distributions. However, WCP requires knowledge of the nature of the
covariate shift-specifically,the likelihood ratio between the test and training
covariate distributions. In practice, since this likelihood ratio is estimated
rather than known exactly, the coverage guarantee may degrade due to the
estimation error. In this paper, we consider a special scenario where
observations belong to a finite number of groups, and these groups determine
the covariate shift between the training and test distributions-for instance,
this may arise if the training set is collected via stratified sampling. Our
results demonstrate that in this special case, the predictive coverage
guarantees of WCP can be drastically improved beyond the bounds given by
existing estimation error bounds
Test-time Recalibration of Conformal Predictors Under Distribution Shift Based on Unlabeled Examples
Modern image classifiers achieve high predictive accuracy, but the
predictions typically come without reliable uncertainty estimates. Conformal
prediction algorithms provide uncertainty estimates by predicting a set of
classes based on the probability estimates of the classifier (for example, the
softmax scores). To provide such sets, conformal prediction algorithms often
rely on estimating a cutoff threshold for the probability estimates, and this
threshold is chosen based on a calibration set. Conformal prediction methods
guarantee reliability only when the calibration set is from the same
distribution as the test set. Therefore, the methods need to be recalibrated
for new distributions. However, in practice, labeled data from new
distributions is rarely available, making calibration infeasible. In this work,
we consider the problem of predicting the cutoff threshold for a new
distribution based on unlabeled examples only. While it is impossible in
general to guarantee reliability when calibrating based on unlabeled examples,
we show that our method provides excellent uncertainty estimates under natural
distribution shifts, and provably works for a specific model of a distribution
shift
- …