81 research outputs found
From conformal to probabilistic prediction
This paper proposes a new method of probabilistic prediction, which is based
on conformal prediction. The method is applied to the standard USPS data set
and gives encouraging results.Comment: 12 pages, 2 table
Efficiency of conformalized ridge regression
Conformal prediction is a method of producing prediction sets that can be
applied on top of a wide range of prediction algorithms. The method has a
guaranteed coverage probability under the standard IID assumption regardless of
whether the assumptions (often considerably more restrictive) of the underlying
algorithm are satisfied. However, for the method to be really useful it is
desirable that in the case where the assumptions of the underlying algorithm
are satisfied, the conformal predictor loses little in efficiency as compared
with the underlying algorithm (whereas being a conformal predictor, it has the
stronger guarantee of validity). In this paper we explore the degree to which
this additional requirement of efficiency is satisfied in the case of Bayesian
ridge regression; we find that asymptotically conformal prediction sets differ
little from ridge regression prediction intervals when the standard Bayesian
assumptions are satisfied.Comment: 22 pages, 1 figur
Multiple Testing Framework for Out-of-Distribution Detection
We study the problem of Out-of-Distribution (OOD) detection, that is,
detecting whether a learning algorithm's output can be trusted at inference
time. While a number of tests for OOD detection have been proposed in prior
work, a formal framework for studying this problem is lacking. We propose a
definition for the notion of OOD that includes both the input distribution and
the learning algorithm, which provides insights for the construction of
powerful tests for OOD detection. We propose a multiple hypothesis testing
inspired procedure to systematically combine any number of different statistics
from the learning algorithm using conformal p-values. We further provide strong
guarantees on the probability of incorrectly classifying an in-distribution
sample as OOD. In our experiments, we find that threshold-based tests proposed
in prior work perform well in specific settings, but not uniformly well across
different types of OOD instances. In contrast, our proposed method that
combines multiple statistics performs uniformly well across different datasets
and neural networks
Agreeing to Stop: Reliable Latency-Adaptive Decision Making via Ensembles of Spiking Neural Networks
Spiking neural networks (SNNs) are recurrent models that can leverage
sparsity in input time series to efficiently carry out tasks such as
classification. Additional efficiency gains can be obtained if decisions are
taken as early as possible as a function of the complexity of the input time
series. The decision on when to stop inference and produce a decision must rely
on an estimate of the current accuracy of the decision. Prior work demonstrated
the use of conformal prediction (CP) as a principled way to quantify
uncertainty and support adaptive-latency decisions in SNNs. In this paper, we
propose to enhance the uncertainty quantification capabilities of SNNs by
implementing ensemble models for the purpose of improving the reliability of
stopping decisions. Intuitively, an ensemble of multiple models can decide when
to stop more reliably by selecting times at which most models agree that the
current accuracy level is sufficient. The proposed method relies on different
forms of information pooling from ensemble models, and offers theoretical
reliability guarantees. We specifically show that variational inference-based
ensembles with p-variable pooling significantly reduce the average latency of
state-of-the-art methods, while maintaining reliability guarantees.Comment: Under revie
Conformal Inference for Invariant Risk Minimization
The application of machine learning models can be significantly impeded by
the occurrence of distributional shifts, as the assumption of homogeneity
between the population of training and testing samples in machine learning and
statistics may not be feasible in practical situations. One way to tackle this
problem is to use invariant learning, such as invariant risk minimization
(IRM), to acquire an invariant representation that aids in generalization with
distributional shifts. This paper develops methods for obtaining
distribution-free prediction regions to describe uncertainty estimates for
invariant representations, accounting for the distribution shifts of data from
different environments. Our approach involves a weighted conformity score that
adapts to the specific environment in which the test sample is situated. We
construct an adaptive conformal interval using the weighted conformity score
and prove its conditional average under certain conditions. To demonstrate the
effectiveness of our approach, we conduct several numerical experiments,
including simulation studies and a practical example using real-world data.Comment: arXiv admin note: text overlap with arXiv:2209.1135
Offline Policy Evaluation with Out-of-Sample Guarantees
We consider the problem of evaluating the performance of a decision policy
using past observational data. The outcome of a policy is measured in terms of
a loss or disutility (or negative reward) and the problem is to draw valid
inferences about the out-of-sample loss of the specified policy when the past
data is observed under a, possibly unknown, policy. Using a sample-splitting
method, we show that it is possible to draw such inferences with finite-sample
coverage guarantees that evaluate the entire loss distribution. Importantly,
the method takes into account model misspecifications of the past policy --
including unmeasured confounding. The evaluation method can be used to certify
the performance of a policy using observational data under an explicitly
specified range of credible model assumptions
Conformal Risk Control
We extend conformal prediction to control the expected value of any monotone
loss function. The algorithm generalizes split conformal prediction together
with its coverage guarantee. Like conformal prediction, the conformal risk
control procedure is tight up to an factor. Worked examples
from computer vision and natural language processing demonstrate the usage of
our algorithm to bound the false negative rate, graph distance, and token-level
F1-score.Comment: Code available at https://github.com/aangelopoulos/conformal-ris
- …