2,130 research outputs found
High-Dimensional Prediction for Sequential Decision Making
We study the problem of making predictions of an adversarially chosen
high-dimensional state that are unbiased subject to an arbitrary collection of
conditioning events, with the goal of tailoring these events to downstream
decision makers. We give efficient algorithms for solving this problem, as well
as a number of applications that stem from choosing an appropriate set of
conditioning events.
For example, we can efficiently make predictions targeted at polynomially
many decision makers, giving each of them optimal swap regret if they
best-respond to our predictions. We generalize this to online combinatorial
optimization, where the decision makers have a very large action space, to give
the first algorithms offering polynomially many decision makers no regret on
polynomially many subsequences that may depend on their actions and the
context. We apply these results to get efficient no-subsequence-regret
algorithms in extensive-form games (EFGs), yielding a new family of regret
guarantees for EFGs that generalizes some existing EFG regret notions, e.g.
regret to informed causal deviations, and is generally incomparable to other
known such notions.
Next, we develop a novel transparent alternative to conformal prediction for
building valid online adversarial multiclass prediction sets. We produce class
scores that downstream algorithms can use for producing valid-coverage
prediction sets, as if these scores were the true conditional class
probabilities. We show this implies strong conditional validity guarantees
including set-size-conditional and multigroup-fair coverage for polynomially
many downstream prediction sets. Moreover, our class scores can be guaranteed
to have improved loss, cross-entropy loss, and generally any Bregman
loss, compared to any collection of benchmark models, yielding a
high-dimensional real-valued version of omniprediction.Comment: Added references, Arxiv abstract edite
Constructing Prediction Intervals with Neural Networks: An Empirical Evaluation of Bootstrapping and Conformal Inference Methods
Artificial neural networks (ANNs) are popular tools for accomplishing many machine learning tasks, including predicting continuous outcomes. However, the general lack of confidence measures provided with ANN predictions limit their applicability, especially in military settings where accuracy is paramount. Supplementing point predictions with prediction intervals (PIs) is common for other learning algorithms, but the complex structure and training of ANNs renders constructing PIs difficult. This work provides the network design choices and inferential methods for creating better performing PIs with ANNs to enable their adaptation for military use. A two-step experiment is executed across 11 datasets, including an imaged-based dataset. Two non-parametric methods for constructing PIs, bootstrapping and conformal inference, are considered. The results of the first experimental step reveal that the choices inherent to building an ANN affect PI performance. Guidance is provided for optimizing PI performance with respect to each network feature and PI method. In the second step, 20 algorithms for constructing PIs—each using the principles of bootstrapping or conformal inference—are implemented to determine which provides the best performance while maintaining reasonable computational burden. In general, this trade-off is optimized when implementing the cross-conformal method, which maintained interval coverage and efficiency with decreased computational burden
Distribution-Free Model-Agnostic Regression Calibration via Nonparametric Methods
In this paper, we consider the uncertainty quantification problem for
regression models. Specifically, we consider an individual calibration
objective for characterizing the quantiles of the prediction model. While such
an objective is well-motivated from downstream tasks such as newsvendor cost,
the existing methods have been largely heuristic and lack of statistical
guarantee in terms of individual calibration. We show via simple examples that
the existing methods focusing on population-level calibration guarantees such
as average calibration or sharpness can lead to harmful and unexpected results.
We propose simple nonparametric calibration methods that are agnostic of the
underlying prediction model and enjoy both computational efficiency and
statistical consistency. Our approach enables a better understanding of the
possibility of individual calibration, and we establish matching upper and
lower bounds for the calibration error of our proposed methods. Technically,
our analysis combines the nonparametric analysis with a covering number
argument for parametric analysis, which advances the existing theoretical
analyses in the literature of nonparametric density estimation and quantile
bandit problems. Importantly, the nonparametric perspective sheds new
theoretical insights into regression calibration in terms of the curse of
dimensionality and reconciles the existing results on the impossibility of
individual calibration. To our knowledge, we make the first effort to reach
both individual calibration and finite-sample guarantee with minimal
assumptions in terms of conformal prediction. Numerical experiments show the
advantage of such a simple approach under various metrics, and also under
covariates shift. We hope our work provides a simple benchmark and a starting
point of theoretical ground for future research on regression calibration.Comment: Accepted at NeurIPS 2023 and update a camera-ready version; Add some
experiments and literature review
- …