Conformal Prediction: a Unified Review of Theory and New Challenges

In this work we provide a review of basic ideas and novel developments about Conformal Prediction -- an innovative distribution-free, non-parametric forecasting method, based on minimal assumptions -- that is able to yield in a very straightforward way predictions sets that are valid in a statistical sense also in in the finite sample case. The in-depth discussion provided in the paper covers the theoretical underpinnings of Conformal Prediction, and then proceeds to list the more advanced developments and adaptations of the original idea.Comment: arXiv admin note: text overlap with arXiv:0706.3188, arXiv:1604.04173, arXiv:1709.06233, arXiv:1203.5422 by other author

Strong validity, consonance, and conformal prediction

Valid prediction of future observations is an important and challenging problem. The two general approaches for quantifying uncertainty about the future value employ prediction regions and predictive distribution, respectively, with the latter usually considered to be more informative because it performs other prediction-related tasks. Standard notions of validity focus on the former, i.e., coverage probability bounds for prediction regions, but a notion of validity relevant to the other prediction-related tasks performed by the latter is lacking. In this paper, we present a new notion---strong prediction validity---relevant to these more general prediction tasks. We show that strong validity is connected to more familiar notions of coherence, and argue that imprecise probability considerations are required in order to achieve it. We go on to show that strong prediction validity can be achieved by interpreting the conformal prediction output as the contour function of a consonant plausibility function. We also offer an alternative characterization, based on a new nonparametric inferential model construction, wherein the appearance of consonance is more natural, and prove strong prediction validity.Comment: 34 pages, 3 figures, 2 tables. Comments welcome at https://www.researchers.one/article/2020-01-1

A review of probabilistic forecasting and prediction with machine learning

Predictions and forecasts of machine learning models should take the form of probability distributions, aiming to increase the quantity of information communicated to end users. Although applications of probabilistic prediction and forecasting with machine learning models in academia and industry are becoming more frequent, related concepts and methods have not been formalized and structured under a holistic view of the entire field. Here, we review the topic of predictive uncertainty estimation with machine learning algorithms, as well as the related metrics (consistent scoring functions and proper scoring rules) for assessing probabilistic predictions. The review covers a time period spanning from the introduction of early statistical (linear regression and time series models, based on Bayesian statistics or quantile regression) to recent machine learning algorithms (including generalized additive models for location, scale and shape, random forests, boosting and deep learning algorithms) that are more flexible by nature. The review of the progress in the field, expedites our understanding on how to develop new algorithms tailored to users' needs, since the latest advancements are based on some fundamental concepts applied to more complex algorithms. We conclude by classifying the material and discussing challenges that are becoming a hot topic of research.Comment: 83 pages, 5 figure