4,659 research outputs found
Calibration: Respice, Adspice, Prospice
“Those who claim for themselves to judge the truth are bound to possess a criterion of truth.” JEL Code: C18, C53, D89calibration, prediction
Estimating Uncertainty Online Against an Adversary
Assessing uncertainty is an important step towards ensuring the safety and
reliability of machine learning systems. Existing uncertainty estimation
techniques may fail when their modeling assumptions are not met, e.g. when the
data distribution differs from the one seen at training time. Here, we propose
techniques that assess a classification algorithm's uncertainty via calibrated
probabilities (i.e. probabilities that match empirical outcome frequencies in
the long run) and which are guaranteed to be reliable (i.e. accurate and
calibrated) on out-of-distribution input, including input generated by an
adversary. This represents an extension of classical online learning that
handles uncertainty in addition to guaranteeing accuracy under adversarial
assumptions. We establish formal guarantees for our methods, and we validate
them on two real-world problems: question answering and medical diagnosis from
genomic data
Adversarial Calibrated Regression for Online Decision Making
Accurately estimating uncertainty is an essential component of
decision-making and forecasting in machine learning. However, existing
uncertainty estimation methods may fail when data no longer follows the
distribution seen during training. Here, we introduce online uncertainty
estimation algorithms that are guaranteed to be reliable on arbitrary streams
of data points, including data chosen by an adversary. Specifically, our
algorithms perform post-hoc recalibration of a black-box regression model and
produce outputs that are provably calibrated -- i.e., an 80% confidence
interval will contain the true outcome 80% of the time -- and that have low
regret relative to the learning objective of the base model. We apply our
algorithms in the context of Bayesian optimization, an online model-based
decision-making task in which the data distribution shifts over time, and
observe accelerated convergence to improved optima. Our results suggest that
robust uncertainty quantification has the potential to improve online
decision-making.Comment: arXiv admin note: text overlap with arXiv:1607.0359
Treatment of input uncertainty in hydrologic modeling: Doing hydrology backward with Markov chain Monte Carlo simulation
There is increasing consensus in the hydrologic literature that an appropriate framework for streamflow forecasting and simulation should include explicit recognition of forcing and parameter and model structural error. This paper presents a novel Markov chain Monte Carlo (MCMC) sampler, entitled differential evolution adaptive Metropolis (DREAM), that is especially designed to efficiently estimate the posterior probability density function of hydrologic model parameters in complex, high-dimensional sampling problems. This MCMC scheme adaptively updates the scale and orientation of the proposal distribution during sampling and maintains detailed balance and ergodicity. It is then demonstrated how DREAM can be used to analyze forcing data error during watershed model calibration using a five-parameter rainfall-runoff model with streamflow data from two different catchments. Explicit treatment of precipitation error during hydrologic model calibration not only results in prediction uncertainty bounds that are more appropriate but also significantly alters the posterior distribution of the watershed model parameters. This has significant implications for regionalization studies. The approach also provides important new ways to estimate areal average watershed precipitation, information that is of utmost importance for testing hydrologic theory, diagnosing structural errors in models, and appropriately benchmarking rainfall measurement devices
- …