177,706 research outputs found
Calibrated Prediction Intervals for Neural Network Regressors
Ongoing developments in neural network models are continually advancing the
state of the art in terms of system accuracy. However, the predicted labels
should not be regarded as the only core output; also important is a
well-calibrated estimate of the prediction uncertainty. Such estimates and
their calibration are critical in many practical applications. Despite their
obvious aforementioned advantage in relation to accuracy, contemporary neural
networks can, generally, be regarded as poorly calibrated and as such do not
produce reliable output probability estimates. Further, while post-processing
calibration solutions can be found in the relevant literature, these tend to be
for systems performing classification. In this regard, we herein present two
novel methods for acquiring calibrated predictions intervals for neural network
regressors: empirical calibration and temperature scaling. In experiments using
different regression tasks from the audio and computer vision domains, we find
that both our proposed methods are indeed capable of producing calibrated
prediction intervals for neural network regressors with any desired confidence
level, a finding that is consistent across all datasets and neural network
architectures we experimented with. In addition, we derive an additional
practical recommendation for producing more accurate calibrated prediction
intervals. We release the source code implementing our proposed methods for
computing calibrated predicted intervals. The code for computing calibrated
predicted intervals is publicly available
Contact intervals, survival analysis of epidemic data, and estimation of R_0
We argue that the time from the onset of infectiousness to infectious
contact, which we call the contact interval, is a better basis for inference in
epidemic data than the generation or serial interval. Since contact intervals
can be right-censored, survival analysis is the natural approach to estimation.
Estimates of the contact interval distribution can be used to estimate R_0 in
both mass-action and network-based models.Comment: 30 pages, 4 figures; submitted to Biostatistic
The ROMES method for statistical modeling of reduced-order-model error
This work presents a technique for statistically modeling errors introduced
by reduced-order models. The method employs Gaussian-process regression to
construct a mapping from a small number of computationally inexpensive `error
indicators' to a distribution over the true error. The variance of this
distribution can be interpreted as the (epistemic) uncertainty introduced by
the reduced-order model. To model normed errors, the method employs existing
rigorous error bounds and residual norms as indicators; numerical experiments
show that the method leads to a near-optimal expected effectivity in contrast
to typical error bounds. To model errors in general outputs, the method uses
dual-weighted residuals---which are amenable to uncertainty control---as
indicators. Experiments illustrate that correcting the reduced-order-model
output with this surrogate can improve prediction accuracy by an order of
magnitude; this contrasts with existing `multifidelity correction' approaches,
which often fail for reduced-order models and suffer from the curse of
dimensionality. The proposed error surrogates also lead to a notion of
`probabilistic rigor', i.e., the surrogate bounds the error with specified
probability
Dynamic Control of Explore/Exploit Trade-Off In Bayesian Optimization
Bayesian optimization offers the possibility of optimizing black-box
operations not accessible through traditional techniques. The success of
Bayesian optimization methods such as Expected Improvement (EI) are
significantly affected by the degree of trade-off between exploration and
exploitation. Too much exploration can lead to inefficient optimization
protocols, whilst too much exploitation leaves the protocol open to strong
initial biases, and a high chance of getting stuck in a local minimum.
Typically, a constant margin is used to control this trade-off, which results
in yet another hyper-parameter to be optimized. We propose contextual
improvement as a simple, yet effective heuristic to counter this - achieving a
one-shot optimization strategy. Our proposed heuristic can be swiftly
calculated and improves both the speed and robustness of discovery of optimal
solutions. We demonstrate its effectiveness on both synthetic and real world
problems and explore the unaccounted for uncertainty in the pre-determination
of search hyperparameters controlling explore-exploit trade-off.Comment: Accepted for publication in the proceedings of 2018 Computing
Conferenc
- …