23,832 research outputs found
The Optimal Uncertainty Algorithm in the Mystic Framework
We have recently proposed a rigorous framework for Uncertainty Quantification
(UQ) in which UQ objectives and assumption/information set are brought into the
forefront, providing a framework for the communication and comparison of UQ
results. In particular, this framework does not implicitly impose inappropriate
assumptions nor does it repudiate relevant information. This framework, which
we call Optimal Uncertainty Quantification (OUQ), is based on the observation
that given a set of assumptions and information, there exist bounds on
uncertainties obtained as values of optimization problems and that these bounds
are optimal. It provides a uniform environment for the optimal solution of the
problems of validation, certification, experimental design, reduced order
modeling, prediction, extrapolation, all under aleatoric and epistemic
uncertainties. OUQ optimization problems are extremely large, and even though
under general conditions they have finite-dimensional reductions, they must
often be solved numerically. This general algorithmic framework for OUQ has
been implemented in the mystic optimization framework. We describe this
implementation, and demonstrate its use in the context of the Caltech surrogate
model for hypervelocity impact
Bayesian Uncertainty Integration for Model Calibration, Validation, and Prediction
This paper proposes a comprehensive approach to prediction under uncertainty by application to the Sandia National Laboratories verification and validation challenge problem. In this problem, legacy data and experimental measurements of different levels of fidelity and complexity (e.g., coupon tests, material and fluid characterizations, and full system tests/measurements) compose a hierarchy of information where fewer observations are available at higher levels of system complexity. This paper applies a Bayesian methodology in order to incorporate information at different levels of the hierarchy and include the impact of sparse data in the prediction uncertainty for the system of interest. Since separation of aleatory and epistemic uncertainty sources is a pervasive issue in calibration and validation, maintaining this separation in order to perform these activities correctly is the primary focus of this paper. Toward this goal, a Johnson distribution family approach to calibration is proposed in order to enable epistemic and aleatory uncertainty to be separated in the posterior parameter distributions. The model reliability metric approach to validation is then applied, and a novel method of handling combined aleatory and epistemic uncertainty is introduced. The quality of the validation assessment is used to modify the parameter uncertainty and add conservatism to the prediction of interest. Finally, this prediction with its associated uncertainty is used to assess systemlevel reliability (a prediction goal for the challenge problem)
Paying attention to cardiac surgical risk: An interpretable machine learning approach using an uncertainty-aware attentive neural network
Machine learning (ML) is increasingly applied to predict adverse postoperative outcomes in
cardiac surgery. Commonly used ML models fail to translate to clinical practice due to
absent model explainability, limited uncertainty quantification, and no flexibility to missing
data. We aimed to develop and benchmark a novel ML approach, the uncertainty-aware
attention network (UAN), to overcome these common limitations. Two Bayesian uncertainty
quantification methods were tested, generalized variational inference (GVI) or a posterior
network (PN). The UAN models were compared with an ensemble of XGBoost models and
a Bayesian logistic regression model (LR) with imputation. The derivation datasets consisted
of 153,932 surgery events from the Australian and New Zealand Society of Cardiac
and Thoracic Surgeons (ANZSCTS) Cardiac Surgery Database. An external validation consisted
of 7343 surgery events which were extracted from the Medical Information Mart for
Intensive Care (MIMIC) III critical care dataset. The highest performing model on the external
validation dataset was a UAN-GVI with an area under the receiver operating characteristic
curve (AUC) of 0.78 (0.01). Model performance improved on high confidence samples
with an AUC of 0.81 (0.01). Confidence calibration for aleatoric uncertainty was excellent for
all models. Calibration for epistemic uncertainty was more variable, with an ensemble of
XGBoost models performing the best with an AUC of 0.84 (0.08). Epistemic uncertainty was
improved using the PN approach, compared to GVI. UAN is able to use an interpretable and
flexible deep learning approach to provide estimates of model uncertainty alongside stateof-
the-art predictions. The model has been made freely available as an easy-to-use web
application demonstrating that by designing uncertainty-aware models with innately explainable
predictions deep learning may become more suitable for routine clinical use.The ANZSCTS Cardiac Surgery Database
Program is funded by the Department of Health
(Victoria), the Clinical Excellence Commission
(NSW)Queensland Health (QLD)ANZSCTS Database Research
activities are supported through a National Health
and Medical Research Council Principal Research
Fellowship (APP 1136372)Program Grant
(APP 1092642
Uncertainty Estimation in One-Stage Object Detection
Environment perception is the task for intelligent vehicles on which all
subsequent steps rely. A key part of perception is to safely detect other road
users such as vehicles, pedestrians, and cyclists. With modern deep learning
techniques huge progress was made over the last years in this field. However
such deep learning based object detection models cannot predict how certain
they are in their predictions, potentially hampering the performance of later
steps such as tracking or sensor fusion. We present a viable approaches to
estimate uncertainty in an one-stage object detector, while improving the
detection performance of the baseline approach. The proposed model is evaluated
on a large scale automotive pedestrian dataset. Experimental results show that
the uncertainty outputted by our system is coupled with detection accuracy and
the occlusion level of pedestrians
Physics-related epistemic uncertainties in proton depth dose simulation
A set of physics models and parameters pertaining to the simulation of proton
energy deposition in matter are evaluated in the energy range up to
approximately 65 MeV, based on their implementations in the Geant4 toolkit. The
analysis assesses several features of the models and the impact of their
associated epistemic uncertainties, i.e. uncertainties due to lack of
knowledge, on the simulation results. Possible systematic effects deriving from
uncertainties of this kind are highlighted; their relevance in relation to the
application environment and different experimental requirements are discussed,
with emphasis on the simulation of radiotherapy set-ups. By documenting
quantitatively the features of a wide set of simulation models and the related
intrinsic uncertainties affecting the simulation results, this analysis
provides guidance regarding the use of the concerned simulation tools in
experimental applications; it also provides indications for further
experimental measurements addressing the sources of such uncertainties.Comment: To be published in IEEE Trans. Nucl. Sc
Concrete Dropout
Dropout is used as a practical tool to obtain uncertainty estimates in large
vision models and reinforcement learning (RL) tasks. But to obtain
well-calibrated uncertainty estimates, a grid-search over the dropout
probabilities is necessary - a prohibitive operation with large models, and an
impossible one with RL. We propose a new dropout variant which gives improved
performance and better calibrated uncertainties. Relying on recent developments
in Bayesian deep learning, we use a continuous relaxation of dropout's discrete
masks. Together with a principled optimisation objective, this allows for
automatic tuning of the dropout probability in large models, and as a result
faster experimentation cycles. In RL this allows the agent to adapt its
uncertainty dynamically as more data is observed. We analyse the proposed
variant extensively on a range of tasks, and give insights into common practice
in the field where larger dropout probabilities are often used in deeper model
layers
- …