35,717 research outputs found
Bayesian Verification under Model Uncertainty
Machine learning enables systems to build and update domain models based on
runtime observations. In this paper, we study statistical model checking and
runtime verification for systems with this ability. Two challenges arise: (1)
Models built from limited runtime data yield uncertainty to be dealt with. (2)
There is no definition of satisfaction w.r.t. uncertain hypotheses. We propose
such a definition of subjective satisfaction based on recently introduced
satisfaction functions. We also propose the BV algorithm as a Bayesian solution
to runtime verification of subjective satisfaction under model uncertainty. BV
provides user-definable stochastic bounds for type I and II errors. We discuss
empirical results from an example application to illustrate our ideas.Comment: Accepted at SEsCPS @ ICSE 201
Interval Change-Point Detection for Runtime Probabilistic Model Checking
Recent probabilistic model checking techniques can verify reliability and performance properties of software systems affected by parametric uncertainty. This involves modelling the system behaviour using interval Markov chains, i.e., Markov models with transition probabilities or rates specified as intervals. These intervals can be updated continually using Bayesian estimators with imprecise priors, enabling the verification of the system properties of interest at runtime. However, Bayesian estimators are slow to react to sudden changes in the actual value of the estimated parameters, yielding inaccurate intervals and leading to poor verification results after such changes. To address this limitation, we introduce an efficient interval change-point detection method, and we integrate it with a state-of-the-art Bayesian estimator with imprecise priors. Our experimental results show that the resulting end-to-end Bayesian approach to change-point detection and estimation of interval Markov chain parameters handles effectively a wide range of sudden changes in parameter values, and supports runtime probabilistic model checking under parametric uncertainty
Data-driven and Model-based Verification: a Bayesian Identification Approach
This work develops a measurement-driven and model-based formal verification
approach, applicable to systems with partly unknown dynamics. We provide a
principled method, grounded on reachability analysis and on Bayesian inference,
to compute the confidence that a physical system driven by external inputs and
accessed under noisy measurements, verifies a temporal logic property. A case
study is discussed, where we investigate the bounded- and unbounded-time safety
of a partly unknown linear time invariant system
Improving the applicability of radar rainfall estimates for urban pluvial flood modelling and forecasting
This work explores the possibility of improving the applicability of radar rainfall estimates (whose accuracy is generally insufficient) to the verification and operation of urban storm-water drainage models by employing a number of local gauge-based radar rainfall adjustment techniques. The adjustment techniques tested in this work include a simple mean-field bias (MFB) adjustment, as well as a more complex Bayesian radar-raingauge data merging method which aims at better preserving the spatial structure of rainfall fields. In addition, a novel technique (namely, local singularity analysis) is introduced and shown to improve the Bayesian method by better capturing and reproducing storm patterns and peaks. Two urban catchments were used as case studies in this work: the Cranbrook catchment (9 km2) in North-East London, and the Portobello catchment (53 km2) in the East of Edinburgh. In the former, the potential benefits of gauge-based adjusted radar rainfall estimates in an operational context were analysed, whereas in the latter the potential benefits of adjusted estimates for model verification purposes were explored. Different rainfall inputs, including raingauge, original radar and the aforementioned merged estimates were fed into the urban drainage models of the two catchments. The hydraulic outputs were compared against available flow and depth records. On the whole, the tested adjustment techniques proved to improve the applicability of radar rainfall estimates to urban hydrological applications, with the Bayesian-based methods, in particular the singularity sensitive one, providing more realistic and accurate rainfall fields which result in better reproduction of the urban drainage system’s dynamics. Further testing is still necessary in order to better assess the benefits of these adjustment methods, identify their shortcomings and improve them accordingly
A Bayesian framework for verification and recalibration of ensemble forecasts: How uncertain is NAO predictability?
Predictability estimates of ensemble prediction systems are uncertain due to
limited numbers of past forecasts and observations. To account for such
uncertainty, this paper proposes a Bayesian inferential framework that provides
a simple 6-parameter representation of ensemble forecasting systems and the
corresponding observations. The framework is probabilistic, and thus allows for
quantifying uncertainty in predictability measures such as correlation skill
and signal-to-noise ratios. It also provides a natural way to produce
recalibrated probabilistic predictions from uncalibrated ensembles forecasts.
The framework is used to address important questions concerning the skill of
winter hindcasts of the North Atlantic Oscillation for 1992-2011 issued by the
Met Office GloSea5 climate prediction system. Although there is much
uncertainty in the correlation between ensemble mean and observations, there is
strong evidence of skill: the 95% credible interval of the correlation
coefficient of [0.19,0.68] does not overlap zero. There is also strong evidence
that the forecasts are not exchangeable with the observations: With over 99%
certainty, the signal-to-noise ratio of the forecasts is smaller than the
signal-to-noise ratio of the observations, which suggests that raw forecasts
should not be taken as representative scenarios of the observations. Forecast
recalibration is thus required, which can be coherently addressed within the
proposed framework.Comment: 36 pages, 10 figure
Validating Predictions of Unobserved Quantities
The ultimate purpose of most computational models is to make predictions,
commonly in support of some decision-making process (e.g., for design or
operation of some system). The quantities that need to be predicted (the
quantities of interest or QoIs) are generally not experimentally observable
before the prediction, since otherwise no prediction would be needed. Assessing
the validity of such extrapolative predictions, which is critical to informed
decision-making, is challenging. In classical approaches to validation, model
outputs for observed quantities are compared to observations to determine if
they are consistent. By itself, this consistency only ensures that the model
can predict the observed quantities under the conditions of the observations.
This limitation dramatically reduces the utility of the validation effort for
decision making because it implies nothing about predictions of unobserved QoIs
or for scenarios outside of the range of observations. However, there is no
agreement in the scientific community today regarding best practices for
validation of extrapolative predictions made using computational models. The
purpose of this paper is to propose and explore a validation and predictive
assessment process that supports extrapolative predictions for models with
known sources of error. The process includes stochastic modeling, calibration,
validation, and predictive assessment phases where representations of known
sources of uncertainty and error are built, informed, and tested. The proposed
methodology is applied to an illustrative extrapolation problem involving a
misspecified nonlinear oscillator
- …