Reliable sequential testing for statistical model checking

We introduce a framework for comparing statistical model checking (SMC) techniques and propose a new, more reliable, SMC technique. Statistical model checking has recently been implemented in tools like UPPAAL and PRISM to be able to handle models which are too complex for numerical analysis. However, these techniques turn out to have shortcomings, most notably that the validity of their outcomes depends on parameters that must be chosen a priori. Our new technique does not have this problem; we prove its correctness, and numerically compare its performance to existing techniques

Interactive comparison of hypothesis tests for statistical model checking

We present a web-based interactive comparison of hypothesis tests as are used in statistical model checking, providing users and tool developers with more insight into their characteristics. Parameters can be modified easily and their influence is visualized in real time; an integrated simulation engine further illustrates the behaviour of the tests. Finally, since the source code is available, it can serve as a framework in which newly developed tests can be tried

On hypothesis testing for statistical model checking

Hypothesis testing is an important part of statistical model checking (SMC). It is typically used to verify statements of the form $p>p_0$ or $p<p_0$, where $p$ is an unknown probability intrinsic to the system model and $p_0$ is a given threshold value. Many techniques for this have been introduced in the SMC literature. We give a comprehensive overview and comparison of these techniques, starting by introducing a framework in which they can all be described. We distinguish between three classes of techniques, differing in what type of output correctness guarantees they give when the true $p$ is very close to the threshold $p_0$. For each technique, we show how to parametrise it in terms of quantities that are meaningful to the user. Having parametrised them consistently, we graphically compare the boundaries of their decision thresholds, and numerically compare the correctness, power and efficiency of the tests. A companion website allows users to get more insight in the properties of the tests by interactively manipulating the parameters