3 research outputs found
Time-Staging Enhancement of Hybrid System Falsification
Optimization-based falsification employs stochastic optimization algorithms
to search for error input of hybrid systems. In this paper we introduce a
simple idea to enhance falsification, namely time staging, that allows the
time-causal structure of time-dependent signals to be exploited by the
optimizers. Time staging consists of running a falsification solver multiple
times, from one interval to another, incrementally constructing an input signal
candidate. Our experiments show that time staging can dramatically increase
performance in some realistic examples. We also present theoretical results
that suggest the kinds of models and specifications for which time staging is
likely to be effective
Conformance Testing for Stochastic Cyber-Physical Systems
Conformance is defined as a measure of distance between the behaviors of two
dynamical systems. The notion of conformance can accelerate system design when
models of varying fidelities are available on which analysis and control design
can be done more efficiently. Ultimately, conformance can capture distance
between design models and their real implementations and thus aid in robust
system design. In this paper, we are interested in the conformance of
stochastic dynamical systems. We argue that probabilistic reasoning over the
distribution of distances between model trajectories is a good measure for
stochastic conformance. Additionally, we propose the non-conformance risk to
reason about the risk of stochastic systems not being conformant. We show that
both notions have the desirable transference property, meaning that conformant
systems satisfy similar system specifications, i.e., if the first model
satisfies a desirable specification, the second model will satisfy (nearly) the
same specification. Lastly, we propose how stochastic conformance and the
non-conformance risk can be estimated from data using statistical tools such as
conformal prediction. We present empirical evaluations of our method on an F-16
aircraft, an autonomous vehicle, a spacecraft, and Dubin's vehicle
On Optimization-Based Falsification of Cyber-Physical Systems
In what is commonly referred to as cyber-physical systems (CPSs), computational and physical resources are closely interconnected. An example is the closed-loop behavior of perception, planning, and control algorithms, executing on a computer and interacting with a physical environment. Many CPSs are safety-critical, and it is thus important to guarantee that they behave according to given specifications that define the correct behavior. CPS models typically include differential equations, state machines, and code written in general-purpose programming languages. This heterogeneity makes it generally not feasible to use analytical methods to evaluate the system’s correctness. Instead, model-based testing of a simulation of the system is more viable. Optimization-based falsification is an approach to, using a simulation model, automatically check for the existence of input signals that make the CPS violate given specifications. Quantitative semantics estimate how far the specification is from being violated for a given scenario. The decision variables in the optimization problems are parameters that determine the type and shape of generated input signals. This thesis contributes to the increased efficiency of optimization-based falsification in four ways. (i) A method for using multiple quantitative semantics during optimization-based falsification. (ii) A direct search approach, called line-search falsification that prioritizes extreme values, which are known to often falsify specifications, and has a good balance between exploration and exploitation of the parameter space. (iii) An adaptation of Bayesian optimization that allows for injecting prior knowledge and uses a special acquisition function for finding falsifying points rather than the global minima. (iv) An investigation of different input signal parameterizations and their coverability of the space and time and frequency domains. The proposed methods have been implemented and evaluated on standard falsification benchmark problems. Based on these empirical studies, we show the efficiency of the proposed methods. Taken together, the proposed methods are important contributions to the falsification of CPSs and in enabling a more efficient falsification process