41 research outputs found
Using Valued Booleans to Find Simpler Counterexamples in Random Testing of Cyber-Physical Systems
We propose a new logic of valued Booleans for writing properties which are not just true or false but compute how severely they are falsified. The logic is reminiscent of STL or MTL but gives the tester control over what severity means in the particular problem domain. We use this logic to simplify failing test inputs in the context of random testing of cyber-physical systems and show that it improves the quality of counterexamples found. The logic of valued Booleans might also be used as an alternative to the standard robust semantics of STL formulas in optimization-based approaches to falsification
Average-based Robustness for Continuous-Time Signal Temporal Logic
We propose a new robustness score for continuous-time Signal Temporal Logic
(STL) specifications. Instead of considering only the most severe point along
the evolution of the signal, we use average scores to extract more information
from the signal, emphasizing robust satisfaction of all the specifications'
subformulae over their entire time interval domains. We demonstrate the
advantages of this new score in falsification and control synthesis problems in
systems with complex dynamics and multi-agent systems.Comment: Accepted for publication in the proceedings of Conference on Decision
and Control 201
Time-Staging Enhancement of Hybrid System Falsification
Optimization-based falsification employs stochastic optimization algorithms
to search for error input of hybrid systems. In this paper we introduce a
simple idea to enhance falsification, namely time staging, that allows the
time-causal structure of time-dependent signals to be exploited by the
optimizers. Time staging consists of running a falsification solver multiple
times, from one interval to another, incrementally constructing an input signal
candidate. Our experiments show that time staging can dramatically increase
performance in some realistic examples. We also present theoretical results
that suggest the kinds of models and specifications for which time staging is
likely to be effective
Falsification of Cyber-Physical Systems with Robustness-Guided Black-Box Checking
For exhaustive formal verification, industrial-scale cyber-physical systems
(CPSs) are often too large and complex, and lightweight alternatives (e.g.,
monitoring and testing) have attracted the attention of both industrial
practitioners and academic researchers. Falsification is one popular testing
method of CPSs utilizing stochastic optimization. In state-of-the-art
falsification methods, the result of the previous falsification trials is
discarded, and we always try to falsify without any prior knowledge. To
concisely memorize such prior information on the CPS model and exploit it, we
employ Black-box checking (BBC), which is a combination of automata learning
and model checking. Moreover, we enhance BBC using the robust semantics of STL
formulas, which is the essential gadget in falsification. Our experiment
results suggest that our robustness-guided BBC outperforms a state-of-the-art
falsification tool.Comment: Accepted to HSCC 202
Falsification of Signal-Based Specifications for Cyber-Physical Systems
In the development of software for modern Cyber-Physical Systems, testing is an integral part that is rightfully given a lot of attention. Testing is done on many different abstraction levels, and especially for large-scale industrial systems, it can be difficult to know when the testing should conclude and the software can be considered correct enough for making its way into production. This thesis proposes new methods for analyzing and generating test cases as a means of being more certain that proper testing has been performed for the system under test. For analysis, the proposed approach includes automatically finding how much a given test suite has executed the physical properties of the simulated system. For test case generation, an up-and-coming approach to find errors in Cyber-Physical Systems is simulation-based falsification. While falsification is suitable also for some large-scale industrial systems, sometimes there is a gap between what has been researched and what problems need to be solved to make the approach tractable in the industry. This thesis attempts to close this gap by applying falsification techniques to real-world models from Volvo Car Corporation, and adapting the falsification procedure where it has shortcomings for certain classes of systems. Specifically, the thesis includes a method for automatically transforming a signal-based specification into a formal specification in temporal logic, as well as a modification to the underlying optimization problem that makes falsification more viable in an industrial setting. The proposed methods have been evaluated for both academic benchmark examples and real-world industrial models. One of the main conclusions is that the proposed additions and changes to analysis and generation of tests can be useful, given that one has enough information about the system under test. It is difficult to provide a general solution that will always work best -- instead, the challenge lies in identifying which properties of the given system should be taken into account when trying to find potential errors in the system