Software Engineering Challenges for Investigating Cyber-Physical Incidents

Cyber-Physical Systems (CPS) are characterized by the interplay between digital and physical spaces. This characteristic has extended the attack surface that could be exploited by an offender to cause harm. An increasing number of cyber-physical incidents may occur depending on the configuration of the physical and digital spaces and their interplay. Traditional investigation processes are not adequate to investigate these incidents, as they may overlook the extended attack surface resulting from such interplay, leading to relevant evidence being missed and testing flawed hypotheses explaining the incidents. The software engineering research community can contribute to addressing this problem, by deploying existing formalisms to model digital and physical spaces, and using analysis techniques to reason about their interplay and evolution. In this paper, supported by a motivating example, we describe some emerging software engineering challenges to support investigations of cyber-physical incidents. We review and critique existing research proposed to address these challenges, and sketch an initial solution based on a meta-model to represent cyber-physical incidents and a representation of the topology of digital and physical spaces that supports reasoning about their interplay

Evaluating Model Testing and Model Checking for Finding Requirements Violations in Simulink Models

Matlab/Simulink is a development and simulation language that is widely used by the Cyber-Physical System (CPS) industry to model dynamical systems. There are two mainstream approaches to verify CPS Simulink models: model testing that attempts to identify failures in models by executing them for a number of sampled test inputs, and model checking that attempts to exhaustively check the correctness of models against some given formal properties. In this paper, we present an industrial Simulink model benchmark, provide a categorization of different model types in the benchmark, describe the recurring logical patterns in the model requirements, and discuss the results of applying model checking and model testing approaches to identify requirements violations in the benchmarked models. Based on the results, we discuss the strengths and weaknesses of model testing and model checking. Our results further suggest that model checking and model testing are complementary and by combining them, we can significantly enhance the capabilities of each of these approaches individually. We conclude by providing guidelines as to how the two approaches can be best applied together.Comment: 10 pages + 2 page reference

Diversity of graph models and graph generators in mutation testing

When custom modeling tools are used for designing complex safety-critical systems (e.g., critical cyber-physical systems), the tools themselves need to be validated by systematic testing to prevent tool-specific bugs reaching the system. Testing of such modeling tools relies upon an automatically generated set of models as a test suite. While many software testing practices recommend that this test suite should be diverse, model diversity has not been studied systematically for graph models. In the paper, we propose different diversity metrics for models by generalizing and exploiting neighborhood and predicate shapes as abstraction. We evaluate such shape-based diversity metrics using various distance functions in the context of mutation testing of graph constraints and access policies for two separate industrial DSLs. Furthermore, we evaluate the quality (i.e., bug detection capability) of different (random and consistent) model generation techniques for mutation testing purposes

From Verification to Implementation: A Model Translation Tool and a Pacemaker Case Study

Model-Driven Design (MDD) of cyber-physical systems advocates for design procedures that start with formal modeling of the real-time system, followed by the model’s verification at an early stage. The verified model must then be translated to a more detailed model for simulation-based testing and finally translated into executable code in a physical implementation. As later stages build on the same core model, it is essential that models used earlier in the pipeline are valid approximations of the more detailed models developed downstream. The focus of this effort is on the design and development of a model translation tool, UPP2SF, and how it integrates system modeling, verification, model-based WCET analysis, simulation, code generation and testing into an MDD based framework. UPP2SF facilitates automatic conversion of verified timed automata-based models (in UPPAAL) to models that may be simulated and tested (in Simulink/Stateflow). We describe the design rules to ensure the conversion is correct, efficient and applicable to a large class of models. We show how the tool enables MDD of an implantable cardiac pacemaker. We demonstrate that UPP2SF preserves behaviors of the pacemaker model from UPPAAL to Stateflow. The resultant Stateflow chart is automatically converted into C and tested on a hardware platform for a set of requirements

On Falsification of Large-Scale Cyber-Physical Systems

In the development of modern Cyber-Physical Systems, Model-Based Testingof the closed-loop system is an approach for finding potential faults andincreasing quality of developed products. Testing is done on many differentabstraction levels, and for large-scale industrial systems, there are severalchallenges. Executing tests on the systems can be time-consuming and largenumbers of complex specifications need to be thoroughly tested, while manyof the popular academic benchmarks do not necessarily reflect on this complexity.This thesis proposes new methods for analyzing and generating test casesas a means for being more certain that proper testing has been performed onthe system under test. For analysis, the proposed approach can automaticallyfind out how much of the physical parts of the system that the test suite hasexecuted.For test case generation, an approach to find errors is optimization-basedfalsification. This thesis attempts to close the gap between academia and industryby applying falsification techniques to real-world models from VolvoCar Corporation and adapting the falsification procedure where it has shortcomingsfor certain classes of systems. Specifically, the main contributionsof this thesis are (i) a method for automatically transforming a signal-basedspecification into a formal specification allowing an optimization-based falsificationapproach, (ii) a new collection of specifications inspired by large-scalespecifications from industry, (iii) an algorithm to perform optimization-basedfalsification for such a large set of specifications, and (iv) a new type of coveragecriterion for Cyber-Physical Systems that can help to assess when testingcan be concluded.The proposed methods have been evaluated for both academic benchmarkexamples and real-world industrial models. One of the main conclusions isthat the proposed additions and changes to the analysis and generation oftests can be useful, given that one has enough information about the systemunder test. The methods presented in this thesis have been applied to realworldmodels in a way that allows for higher-quality products by finding morefaults in early phases of development