Evaluating Model Testing and Model Checking for Finding Requirements Violations in Simulink Models

Matlab/Simulink is a development and simulation language that is widely used by the Cyber-Physical System (CPS) industry to model dynamical systems. There are two mainstream approaches to verify CPS Simulink models: model testing that attempts to identify failures in models by executing them for a number of sampled test inputs, and model checking that attempts to exhaustively check the correctness of models against some given formal properties. In this paper, we present an industrial Simulink model benchmark, provide a categorization of different model types in the benchmark, describe the recurring logical patterns in the model requirements, and discuss the results of applying model checking and model testing approaches to identify requirements violations in the benchmarked models. Based on the results, we discuss the strengths and weaknesses of model testing and model checking. Our results further suggest that model checking and model testing are complementary and by combining them, we can significantly enhance the capabilities of each of these approaches individually. We conclude by providing guidelines as to how the two approaches can be best applied together.Comment: 10 pages + 2 page reference

Generating Automated and Online Test Oracles for Simulink Models with Continuous and Uncertain Behaviors

Test automation requires automated oracles to assess test outputs. For cyber physical systems (CPS), oracles, in addition to be automated, should ensure some key objectives: (i) they should check test outputs in an online manner to stop expensive test executions as soon as a failure is detected; (ii) they should handle time- and magnitude-continuous CPS behaviors; (iii) they should provide a quantitative degree of satisfaction or failure measure instead of binary pass/fail outputs; and (iv) they should be able to handle uncertainties due to CPS interactions with the environment. We propose an automated approach to translate CPS requirements specified in a logic-based language into test oracles specified in Simulink - a widely-used development and simulation language for CPS. Our approach achieves the objectives noted above through the identification of a fragment of Signal First Order logic (SFOL) to specify requirements, the definition of a quantitative semantics for this fragment and a sound translation of the fragment into Simulink. The results from applying our approach on 11 industrial case studies show that: (i) our requirements language can express all the 98 requirements of our case studies; (ii) the time and effort required by our approach are acceptable, showing potentials for the adoption of our work in practice, and (iii) for large models, our approach can dramatically reduce the test execution time compared to when test outputs are checked in an offline manner

Verification of design models of cyber-physical systems specified in Simulink

Recent advances in cyber-physical systems (CPS) have allowed highly available and approachable technologies with interconnected systems between the physical assets and the computational software components. This has resulted in more complex systems with wider capabilities. For example, they can be applied in various domains such as safe transport, efficient medical devices, integrated systems, critical infrastructure control and more. The development of such critical systems requires advanced new models, algorithms, methods and tools to verify and validate the software components and the entire system. The verification of cyber-physical systems has become challenging: (1) The complex and dynamical behaviour of systems requires resilient automated monitors and test oracles that can cope with time-varying variables of CPS. (2) Given the wide range of existing verification and testing techniques from formal to empirical methods, there is no clear guidance as to how different techniques fare in the context of CPS. (3) Due to serious issues when applying exhaustive verification to complex systems, a common practice is needed to verify system components separately. This requires adding implicit assumptions about the operational environment of system components to ensure correct verification. However, identifying environment assumptions for cyber-physical systems with complex, mathematical behaviors is not trivial. In this dissertation, we focus on addressing these challenges. In this dissertation, we propose a set of effective approaches to verify design models of CPS. The work presented in this dissertation is motivated by ESAIL maritime micro-satellite system, developed by LuxSpace, a leading provider of space systems, applications and services in Luxembourg. In addition to ESAIL, we use a benchmark of eleven public-domain Simulink models provided by Lockheed Martin, which are representative of different categories of CPS models in the aerospace and defence sector. To address the aforementioned challenges, we propose (1) an automated approach to translate CPS requirements specified in a logic-based language into test oracles specified in Simulink. The generated oracles are able to deal with CPS complex behaviours and interactions with the system environment; (2) An empirical study to evaluate the fault-finding capabilities of model testing and model checking techniques for Simulink models. We also provide a categorization of model types and a set of common logical patterns for CPS requirements; (3) An automated approach to synthesize environment assumptions for a component under analysis by combining search-based testing, machine learning and model checking procedures. We also propose a novel technique to guide the test generation based on the feedback received from the machine learning process; and (4) An extension of (3) to learn more complex assumptions with arithmetic expressions over multiple signals and numerical variables

Optimal deployment of configurable business processes in cloud federations

Configurable processes are increasingly being adopted by enterprises that seek experience sharing and best practice adoption. A configurable process is a customizable model that specifies how different enterprises perform similar processes. At the modeling level, a configurable process model provides for flexible business process (BP) reuse by (de)selecting the (ir)relevant parts to derive a particular process variant. At the exploitation level, it offers flexibility and agility to an enterprise looking to outsource its BP to different providers cooperating in a cloud federation. More specifically, an enterprise can use a configurable process model to derive particular process variants that it outsources depending on its objectives. In particular, it may opt for outsourcing the variant that results in the optimal deployment, e.g., having the minimal cost of allocated cloud services that fulfill the user quality of service (QoS) requirements. However, identifying the optimal deployment variants is a complex problem because of the heterogeneity of services within a cloud federation and the number of possible variants that can be derived from a configurable process model. In addition, the complexity of this problem increases for variable user QoS requirements. In this paper, we propose an approach to derive, from a configurable process model, the variant that has the optimal deployment in a cloud federation. We propose a linear programming approach that accounts for the variability of both the BP model and the user QoS requirements, and that ensures an optimal time-aware cloud service allocation. We experimentally show the effectiveness and flexibility of our approach on a generated testbed

Optimal deployment of configurable business processes in cloud federations

\u3cp\u3eConfigurable processes are increasingly being adopted by enterprises that seek experience sharing and best practice adoption. A configurable process is a customizable model that specifies how different enterprises perform similar processes. At the modeling level, a configurable process model provides for flexible business process (BP) reuse by (de)selecting the (ir)relevant parts to derive a particular process variant. At the exploitation level, it offers flexibility and agility to an enterprise looking to outsource its BP to different providers cooperating in a cloud federation. More specifically, an enterprise can use a configurable process model to derive particular process variants that it outsources depending on its objectives. In particular, it may opt for outsourcing the variant that results in the optimal deployment, e.g., having the minimal cost of allocated cloud services that fulfill the user quality of service (QoS) requirements. However, identifying the optimal deployment variants is a complex problem because of the heterogeneity of services within a cloud federation and the number of possible variants that can be derived from a configurable process model. In addition, the complexity of this problem increases for variable user QoS requirements. In this paper, we propose an approach to derive, from a configurable process model, the variant that has the optimal deployment in a cloud federation. We propose a linear programming approach that accounts for the variability of both the BP model and the user QoS requirements, and that ensures an optimal time-aware cloud service allocation. We experimentally show the effectiveness and flexibility of our approach on a generated testbed.\u3c/p\u3

Combining Genetic Programming and Model Checking to Generate Environment Assumptions

Software verification may yield spurious failures when environment assumptions are not accounted for. Environment assumptions are the expectations that a system or a component makes about its operational environment and are often specified in terms of conditions over the inputs of that system or component. In this article, we propose an approach to automatically infer environment assumptions for Cyber-Physical Systems (CPS). Our approach improves the state-of-the-art in three different ways: First, we learn assumptions for complex CPS models involving signal and numeric variables; second, the learned assumptions include arithmetic expressions defined over multiple variables; third, we identify the trade-off between soundness and coverage of environment assumptions and demonstrate the flexibility of our approach in prioritizing either of these criteria. We evaluate our approach using a public domain benchmark of CPS models from Lockheed Martin and a component of a satellite control system from LuxSpace, a satellite system provider. The results show that our approach outperforms state-of-the-art techniques on learning assumptions for CPS models, and further, when applied to our industrial CPS model, our approach is able to learn assumptions that are sufficiently close to the assumptions manually developed by engineers to be of practical value

Evaluating Model Testing and Model Checking for Finding Requirements Violations in Simulink Models

Matlab/Simulink is a development and simulation language that is widely used by the Cyber-Physical System (CPS) industry to model dynamical systems. There are two mainstream approaches to verify CPS Simulink models: model testing that attempts to identify failures in models by executing them for a number of sampled test inputs, and model checking that attempts to exhaustively check the correctness of models against some given formal properties. In this paper, we present an industrial Simulink model benchmark, provide a categorization of different model types in the benchmark, describe the recurring logical patterns in the model requirements, and discuss the results of applying model checking and model testing approaches to identify requirements violations in the benchmarked models. Based on the results, we discuss the strengths and weaknesses of model testing and model checking. Our results further suggest that model checking and model testing are complementary and by combining them, we can significantly enhance the capabilities of each of these approaches individually. We conclude by providing guidelines as to how the two approaches can be best applied together