Improving System-Level Verification of SystemC Models with SPIN

SystemC is a de-facto industry standard for developing, modelling, and simulating embedded systems. As embedded systems become more and more integrated into many aspects of human lives (e.g., transportation, surveillance systems, ...), failures of embedded systems might cause dangerous hazards to individuals or groups. Guaranteeing safety of such systems makes formal verification crucial. In this paper we present a novel approach for verifying SystemC models with SPIN. Focusing on system-level verification we reuse compiled and executable code from the original model and embed it into the verifier generated by SPIN. In contrast to most other approaches, which require a complete model transformation, in our approach the transformation focuses only on the relevant parts of the model while leaving functional blocks untransformed. Our technique aims at reducing the state vector size managed by the verifier of SPIN, at improving state exploration performance by avoiding unnecessary model transformation steps, and at concentrating on verifying properties that emerge from the composition of multiple functional units

Automated formal verification and testing of C programs for embedded systems

“This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder." “Copyright IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.”In this paper, we introduce an approach for automated verification and testing of ANSI C programs for embedded systems. We automatically extract an automaton model from the C code of the SUT (system under test). This automaton model is on the one hand used for formal verification of the requirements defined in the system specification, on the other hand, we can derive test cases from this model, for both methods we use a model checker. We describe our techniques for test case generation, based on producing counterexamples with a model checker by formulating trap properties. The resulting test cases can then be applied to the SUT on different test levels. An important issue for model checking C-source code, is the correct modeling of the semantics of a C program for an embedded system. We focus on challenges and possible restrictions that appear, when model checking is used for the verification of C-source code. We specifically show how to deal with arithmetic expressions in the model checker NuSMV and how to preserve the numerical results in case of modeling the platform-specific semantics of C

Nebent. : Methode zur systematischen Testfallgenerierung basierend auf Requirements für sicherheitskritische eingebettete Systeme

Sicherheitskritische Systeme müssen ausreichend getestet werden um sicher zu stellen, dass sie sich nicht fehlerhaft verhalten, da Fehler fatale Auswirkungen haben können. In relevanten Standards sind Bedingungen für den Testprozess definiert, zum Beispiel zu erfüllende Coverage-Kriterien (wie z.B. MC/DC) oder "Traceability", d.h., dass es möglich ist, die generierten Testfälle auf die in der Spezifikation definierten Requirements zurückzuführen. Es existiert eine Vielzahl an Methoden, Testfälle zu erzeugen, aber es ist immer noch eine Herausforderung, Testfälle so systematisch zu generieren, dass sie einerseits auf den Requirements basieren und andererseits mit der resultierenden Menge an Testfällen eine Abdeckung von 100% MC/DC am zu testenden System erreicht wird. Das Ziel dieser Doktorarbeit ist es, einen signifikanten Beitrag zu diesem Problem zu leisten.In dieser Arbeit wird ein Testframework entwickelt mit einer Testfallgenerierungsmethode, die die Testfälle auf Basis der Requirements generiert. Die resultierenden Testfälle haben direkten Bezug zu den in der Spezifikation festgelegten Requirements. Mit den generierten Testfällen erreichen wir maximal mögliche MC/DC Abdeckung auf dem Quellcode des zu testenden Systems für eine sicherheitskritische Fallstudie aus dem Automobilbereich. Zusätzlich evaluieren wir die tatsächliche Fehlererkennungsrate für drei verschiedene Fehlerszenarien (Fehler in den Werten, in den Variablennamen und in den Operatoren).Die Ergebnisse zeigen, dass die Fehlererkennungsrate für Werte durchaus ausreichend ist, wohingegen die Fehlererkennungsraten für Variablennamen bzw. Operatoren weit unter den erwarteten Werten liegen. Diese Ergebnisse sind ein wichtiger Beitrag zur Diskussion, inwieweit MC/DC eine geeignete Metrik für sicherheitskritische Systeme ist.Safety-critical systems have to be tested exhaustively to ensure that there is no erroneous behavior, because failures may have serious impact. In relevant standards requirements for the testing process are defined, for instance, the required coverage metrics (like MC/DC) or "traceability", that means that the generated test cases have to map to the requirements originally defined in the system specification. There exist many test-case generation methods, but it is still a challenge to generate the test cases systematically (based on the requirements) and to guarantee that the resulting test set achieves full MC/DC on the system under test. The aim of this PhD thesis is to make a significant contribution to solve this problem.In this PhD thesis a testing framework is developed that provides a test-case generation method that is able to generate the test cases based on the requirements. The resulting test cases are traceable back to the system requirements. With the generated test set we achieve maximum possible MC/DC on the code of the SUT for a safety-critical application from the automotive domain. Furthermore we evaluate the actual error detection rate of the test set by defining three different error scenarios (errors in the value domain, errors in the variable domain, and errors in the operator domain).The results show that the error detection probability for the value domain is quite sufficient, whereas the error detection rates for the variable and operator domain are significantly less than expected. The results are important for the discussion about whether MC/DC is a suitable coverage metric for safety-critical systems.15

Error Detection Rate of MC/DC for a Case Study from the Automotive Domain

Chilenski and Miller [1] claim that the error detection probability of a test set with full modified condition/decision coverage (MC/DC) on the system under test converges to 100% for an increasing number of test cases, but there are also examples where the error detection probability of an MC/DC adequate test set is indeed zero. In this work we analyze the effective error detection rate of a test set that achieves maximum possible MC/DC on the code for a case study from the automotive domain. First we generate the test cases automatically with a model checker. Then we mutate the original program to generate three different error scenarios: the first error scenario focuses on errors in the value domain, the second error scenario focuses on errors in the domain of the variable names and the third error scenario focuses on errors within the operators of the boolean expressions in the decisions of the case study. Applying the test set to these mutated program versions shows that all errors of the values are detected, but the error detection rate for mutated variable names or mutated operators is quite disappointing (for our case study 22% of the mutated variable names, resp. 8% of the mutated operators are not detected by the original MC/DC test set). With this work we show that testing a system with a test set that achieves maximum possible MC/DC on the code detects less errors than expected

Applicability of Formal Methods for Safety-Critical Systems in the Context of ISO 26262

Formal methods are a means for verification and validation with the main advantage that a system property can be verified for the overall system (including all possible system states). The drawbacks of formal methods are the additional effort for the formalisation of the requirements and for building a model of the system, and, the limitations due to computational restrictions (handling the state-space explosion). ISO 26262 “Road Vehicles - Functional Safety” is a standard for the assessment of the development process for safety-relevant components in the automotive domain. The standard addresses formal methods for the specification of safety requirements and for the product development at software level. Formal methods for the hardware development or at system level are (by now) not explicitly foreseen by the standard. In this work we will give an overview on the basic principles and the state-of-the-art of formal methods (in detail, model checking). Then we will present different approaches for the application of formal methods at system level including some preliminary evaluation results for an industrial use case. Based on these experiences we will discuss the applicabi lity of formal methods in the context of ISO 26262 (i.e., for automotive components) in view of the limitations of formal techniques for applications inthe automotive domain