Development of a framework for automated systematic testing of safety-critical embedded systems

“This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder." “Copyright IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.”In this paper we introduce the development of a framework for testing safety-critical embedded systems based on the concepts of model-based testing. In model-based testing the test cases are derived from a model of the system under test. In our approach the model is an automaton model that is automatically extracted from the C-source code of the system under test. Beside random test data generation the test case generation uses formal methods, in detail model checking techniques. To find appropriate test cases we use the requirements defined in the system specification. To cover further execution paths we developed an additional, to our best knowledge, novel method based on special structural coverage criteria. We present preliminary results on the model extraction using a concrete industrial case study from the automotive domain

JWalk: a tool for lazy, systematic testing of java classes by design introspection and user interaction

Popular software testing tools, such as JUnit, allow frequent retesting of modified code; yet the manually created test scripts are often seriously incomplete. A unit-testing tool called JWalk has therefore been developed to address the need for systematic unit testing within the context of agile methods. The tool operates directly on the compiled code for Java classes and uses a new lazy method for inducing the changing design of a class on the fly. This is achieved partly through introspection, using Java’s reflection capability, and partly through interaction with the user, constructing and saving test oracles on the fly. Predictive rules reduce the number of oracle values that must be confirmed by the tester. Without human intervention, JWalk performs bounded exhaustive exploration of the class’s method protocols and may be directed to explore the space of algebraic constructions, or the intended design state-space of the tested class. With some human interaction, JWalk performs up to the equivalent of fully automated state-based testing, from a specification that was acquired incrementally

Automated Testcase Generation for Numerical Support Functions in Embedded Systems

We present a tool for the automatic generation of test stimuli for small numerical support functions, e.g., code for trigonometric functions, quaternions, filters, or table lookup. Our tool is based on KLEE to produce a set of test stimuli for full path coverage. We use a method of iterative deepening over abstractions to deal with floating-point values. During actual testing the stimuli exercise the code against a reference implementation. We illustrate our approach with results of experiments with low-level trigonometric functions, interpolation routines, and mathematical support functions from an open source UAS autopilot

Use of domain-specific language in test automation

The primary aim of this research project was to investigate techniques to replace the complicated process of testing embedded systems in automotive domain. The multi-component domain was composed of different hardware to be used in testing procedure which increased the level of difficulty in testing for an operator. As a result, an existing semi-automated testing procedure was replaced by more simpler and efficient framework (ViBATA). A key step taken in this scenario was the replacement of manual GUI interface with the scriptable one to enhance the automation. This was achieved by building a Domain-specific language which allowed test definition in the form of human readable scripts which could be stored for later use. A DSL is a scripting language defined for a particular domain with compact expressiveness. In this case the domain is testing embedded systems in general and automotive systems in particular. The final product was a test case specification document in the form of XML as an output of generated code from this DSL which will be input to ViBATA to make test specification component automated. In this research a comparative analysis of existing DSLs for alternative domains and investigation of their applicability to the presented domain was also performed. The technologies used in this project are Xtext to define the DSL grammar, Xtend to generate code in Java and Simple framework to generate output in XML. The stages involved in DSL development and how these stages were implemented is covered in this thesis. The developed DSL for this domain is tested for automotive and calculator systems in this thesis which proved that this is more general and flexible. The DSL is consistent, efficient and automated test specification component of testing framework in embedded systems

Linking Unit Tests and Properties

QuickCheck allows us to verify software against particular proper- ties. A property can be regarded as an abstraction over many unit tests. QuickCheck uses generated random input data to test such properties. If a counterexample is found, it becomes immediately clear what we have tested. This is not the case when all tests pass, since we do not (and shall not) see the actual generated test cases. How can we be sure about what is tested? QuickCheck has the ability to gather statistics about the test cases, which is insightful. But still it does not tell us whether the particular unit test scenarios we have in mind are included. For this reason, we have developed a tool that can answer this question. It checks if a given unit test can be generated by a property, making it easier to judge the property’s quality. We have applied our tool to an industrial use case of testing the AUTOSAR basic software modules and shows that it can handle complex models and large unit tests