Development of a framework for automated systematic testing of safety-critical embedded systems

“This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder." “Copyright IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.”In this paper we introduce the development of a framework for testing safety-critical embedded systems based on the concepts of model-based testing. In model-based testing the test cases are derived from a model of the system under test. In our approach the model is an automaton model that is automatically extracted from the C-source code of the system under test. Beside random test data generation the test case generation uses formal methods, in detail model checking techniques. To find appropriate test cases we use the requirements defined in the system specification. To cover further execution paths we developed an additional, to our best knowledge, novel method based on special structural coverage criteria. We present preliminary results on the model extraction using a concrete industrial case study from the automotive domain

Functional Requirements-Based Automated Testing for Avionics

We propose and demonstrate a method for the reduction of testing effort in safety-critical software development using DO-178 guidance. We achieve this through the application of Bounded Model Checking (BMC) to formal low-level requirements, in order to generate tests automatically that are good enough to replace existing labor-intensive test writing procedures while maintaining independence from implementation artefacts. Given that existing manual processes are often empirical and subjective, we begin by formally defining a metric, which extends recognized best practice from code coverage analysis strategies to generate tests that adequately cover the requirements. We then formulate the automated test generation procedure and apply its prototype in case studies with industrial partners. In review, the method developed here is demonstrated to significantly reduce the human effort for the qualification of software products under DO-178 guidance

Predicate Abstraction with Under-approximation Refinement

We propose an abstraction-based model checking method which relies on refinement of an under-approximation of the feasible behaviors of the system under analysis. The method preserves errors to safety properties, since all analyzed behaviors are feasible by definition. The method does not require an abstract transition relation to be generated, but instead executes the concrete transitions while storing abstract versions of the concrete states, as specified by a set of abstraction predicates. For each explored transition the method checks, with the help of a theorem prover, whether there is any loss of precision introduced by abstraction. The results of these checks are used to decide termination or to refine the abstraction by generating new abstraction predicates. If the (possibly infinite) concrete system under analysis has a finite bisimulation quotient, then the method is guaranteed to eventually explore an equivalent finite bisimilar structure. We illustrate the application of the approach for checking concurrent programs.Comment: 22 pages, 3 figures, accepted for publication in Logical Methods in Computer Science journal (special issue CAV 2005

A Semantic Framework for Test Coverage

Since testing is inherently incomplete, test selection is of vital importance. Coverage measures evaluate the quality of a test suite and help the tester select test cases with maximal impact at minimum cost. Existing coverage criteria for test suites are usually defined in terms of syntactic characteristics of the implementation under test or its specification. Typical black-box coverage metrics are state and transition coverage of the specification. White-box testing often considers statement, condition and path coverage. A disadvantage of this syntactic approach is that different coverage figures are assigned to systems that are behaviorally equivalent, but syntactically different. Moreover, those coverage metrics do not take into account that certain failures are more severe than others, and that more testing effort should be devoted to uncover the most important bugs, while less critical system parts can be tested less thoroughly. This paper introduces a semantic approach to test coverage. Our starting point is a weighted fault model, which assigns a weight to each potential error in an implementation. We define a framework to express coverage measures that express how well a test suite covers such a specification, taking into account the error weight. Since our notions are semantic, they are insensitive to replacing a specification by one with equivalent behaviour.We present several algorithms that, given a certain minimality criterion, compute a minimal test suite with maximal coverage. These algorithms work on a syntactic representation of weighted fault models as fault automata. They are based on existing and novel optimization\ud problems. Finally, we illustrate our approach by analyzing and comparing a number of test suites for a chat protocol

Associer des techniques de preuve et de résolution de contraintes pour la construction d'abstractions

National audienceCet article présente une méthode de génération assistée de tests. Elle applique des critères dynamiques de sélection des tests (TP) sur un modèle formel comportemental (M) utilisé auparavant, par exemple par LTG, pour générer des tests fonctionnels à partir de critères statiques de sélection. On peut appliquer à M un critère dynamique de sélection TP mais ceci nécessite de représenterM par un automate. Pour des applications réelles, sa taille en nombre d'états et de transitions est beaucoup trop grande (voir infinie) pour être utilisable. Nous proposons une méthode pour extraire une abstraction de M à partir d'un objectif de test TP. Nous effectuons un produit synchronisé de cette abstraction avec TP afin de cibler les exécutions du système sous test qui satisfont TP. Puis nous générons des tests abstraits symboliques à partir de ce modèle réduit en appliquant les critères de couverture tous les états ou toutes les transitions. Cet ensemble de tests est valué à partir de M, concrétisé puis exécuté sur l'implémentation sous test. Cette méthode est proposée pour compléter la méthode BZ-TT de génération de tests à partir de critères statiques de sélection. L'utilisateur obtient des tests complémentaires en fournissant un critère dynamique de sélection. La méthode réutilise M, la couche de concrétisation des tests et l'infrastructure d'exécution des tests. L'originalité de l'approche est de construire une abstraction du modèle issue automatiquement de l'analyse statique d'un objectif de test formalisant des besoins de test d'une propriété dynamique du système

Association of Under-Approximation Techniques for Generating Tests from Models

International audienceIn this paper we present a Model-Based Testing approach with which we generate tests from an abstraction of a source behavioural model. We show a new algorithm that computes the abstraction as an under-approximation of the source model. Our first contribution is to combine two previous approaches proposed by Ball and Pasareanu et al. to compute May, Must+ and Must- abstract transition relations. Prooftechniques are used to compute these transition relations. The tests obtained by covering the abstract transitions have to be instantiated from the source model. So, following Pasareanu et al., our algorithm additionally computes a concrete transition relation: the tests obtained as sequences of concrete transitions need not be instantiated from the source model. Another contribution is to propose a choice of relevant paramaters and heuristics to pilot the tests computation. We experiment our approach and compare it with a previous approach of ours to compute tests from an abstraction that over-approximates the source model

Help, help, Im being suppressed the significance of suppressors in software testing

pre-printAbstract-Test features are basic compositional units used to describe what a test does (and does not) involve. For example, in API-based testing, the most obvious features are function calls; in grammar-based testing, the obvious features are the elements of the grammar. The relationship between features as abstractions of tests and produced behaviors of the tested program is surprisingly poorly understood. This paper shows how large-scale random testing modified to use diverse feature sets can uncover causal relationships between what a test contains and what the program being tested does. We introduce a general notion of observable behaviors as targets, where a target can be a detected fault, an executed branch or statement, or a complex coverage entity such as a state, predicate-valuation, or program path. While it is obvious that targets have triggers - features without which they cannot be hit by a test - the notion of suppressors - features which make a test less likely to hit a target - has received little attention despite having important implications for automated test generation and program understanding. For a set of subjects including C compilers, a flash file system, and JavaScript engines, we show that suppression is both common and important