Dynamic assertion testing of flight control software

Digital Flight Control System (DFCS) software was used as a test case for assertion testing. The assertions were written and embedded in the code, then errors were inserted (seeded) one at a time and the code executed. Results indicate that assertion testing is an effective and efficient method of detecting errors in flight software. Most errors are eliminate at an earlier stage in the development than before

Development of a flight software testing methodology

The research to develop a testing methodology for flight software is described. An experiment was conducted in using assertions to dynamically test digital flight control software. The experiment showed that 87% of typical errors introduced into the program would be detected by assertions. Detailed analysis of the test data showed that the number of assertions needed to detect those errors could be reduced to a minimal set. The analysis also revealed that the most effective assertions tested program parameters that provided greater indirect (collateral) testing of other parameters. In addition, a prototype watchdog task system was built to evaluate the effectiveness of executing assertions in parallel by using the multitasking features of Ada

Formal Generation of Executable Assertions for Application-Oriented Fault Tolerance

Executable assertions embedded into a distributed computing system can provide run-time assurance by ensuring that the program state, in the actual run-time environment, is consistent with the logical stage specified in the assertions; if not, then an error has occurred and a reliable communication of this diagnostic information is provided to the system such that reconfiguration and recovery can take place. Application- oriented fault tolerance is a method that provides fault detection using executable assertions based on the natural constraints of the application. This paper focuses on giving application-oriented fault tolerance a theoretical foundation by providing a mathematical model for the generation of executable assertions which detect faults in the presence of arbitrary failures. The mathematical model of choice was axiomatic program verification. A method was developed that translates a concurrent verification proof outline into an error-detecting concurrent program. This paper shows the application of the developed method to several applications

A methodology for producing reliable software, volume 1

An investigation into the areas having an impact on producing reliable software including automated verification tools, software modeling, testing techniques, structured programming, and management techniques is presented. This final report contains the results of this investigation, analysis of each technique, and the definition of a methodology for producing reliable software

Ein verallgemeinerter Prozess zur Verifikation und Validerung von Modellen und Simulationsergebnissen

With technologies increasing rapidly, symbolic, quantitative modeling and computer-based simulation (M&S) have become affordable and easy-to-apply tools in numerous application areas as, e.g., supply chain management, pilot training, car safety improvement, design of industrial buildings, or theater-level war gaming. M&S help to reduce the resources required for many types of projects, accelerate the development of technical systems, and enable the control and management of systems of high complexity. However, as the impact of M&S on the real world grows, the danger of adverse effects of erroneous or unsuitable models or simu-lation results also increases. These effects may range from the delayed delivery of an item ordered by mail to hundreds of avoidable casualties caused by the simulation-based acquisi-tion (SBA) of a malfunctioning communication system for rescue teams. In order to benefit from advancing M&S, countermeasures against M&S disadvantages and drawbacks must be taken. Verification and Validation (V&V) of models and simulation results are intended to ensure that only correct and suitable models and simulation results are used. However, during the development of any technical system including models for simulation, numerous errors may occur. The later they are detected, and the further they have propagated through the model development process, the more resources they require to correct thus, their propaga-tion should be avoided. If the errors remain undetected, and major decisions are based on in-correct or unsuitable models or simulation results, no benefit is gained from M&S, but a dis-advantage. This thesis proposes a structured and rigorous approach to support the verification and valida-tion of models and simulation results by a) the identification of the most significant of the current deficiencies of model develop-ment (design and implementation) and use, including the need for more meaningful model documentation and the lack of quality assurance (QA) as an integral part of the model development process; b) giving an overview of current quality assurance measures in M&S and in related areas. The transferability of concepts like the capability maturity model for software (SW-CMM) and the ISO9000 standard is discussed, and potentials and limits of documents such as the VV&A Recommended Practices Guide of the US Defense Modeling and Simulation Office are identified; c) analysis of quality assurance measures and so called V&V techniques for similarities and differences, to amplify their strengths and to reduce their weaknesses. d) identification and discussion of influences that drive the required rigor and intensity of V&V measures (risk involved in using models and simulation results) on the one hand, and that limit the maximum reliability of V&V activities (knowledge about both the real system and the model) on the other. This finally leads to the specification of a generalized V&V process - the V&V Triangle. It illustrates the dependencies between numerous V&V objectives, which are derived from spe-cific potential errors that occur during model development, and provides guidance for achiev-ing these objectives by the association of V&V techniques, required input, and evidence made available. The V&V Triangle is applied to an M&S sample project, and the lessons learned from evaluating the results lead to the formulation of future research objectives in M&S V&V

A methodology for the generation of efficient error detection mechanisms

A dependable software system must contain error detection mechanisms and error recovery mechanisms. Software components for the detection of errors are typically designed based on a system specification or the experience of software engineers, with their efficiency typically being measured using fault injection and metrics such as coverage and latency. In this paper, we introduce a methodology for the design of highly efficient error detection mechanisms. The proposed methodology combines fault injection analysis and data mining techniques in order to generate predicates for efficient error detection mechanisms. The results presented demonstrate the viability of the methodology as an approach for the development of efficient error detection mechanisms, as the predicates generated yield a true positive rate of almost 100% and a false positive rate very close to 0% for the detection of failure-inducing states. The main advantage of the proposed methodology over current state-of-the-art approaches is that efficient detectors are obtained by design, rather than by using specification-based detector design or the experience of software engineers

Testing abstract behavioral specifications

We present a range of testing techniques for the Abstract Behavioral Specification (ABS) language and apply them to an industrial case study. ABS is a formal modeling language for highly variable, concurrent, component-based systems. The nature of these systems makes them susceptible to the introduction of subtle bugs that are hard to detect in the presence of steady adaptation. While static analysis techniques are available for an abstract language such as ABS, testing is still indispensable and complements analytic methods. We focus on fully automated testing techniques including black-box and glass-box test generation as well as runtime assertion checking, which are shown to be effective in an industrial setting

Testing abstract behavioral specifications

We present a range of testing techniques for the Abstract Behavioral Specification (ABS) language and apply them to an industrial case study. ABS is a formal modeling language for highly variable, concurrent, component-based systems. The nature of these systems makes them susceptible to the introduction of subtle bugs that are hard to detect in the presence of steady adaptation. While static analysis techniques are available for an abstract language such as ABS, testing is still indispensable and complements analytic methods. We focus on fully automated testing techniques including blackbox and glassbox test generation as well as runtime assertion checking, which are shown to be effective in an industrial setting

CTGEN - a Unit Test Generator for C

We present a new unit test generator for C code, CTGEN. It generates test data for C1 structural coverage and functional coverage based on pre-/post-condition specifications or internal assertions. The generator supports automated stub generation, and data to be returned by the stub to the unit under test (UUT) may be specified by means of constraints. The typical application field for CTGEN is embedded systems testing; therefore the tool can cope with the typical aliasing problems present in low-level C, including pointer arithmetics, structures and unions. CTGEN creates complete test procedures which are ready to be compiled and run against the UUT. In this paper we describe the main features of CTGEN, their technical realisation, and we elaborate on its performance in comparison to a list of competing test generation tools. Since 2011, CTGEN is used in industrial scale test campaigns for embedded systems code in the automotive domain.Comment: In Proceedings SSV 2012, arXiv:1211.587