Validation and Verification of Safety-Critical Systems in Avionics

This research addresses the issues of safety-critical systems verification and validation. Safety-critical systems such as avionics systems are complex embedded systems. They are composed of several hardware and software components whose integration requires verification and testing in compliance with the Radio Technical Commission for Aeronautics standards and their supplements (RTCA DO-178C). Avionics software requires certification before its deployment into an aircraft system, and testing is mandatory for certification. Until now, the avionics industry has relied on expensive manual testing. The industry is searching for better (quicker and less costly) solutions. This research investigates formal verification and automatic test case generation approaches to enhance the quality of avionics software systems, ensure their conformity to the standard, and to provide artifacts that support their certification. The contributions of this thesis are in model-based automatic test case generations approaches that satisfy MC/DC criterion, and bidirectional requirement traceability between low-level requirements (LLRs) and test cases. In the first contribution, we integrate model-based verification of properties and automatic test case generation in a single framework. The system is modeled as an extended finite state machine model (EFSM) that supports both the verification of properties and automatic test case generation. The EFSM models the control and dataflow aspects of the system. For verification, we model the system and some properties and ensure that properties are correctly propagated to the implementation via mandatory testing. For testing, we extended an existing test case generation approach with MC/DC criterion to satisfy RTCA DO-178C requirements. Both local test cases for each component and global test cases for their integration are generated. The second contribution is a model checking-based approach for automatic test case generation. In the third contribution, we developed an EFSM-based approach that uses constraints solving to handle test case feasibility and addresses bidirectional requirements traceability between LLRs and test cases. Traceability elements are determined at a low-level of granularity, and then identified, linked to their source artifact, created, stored, and retrieved for several purposes. Requirements’ traceability has been extensively studied but not at the proposed low-level of granularity

Object Constraint Language Based Test Case Optimisation

Software testing, a pivotal phase in the Software Develop- ment Life Cycle (SDLC), ensures the correctness and perfor- mance of the corresponding software system. Model-based testing (MBT) is a method to validate whether the software system or specification satisfies the pre-defined requirements through design models. However, with modern systems ex- panding in complexity, testing has become a labour-intensive and unpredictable process within the SDLC. Therefore, many test case optimisation (TCO) techniques have been proposed to make the testing process more manageable. But these ap- proaches predominantly focus on code-based strategies, leav- ing systems expressed in Object Constraint Language (OCL) underserved. OCL is a part of the Unified Modeling Lan- guage (UML) standard and is a type of declarative language used to describe system specification by pre- and post- con- ditions. Initially, OCL has been proposed as a constraint language to add more details to the UML model, but along- side the development of OCL itself, there are more and more systems whose specifications are expressed in OCL.This thesis aims to systematically investigate the feasibil- ity of applying TCO techniques to the OCL-defined systems, with an emphasis on test case prioritisation (TCP) and test case minimisation (TCM) processes. A systematic literature review for the directly related topic, UML-based test case generation, is conducted in this thesis. Also, we adapted a set of test case optimisation algorithms and compared the performance between these algorithms under the context of OCL. Moreover, we modified one metric for the TCP evalu- ation process, which made the metric more suitable for the MBT and mutation testing environment. Furthermore, we introduce a full set of mutation operators and corresponding classifications to the OCL standard library, offering practical guidance for optimisation processes.The proposed TCO processes are validated and evalu- ated through four real-world systems expressed in OCL with different complexities. The experiment results demonstrate that for the TCM process, the size of the minimised test suite is reduced from 33.33% to 81.8% without losing any fault de- tection ability. For the TCP process, leveraging the modified evaluation metric, the improvements are up to 50%, indicat- ing that the prioritised test suite can detect system defects earlier when compared to the original one. Evaluating based on the considerations of effectiveness, efficiency and stability, we suggest the NSGA-II for the TCM process and the genetic algorithm for the TCP process. When combining TCP and TCM processes, the TCM process consistently increases the efficiency of the TCP process by reducing the search space for the prioritisation process