Automated Testing: Requirements Propagation via Model Transformation in Embedded Software

Testing is the most common activity to validate software systems and plays a key role in the software development process. In general, the software testing phase takes around 40-70% of the effort, time and cost. This area has been well researched over a long period of time. Unfortunately, while many researchers have found methods of reducing time and cost during the testing process, there are still a number of important related issues such as generating test cases from UCM scenarios and validate them need to be researched. As a result, ensuring that an embedded software behaves correctly is non-trivial, especially when testing with limited resources and seeking compliance with safety-critical software standard. It thus becomes imperative to adopt an approach or methodology based on tools and best engineering practices to improve the testing process. This research addresses the problem of testing embedded software with limited resources by the following. First, a reverse-engineering technique is exercised on legacy software tests aims to discover feasible transformation from test layer to test requirement layer. The feasibility of transforming the legacy test cases into an abstract model is shown, along with a forward engineering process to regenerate the test cases in selected test language. Second, a new model-driven testing technique based on different granularity level (MDTGL) to generate test cases is introduced. The new approach uses models in order to manage the complexity of the system under test (SUT). Automatic model transformation is applied to automate test case development which is a tedious, error-prone, and recurrent software development task. Third, the model transformations that automated the development of test cases in the MDTGL methodology are validated in comparison with industrial testing process using embedded software specification. To enable the validation, a set of timed and functional requirement is introduced. Two case studies are run on an embedded system to generate test cases. The effectiveness of two testing approaches are determined and contrasted according to the generation of test cases and the correctness of the generated workflow. Compared to several techniques, our new approach generated useful and effective test cases with much less resources in terms of time and labor work. Finally, to enhance the applicability of MDTGL, the methodology is extended with the creation of a trace model that records traceability links among generated testing artifacts. The traceability links, often mandated by software development standards, enable the support for visualizing traceability, model-based coverage analysis and result evaluation

Simulation combined model-based testing method for train control systems

A Train Control System (TCS) is utilised to guard the operational safety of the trains in railway systems. Therefore, functional testing is applied to verify consistency between the TCS and specification requirements. Traditional functional testing in TCSs is mainly based on manually designed test cases, which is becoming unsuitable for testing increasingly complex TCSs. Therefore, Model-Based Testing (MBT) methods have been introduced into TCS functional testing, to improve the efficiency and coverage of TCS testing, with application difficulties. To overcome the difficulties of applying MBT methods to test TCSs, the author introduces simulation combined MBT which combines an MBT method with simulation. Modelling method and implementation method for the proposed approach were explained in detail. Two case studies were undertaken to explore the effectiveness of the testing platform developed. The testing results obtained prove that the testing platform can be utilised to implement the functional testing of TCSs. To prove that the MBT platform is effective in detecting errors in the SUT, validation and verification was undertaken, which include validation of specification requirements and verification of the MBT platform. The testing performance is proven to be better than existing MBT methods in terms of coverage and efficiency

Is CADP an Applicable Formal Method?

International audienceCADP is a comprehensive toolbox implementing results of concurrency theory. This paper addresses the question, whether CADP qualifies as an applicable formal method, based on the experience of the authors and feedback reported by users

Verifizierbare Entwicklung eines satellitenbasierten Zugsicherungssystems mit Petrinetzen

Nowadays model-based techniques are widely used in system design and development, especially for safety-critical systems such as train control systems. Given a design model, executable codes could be generated automatically from the model following certain transformation rules. A high-quality model of a system provides a good understanding, a favourable structure, a reasonable scale and abstraction level as well as realistic behaviours with respect to the concurrent operation of independent subsystems. Motivated by this principle, a first Coloured Petri Net (CPN) model of a satellite-based train control system (SatZB) with the capability of continuous simulation is developed employing the BASYSNET method which adopts Petri nets as the means of description during the whole development process. After establishing the system model, the verification tasks are identified based on the hazard analysis of the train control system. To verify the identified tasks for quality assurance, verification by means of simulation, formal analysis and testing is carried out considering the four representing system properties: function, state, structure and behaviour. For structural analysis, the concept of open nets is proposed to check the reproducibility of empty markings of scenario nets, the existence of dead transitions in the scenario nets, and the terminating states of the scenario nets. The system behaviour, in which states are involved, is investigated by reachability analysis. Unlike the conventional method of reachability analysis by calculating the state space of the Petri net, techniques based on Petri net unfoldings are introduced in this thesis. As to the functional verification, two model-based test generation techniques, i.e., CPN-based and SPENAT (Safe Place Transition Nets with Attributes)-based techniques, are presented. In this thesis, the proposed methods are exemplified by the application to the on-board module of SatZB model. According to the verification results, no errors were found in the module. Therefore, the confidence in the quality of the on-board module has been significantly increased.Heutzutage werden in zahlreichen Anwendungen modellbasierte Techniken zur Systementwicklung, insbesondere für sicherheitskritische Systeme wie Eisenbahnleit- und -sicherungssysteme, verwendet. Aus einem Design Modell kann dabei ausführbarer Code automatisch nach bestimmten Transformationsregeln generiert werden. Ein hochwertiges Modell des Systems bietet für die Entwicklung ein gutes Verständnis, eine günstige Struktur, eine angemessene Größenordnung und Abstraktionsebene als auch realistische Verhaltensweisen in Bezug auf den gleichzeitigen Betrieb von unabhängigen Subsystemen. Motiviert von dieses Prinzip wird ein erstes Farbige Petri-Netz (CPN)-Modell eines satellitenbasierten Zugsicherungssystem (SatZB) unter Verwendung der BASYSNET Methode entwickelt, der Petri-Netze als Beschreibungsmittel während des gesamten Entwicklungsprozesses nutzt. Dieses Modell bietet die Möglichkeit zur kontinuierlichen Simulation des Systemverhaltens. Nach der Erstellung des Systemmodells werden die Verifikationsaufgaben auf der Grundlage der Gefährdungsanalyse des Zugsicherungssystems identifiziert. Die abgeleiteten Bedingungen werden zur Qualitätssicherung durch Simulation, formale Analysen und Tests unter Berücksichtigung der vier Systemeigenschaften (Funktion, Zustand, Struktur und Verhalten) verifiziert. Für die Strukturanalyse wird das Konzept der offenen Netzen vorgeschlagen, um die Reproduzierbarkeit der leeren Markierungen der Szenario-Netze, die Existenz der Toten Transitionen in den Szenario-Netze, und die Abschluss Zustände der Szenario-Netze zu prüfen. Das Systemverhalten wird dabei durch Zustände beschrieben und durch eine Erreichbarkeitsanalyse untersucht. Im Gegensatz zu der konventionellen Methode, welche die Erreichbarkeit durch die Berechnung des Zustandsraums des Petri-Netzes analysiert, werden in dieser Arbeit Techniken auf Basis von Petri-Netz-Entfaltung eingeführt. Für die funktionale Verifikation werden zwei modellbasierte Testgenerierungstechniken, eine CPN-basierte und eine SPENAT (Sicheres Petrinetz mit Attributen)-basierte, vorgestellt. In dieser Arbeit werden die vorgeschlagenen Methoden durch die Anwendung auf das On-Board-Modul des SatZB-Modells veranschaulicht. Dabei wurden nach dem Abschluss der Prüfungen keine Fehler im Modul gefunden, wodurch das Vertrauen in die Qualität des On-Board-Moduls deutlich erhöht wurde

Re-use of tests and arguments for assesing dependable mixed-critically systems

The safety assessment of mixed-criticality systems (MCS) is a challenging activity due to system heterogeneity, design constraints and increasing complexity. The foundation for MCSs is the integrated architecture paradigm, where a compact hardware comprises multiple execution platforms and communication interfaces to implement concurrent functions with different safety requirements. Besides a computing platform providing adequate isolation and fault tolerance mechanism, the development of an MCS application shall also comply with the guidelines defined by the safety standards. A way to lower the overall MCS certification cost is to adopt a platform-based design (PBD) development approach. PBD is a model-based development (MBD) approach, where separate models of logic, hardware and deployment support the analysis of the resulting system properties and behaviour. The PBD development of MCSs benefits from a composition of modular safety properties (e.g. modular safety cases), which support the derivation of mixed-criticality product lines. The validation and verification (V&V) activities claim a substantial effort during the development of programmable electronics for safety-critical applications. As for the MCS dependability assessment, the purpose of the V&V is to provide evidences supporting the safety claims. The model-based development of MCSs adds more V&V tasks, because additional analysis (e.g., simulations) need to be carried out during the design phase. During the MCS integration phase, typically hardware-in-the-loop (HiL) plant simulators support the V&V campaigns, where test automation and fault-injection are the key to test repeatability and thorough exercise of the safety mechanisms. This dissertation proposes several V&V artefacts re-use strategies to perform an early verification at system level for a distributed MCS, artefacts that later would be reused up to the final stages in the development process: a test code re-use to verify the fault-tolerance mechanisms on a functional model of the system combined with a non-intrusive software fault-injection, a model to X-in-the-loop (XiL) and code-to-XiL re-use to provide models of the plant and distributed embedded nodes suited to the HiL simulator, and finally, an argumentation framework to support the automated composition and staged completion of modular safety-cases for dependability assessment, in the context of the platform-based development of mixed-criticality systems relying on the DREAMS harmonized platform.La dificultad para evaluar la seguridad de los sistemas de criticidad mixta (SCM) aumenta con la heterogeneidad del sistema, las restricciones de diseño y una complejidad creciente. Los SCM adoptan el paradigma de arquitectura integrada, donde un hardware embebido compacto comprende múltiples plataformas de ejecución e interfaces de comunicación para implementar funciones concurrentes y con diferentes requisitos de seguridad. Además de una plataforma de computación que provea un aislamiento y mecanismos de tolerancia a fallos adecuados, el desarrollo de una aplicación SCM además debe cumplir con las directrices definidas por las normas de seguridad. Una forma de reducir el coste global de la certificación de un SCM es adoptar un enfoque de desarrollo basado en plataforma (DBP). DBP es un enfoque de desarrollo basado en modelos (DBM), en el que modelos separados de lógica, hardware y despliegue soportan el análisis de las propiedades y el comportamiento emergente del sistema diseñado. El desarrollo DBP de SCMs se beneficia de una composición modular de propiedades de seguridad (por ejemplo, casos de seguridad modulares), que facilitan la definición de líneas de productos de criticidad mixta. Las actividades de verificación y validación (V&V) representan un esfuerzo sustancial durante el desarrollo de aplicaciones basadas en electrónica confiable. En la evaluación de la seguridad de un SCM el propósito de las actividades de V&V es obtener las evidencias que apoyen las aseveraciones de seguridad. El desarrollo basado en modelos de un SCM incrementa las tareas de V&V, porque permite realizar análisis adicionales (por ejemplo, simulaciones) durante la fase de diseño. En las campañas de pruebas de integración de un SCM habitualmente se emplean simuladores de planta hardware-in-the-loop (HiL), en donde la automatización de pruebas y la inyección de faltas son la clave para la repetitividad de las pruebas y para ejercitar completamente los mecanismos de tolerancia a fallos. Esta tesis propone diversas estrategias de reutilización de artefactos de V&V para la verificación temprana de un MCS distribuido, artefactos que se emplearán en ulteriores fases del desarrollo: la reutilización de código de prueba para verificar los mecanismos de tolerancia a fallos sobre un modelo funcional del sistema combinado con una inyección de fallos de software no intrusiva, la reutilización de modelo a X-in-the-loop (XiL) y código a XiL para obtener modelos de planta y nodos distribuidos aptos para el simulador HiL y, finalmente, un marco de argumentación para la composición automatizada y la compleción escalonada de casos de seguridad modulares, en el contexto del desarrollo basado en plataformas de sistemas de criticidad mixta empleando la plataforma armonizada DREAMS.Kritikotasun nahastuko sistemen segurtasun ebaluazioa jarduera neketsua da beraien heterogeneotasuna dela eta. Sistema hauen oinarria arkitektura integratuen paradigman datza, non hardware konpaktu batek exekuzio plataforma eta komunikazio interfaze ugari integratu ahal dituen segurtasun baldintza desberdineko funtzio konkurrenteak inplementatzeko. Konputazio plataformek isolamendu eta akatsen aurkako mekanismo egokiak emateaz gain, segurtasun arauek definituriko jarraibideak jarraitu behar dituzte kritikotasun mistodun aplikazioen garapenean. Sistema hauen zertifikazio prozesuaren kostua murrizteko aukera bat plataformetan oinarritutako garapenean (PBD) datza. Garapen planteamendu hau modeloetan oinarrituriko garapena da (MBD) non modeloaren logika, hardware eta garapen desberdinak sistemaren propietateen eta portaeraren aurka aztertzen diren. Kritikotasun mistodun sistemen PBD garapenak etekina ateratzen dio moduluetan oinarrituriko segurtasun propietateei, adibidez: segurtasun kasu modularrak (MSC). Modulu hauek kritikotasun mistodun produktu-lerroak ere hartzen dituzte kontutan. Berifikazio eta balioztatze (V&V) jarduerek esfortzu kontsideragarria eskatzen dute segurtasun-kiritikoetarako elektronika programagarrien garapenean. Kritikotasun mistodun sistemen konfiantzaren ebaluazioaren eta V&V jardueren helburua segurtasun eskariak jasotzen dituzten frogak proportzionatzea da. Kritikotasun mistodun sistemen modelo bidezko garapenek zeregin gehigarriak atxikitzen dizkio V&V jarduerari, fase honetan analisi gehigarriak (hots, simulazioak) zehazten direlako. Bestalde, kritikotasun mistodun sistemen integrazio fasean, hardware-in-the-loop (Hil) simulazio plantek V&V iniziatibak sostengatzen dituzte non testen automatizazioan eta akatsen txertaketan funtsezko jarduerak diren. Jarduera hauek frogen errepikapena eta segurtasun mekanismoak egiaztzea ahalbidetzen dute. Tesi honek V&V artefaktuen berrerabilpenerako estrategiak proposatzen ditu, kritikotasun mistodun sistemen egiaztatze azkarrerako sistema mailan eta garapen prozesuko azken faseetaraino erabili daitezkeenak. Esate baterako, test kodearen berrabilpena akats aurkako mekanismoak egiaztatzeko, modelotik X-in-the-loop (XiL)-ra eta kodetik XiL-rako konbertsioa HiL simulaziorako eta argumentazio egitura bat DREAMS Europear proiektuan definituriko arkitektura estiloan oinarrituriko segurtasun kasu modularrak automatikoki eta gradualki sortzeko

Towards a Model-Centric Software Testing Life Cycle for Early and Consistent Testing Activities

The constant improvement of the available computing power nowadays enables the accomplishment of more and more complex tasks. The resulting implicit increase in the complexity of hardware and software solutions for realizing the desired functionality requires a constant improvement of the development methods used. On the one hand over the last decades the percentage of agile development practices, as well as testdriven development increases. On the other hand, this trend results in the need to reduce the complexity with suitable methods. At this point, the concept of abstraction comes into play, which manifests itself in model-based approaches such as MDSD or MBT. The thesis is motivated by the fact that the earliest possible detection and elimination of faults has a significant influence on product costs. Therefore, a holistic approach is developed in the context of model-driven development, which allows applying testing already in early phases and especially on the model artifacts, i.e. it provides a shift left of the testing activities. To comprehensively address the complexity problem, a modelcentric software testing life cycle is developed that maps the process steps and artifacts of classical testing to the model-level. Therefore, the conceptual basis is first created by putting the available model artifacts of all domains into context. In particular, structural mappings are specified across the included domain-specific model artifacts to establish a sufficient basis for all the process steps of the life cycle. Besides, a flexible metamodel including operational semantics is developed, which enables experts to carry out an abstract test execution on the modellevel. Based on this, approaches for test case management, automated test case generation, evaluation of test cases, and quality verification of test cases are developed. In the context of test case management, a mechanism is realized that enables the selection, prioritization, and reduction of Test Model artifacts usable for test case generation. I.e. a targeted set of test cases is generated satisfying quality criteria like coverage at the model-level. These quality requirements are accomplished by using a mutation-based analysis of the identified test cases, which builds on the model basis. As the last step of the model-centered software testing life cycle two approaches are presented, allowing an abstract execution of the test cases in the model context through structural analysis and a form of model interpretation concerning data flow information. All the approaches for accomplishing the problem are placed in the context of related work, as well as examined for their feasibility by of a prototypical implementation within the Architecture And Analysis Framework. Subsequently, the described approaches and their concepts are evaluated by qualitative as well as quantitative evaluation. Moreover, case studies show the practical applicability of the approach

A Model-driven Approach for the Automatic Generation of System-Level Test Cases

Systems at the basis of the modern society, as the as the homeland security, the environment protection, the public and private transportations, the healthcare or the energy supply depend on the correct functioning of one or more embedded systems. In several cases, such systems shall be considered critical, since the consequences of their failures may result in economic losses, damages to the environment or even injuries to human life. Possible disastrous consequences of embedded critical systems, suggest that discover flaws during systems development and avoid their propagation to the system execution, is a crucial task. In fact, most of the failures found during the usage of embedded critical systems, is due to errors introduced during early stages of the system development. Thus, it is desiderable to start Verification and Validation (V&V) activities during early stages of a system life cycle. However such V&V activities can account over the 50% of times and costs of a system life cycle and there is therefore the need to introduce techniques able to reduce the accounted resources without losses in term efficiency. Among the methodologies found in scientific and industrial literature there is a large interest in the V&V automation. In particular, automatic verification can be performed during different stages of a system development life cycle and can assume different meanings. In this thesis, the focus is on the automation of the test cases generation phase performed at the System level starting from SUT and test specifications. A recent research trend, related to this, is to support such process providing a flexible tool chain allowing for effective Model Driven Engineering (MDE) approaches. The adoption of a model-driven techniques requires the modelling of the SUT to drive the generation process, by using suitable domain-specific modelling languages and model transformations. Thus, a successful application of the MDE principles is related to the choice of the high-level language for SUT specification and the tools and techniques provided to support the V\&V processes. According to this, the model-driven approach define in this thesis relies on three key factors: (1) the definition of new domain-specific modelling languages (DSMLs) for the SUT and the test specifications, (2) the adoption of model checking techniques to realize the generation of the test cases and (3) the implementation of a concrete framework providing a complete tool chain supporting the automation process. This work is partially involved in an ARTEMIS European project CRYSTAL (CRitical sYSTem engineering AcceLeration). CRYSTAL is strongly industry-oriented and aims at achieving technical innovation by a user-driven approach based on the idea to apply engineering methods to industrially relevant Use Cases from the automotive, aerospace, rail and health-care sectors. The DSML that will be presented in this thesis, emerged as an attempt to address the modelling requirements and the design practices of the industrial partners of the project, within a rigorous and well-founded formal specification and verification approach. In fact, the main requirement that a modelling language suitable for the industry should have is to be small and as simple as possible. Thus, the modelling language should provide an adequate set of primitive constructs to allow for a natural modelling of the system of interest. Furthermore, the larger the gap between the design specification and the actual implementation is, the less useful the results of the design analysis would be. The test case generation is supported by model checking techniques; the SUT and test models are in fact translated in specifications expressed by the language adopted by a model checker. The thesis discusses all the issues addressed in the mapping process and provides their implementations by means of model transformations. A class of test specifications is addressed to exemplify the generation process over a common class of reachability requirements. The model-driven approach discussed in the thesis is applied in the contest of the railway control systems, and in particular on some of the key functionalities of the Radio Block Center, the main component of the ERTMS/ETCS standards for the interoperability of the railway control systems in the European Community. The thesis is organized as follows. The first chapter introduces embedded critical systems and outlines the main research trends related to their V&V process. The Chapter 2 outlines the state of the art in testing automation with a particular focus on model-driven approaches for automatic test generation. The same Chapter 2 provides also the necessary technical background supporting to understand the development process of the supporting framework. The Chapter 3 describes the context of the CRYSTAL project and the proposed model-driven approach partially involved in its activities. The Chapter 4 describes the domains pecific modelling languages defined for the modelling of the SUT specifications and of the test generation outcomes. Moreover the guidelines defined for modelling test specifications are discussed. The Chapter 5 focuses on the mapping process that enable the translation of the high-level language for the modelling of the SUT specification to the language adopted by the chosen model checker. The implementation of the overall framework is addressed in Chapter 6. Here model transformations realizing the defined mappings and the architecture of the Test Case Generator (TCG) framework are described and discussed. The Chapter 7 shows the results of the application of the approach in the context of the railway control systems and in particular to the Radio Block Centre system, a key component in the ERTMS/ETCS standard. Chapter 8 end the thesis, giving some conclusive remarks

Certifications of Critical Systems – The CECRIS Experience

In recent years, a considerable amount of effort has been devoted, both in industry and academia, to the development, validation and verification of critical systems, i.e. those systems whose malfunctions or failures reach a critical level both in terms of risks to human life as well as having a large economic impact.Certifications of Critical Systems – The CECRIS Experience documents the main insights on Cost Effective Verification and Validation processes that were gained during work in the European Research Project CECRIS (acronym for Certification of Critical Systems). The objective of the research was to tackle the challenges of certification by focusing on those aspects that turn out to be more difficult/important for current and future critical systems industry: the effective use of methodologies, processes and tools.The CECRIS project took a step forward in the growing field of development, verification and validation and certification of critical systems. It focused on the more difficult/important aspects of critical system development, verification and validation and certification process. Starting from both the scientific and industrial state of the art methodologies for system development and the impact of their usage on the verification and validation and certification of critical systems, the project aimed at developing strategies and techniques supported by automatic or semi-automatic tools and methods for these activities, setting guidelines to support engineers during the planning of the verification and validation phases