ISO 26262 Compliant Automatic Requirements-Based Testing for TargetLink

International audienceThis paper presents an automatic method that has been developed in order to support international standards regarding functional safety, like ISO 26262 for automotive and DO178B for aeronautics. It describes a seamless and integrated method to formalise requirements based on pattern specification automatons and generated C-observer code. Based on such C-Observers then requirements based functional tests can be generated and formal verification can be automated as the generated C-code observers are integrated into a test and verification tool environment. The advantage of such approach includes the possibility to enable requirements-based test case generation, automatic test execution and analysis and test quality measurement by automatic generation of requirements coverage and traceability reports. The described method is in-line with the software quality standards as it is for example specified in the new automotive standard for functional safety ISO 26262. The approach has already been implemented in a first instance for the Matlab/Simulink models and production code generation with TargetLink from dSPACE

Automated Functional Testing based on the Navigation of Web Applications

Web applications are becoming more and more complex. Testing such applications is an intricate hard and time-consuming activity. Therefore, testing is often poorly performed or skipped by practitioners. Test automation can help to avoid this situation. Hence, this paper presents a novel approach to perform automated software testing for web applications based on its navigation. On the one hand, web navigation is the process of traversing a web application using a browser. On the other hand, functional requirements are actions that an application must do. Therefore, the evaluation of the correct navigation of web applications results in the assessment of the specified functional requirements. The proposed method to perform the automation is done in four levels: test case generation, test data derivation, test case execution, and test case reporting. This method is driven by three kinds of inputs: i) UML models; ii) Selenium scripts; iii) XML files. We have implemented our approach in an open-source testing framework named Automatic Testing Platform. The validation of this work has been carried out by means of a case study, in which the target is a real invoice management system developed using a model-driven approach.Comment: In Proceedings WWV 2011, arXiv:1108.208

Search-based Test Generation for Automated Driving Systems: From Perception to Control Logic

abstract: Automated driving systems are in an intensive research and development stage, and the companies developing these systems are targeting to deploy them on public roads in a very near future. Guaranteeing safe operation of these systems is crucial as they are planned to carry passengers and share the road with other vehicles and pedestrians. Yet, there is no agreed-upon approach on how and in what detail those systems should be tested. Different organizations have different testing approaches, and one common approach is to combine simulation-based testing with real-world driving. One of the expectations from fully-automated vehicles is never to cause an accident. However, an automated vehicle may not be able to avoid all collisions, e.g., the collisions caused by other road occupants. Hence, it is important for the system designers to understand the boundary case scenarios where an autonomous vehicle can no longer avoid a collision. Besides safety, there are other expectations from automated vehicles such as comfortable driving and minimal fuel consumption. All safety and functional expectations from an automated driving system should be captured with a set of system requirements. It is challenging to create requirements that are unambiguous and usable for the design, testing, and evaluation of automated driving systems. Another challenge is to define useful metrics for assessing the testing quality because in general, it is impossible to test every possible scenario. The goal of this dissertation is to formalize the theory for testing automated vehicles. Various methods for automatic test generation for automated-driving systems in simulation environments are presented and compared. The contributions presented in this dissertation include (i) new metrics that can be used to discover the boundary cases between safe and unsafe driving conditions, (ii) a new approach that combines combinatorial testing and optimization-guided test generation methods, (iii) approaches that utilize global optimization methods and random exploration to generate critical vehicle and pedestrian trajectories for testing purposes, (iv) a publicly-available simulation-based automated vehicle testing framework that enables application of the existing testing approaches in the literature, including the new approaches presented in this dissertation.Dissertation/ThesisDoctoral Dissertation Computer Engineering 201

Validation Of Quality Requirements Through Requirements-Based Testing With Semi-Formalised Model

The usage of software system and applications has increased massively to fulfill various kind of purpose for organization,business and individual.In this case, high quality software system and application are required to ensure it provides the intended functionalities.To achieve quality software system and application,a good quality of requirements needs to be defined and validated.However,it is difficult to accomplish due to the flexibility of natural language requirements that can be confusing and easily misinterpreted.This can lead to requirements-related problems such as incorrectness, incompleteness and inconsistency.These errors in requirements will produce defective software that can lead to undesirable and non-acceptance by stakeholders. Therefore,it is crucial for the software requirements to fulfill basic quality attributes such as correctness, completeness,and consistency (3Cs).Motivated from these problems,the main objective of this studyis to develop an automated approach to validate the quality of requirements through Requirements-Based Testing methodology with semi-formalized model.This studyproposes a new automated approach to assist the requirements engineer and client-stakeholders to validate the quality of requirements.For this,we generate abstract tests by integratingRequirements-based Testing (RBT) methodology and rapid prototyping with semi-formalized models:Essential Use Cases (EUCs) and Essential User Interface (EUI).Next,we have developed pattern libraries to support the automatic extraction of abstract tests from the EUC model.They are test requirements pattern library and testcase pattern library.Here,an automated tool support called TestMEReq is also developed to realize the approach.The test-authoring template to assist requirements engineer to write accurate tests is also developed.Then,a real-time collaborative approachis also integrated with the tool to encourage users’ involvement in the validation process as well as to support better communication and collaboration among stakeholders.Finally,a comprehensive evaluation of the approach,comprising experiments of correctness test and usability test were conducted.In summary,the findings of the evaluations show that our approach can contribute to the body of knowledge of requirements engineering especially in enhancing the quality of requirements at the earliest stage.It is found that the approach is able to enhance the correctness level of the elicited requirements compared to the manual approach and produce correct generation of test.The results of the usability tests show that the approach is useful and helpful in validating the quality of requirements at the early stage of software development and able to ease the requirements validation process

Incremental bounded model checking for embedded software

Program analysis is on the brink of mainstream usage in embedded systems development. Formal verification of behavioural requirements, finding runtime errors and test case generation are some of the most common applications of automated verification tools based on bounded model checking (BMC). Existing industrial tools for embedded software use an off-the-shelf bounded model checker and apply it iteratively to verify the program with an increasing number of unwindings. This approach unnecessarily wastes time repeating work that has already been done and fails to exploit the power of incremental SAT solving. This article reports on the extension of the software model checker CBMC to support incremental BMC and its successful integration with the industrial embedded software verification tool BTC EMBEDDED TESTER. We present an extensive evaluation over large industrial embedded programs, mainly from the automotive industry. We show that incremental BMC cuts runtimes by one order of magnitude in comparison to the standard non-incremental approach, enabling the application of formal verification to large and complex embedded software. We furthermore report promising results on analysing programs with arbitrary loop structure using incremental BMC, demonstrating its applicability and potential to verify general software beyond the embedded domain

Automatically Generating Test Cases for Safety-Critical Software via Symbolic Execution

Automated test generation based on symbolic execution can be beneficial for systematically testing safety-critical software, to facilitate test engineers to pursue the strict testing requirements mandated by the certification standards, while controlling at the same time the costs of the testing process. At the same time, the development of safety-critical software is often constrained with programming languages or coding conventions that ban linguistic features which are believed to downgrade the safety of the programs, e.g., they do not allow dynamic memory allocation and variable-length arrays, limit the way in which loops are used, forbid recursion, and bound the complexity of control conditions. As a matter of facts, these linguistic features are also the main efficiency-blockers for the test generation approaches based on symbolic execution at the state of the art. This paper contributes new evidence of the effectiveness of generating test cases with symbolic execution for a significant class of industrial safety critical-systems. We specifically focus on Scade, a largely adopted model-based development language for safety-critical embedded software, and we report on a case study in which we exploited symbolic execution to automatically generate test cases for a set of safety-critical programs developed in Scade. To this end, we introduce a novel test generator that we developed in a recent industrial project on testing safety-critical railway software written in Scade, and we report on our experience of using this test generator for testing a set of Scade programs that belong to the development of an on-board signaling unit for high-speed rail. The results provide empirically evidence that symbolic execution is indeed a viable approach for generating high-quality test suites for the safety-critical programs considered in our case study

Constraint-Guided Test Execution Scheduling: An Experience Report at ABB Robotics

Automated test execution scheduling is crucial in modern software development environments, where components are frequently updated with changes that impact their integration with hardware systems. Building test schedules, which focus on the right tests and make optimal use of the available resources, both time and hardware, under consideration of vast requirements on the selection of test cases and their assignment to certain test execution machines, is a complex optimization task. Manual solutions are time-consuming and often error-prone. Furthermore, when software and hardware components and test scripts are frequently added, removed or updated, static test execution scheduling is no longer feasible and the motivation for automation taking care of dynamic changes grows. Since 2012, our work has focused on transferring technology based on constraint programming for automating the testing of industrial robotic systems at ABB Robotics. After having successfully transferred constraint satisfaction models dedicated to test case generation, we present the results of a project called DynTest whose goal is to automate the scheduling of test execution from a large test repository, on distinct industrial robots. This paper reports on our experience and lessons learned for successfully transferring constraint-based optimization models for test execution scheduling at ABB Robotics. Our experience underlines the benefits of a close collaboration between industry and academia for both parties.Comment: SafeComp 202