Automatic Generation of Acceptance Test Cases from Use Case Specifications: an NLP-based Approach

Acceptance testing is a validation activity performed to ensure the conformance of software systems with respect to their functional requirements. In safety critical systems, it plays a crucial role since it is enforced by software standards, which mandate that each requirement be validated by such testing in a clearly traceable manner. Test engineers need to identify all the representative test execution scenarios from requirements, determine the runtime conditions that trigger these scenarios, and finally provide the input data that satisfy these conditions. Given that requirements specifications are typically large and often provided in natural language (e.g., use case specifications), the generation of acceptance test cases tends to be expensive and error-prone. In this paper, we present Use Case Modeling for System-level, Acceptance Tests Generation (UMTG), an approach that supports the generation of executable, system-level, acceptance test cases from requirements specifications in natural language, with the goal of reducing the manual effort required to generate test cases and ensuring requirements coverage. More specifically, UMTG automates the generation of acceptance test cases based on use case specifications and a domain model for the system under test, which are commonly produced in many development environments. Unlike existing approaches, it does not impose strong restrictions on the expressiveness of use case specifications. We rely on recent advances in natural language processing to automatically identify test scenarios and to generate formal constraints that capture conditions triggering the execution of the scenarios, thus enabling the generation of test data. In two industrial case studies, UMTG automatically and correctly translated 95% of the use case specification steps into formal constraints required for test data generation; furthermore, it generated test cases that exercise not only all the test scenarios manually implemented by experts, but also some critical scenarios not previously considered

A Model-Driven approach for functional test case generation

Test phase is one of the most critical phases in software engineering life cycle to assure the final system quality. In this context, functional system test cases verify that the system under test fulfills its functional specification. Thus, these test cases are frequently designed from the different scenarios and alternatives depicted in functional requirements. The objective of this paper is to introduce a systematic process based on the Model-Driven paradigm to automate the generation of functional test cases from functional requirements. For this aim, a set of metamodels and transformations and also a specific language domain to use them is presented. The paper finishes stating learned lessons from the trenches as well as relevant future work and conclusions that draw new research lines in the test cases generation context.Ministerio de Economía y Competitividad TIN2013-46928-C3-3-

Efficient and Effective Generation of Test Cases for Pedestrian Detection - Search-based Software Testing of Baidu Apollo in SVL

With the growing capabilities of autonomous vehicles, there is a higher demand for sophisticated and pragmatic quality assurance approaches for machine learning-enabled systems in the automotive AI context. The use of simulation-based prototyping platforms provides the possibility for early-stage testing, enabling inexpensive testing and the ability to capture critical corner-case test scenarios. Simulation-based testing properly complements conventional on-road testing. However, due to the large space of test input parameters in these systems, the efficient generation of effective test scenarios leading to the unveiling of failures is a challenge. This paper presents a study on testing pedestrian detection and emergency braking system of the Baidu Apollo autonomous driving platform within the SVL simulator. We propose an evolutionary automated test generation technique that generates failure-revealing scenarios for Apollo in the SVL environment. Our approach models the input space using a generic and flexible data structure and benefits a multi-criteria safety-based heuristic for the objective function targeted for optimization. This paper presents the results of our proposed test generation technique in the 2021 IEEE Autonomous Driving AI Test Challenge. In order to demonstrate the efficiency and effectiveness of our approach, we also report the results from a baseline random generation technique. Our evaluation shows that the proposed evolutionary test case generator is more effective at generating failure-revealing test cases and provides higher diversity between the generated failures than the random baseline

Efficient and Effective Generation of Test Cases for Pedestrian Detection - Search-based Software Testing of Baidu Apollo in SVL

With the growing capabilities of autonomous vehicles, there is a higher demand for sophisticated and pragmatic quality assurance approaches for machine learning-enabled systems in the automotive AI context. The use of simulation-based prototyping platforms provides the possibility for early-stage testing, enabling inexpensive testing and the ability to capture critical corner-case test scenarios. Simulation-based testing properly complements conventional on-road testing. However, due to the large space of test input parameters in these systems, the efficient generation of effective test scenarios leading to the unveiling of failures is a challenge. This paper presents a study on testing pedestrian detection and emergency braking system of the Baidu Apollo autonomous driving platform within the SVL simulator. We propose an evolutionary automated test generation technique that generates failure-revealing scenarios for Apollo in the SVL environment. Our approach models the input space using a generic and flexible data structure and benefits a multi-criteria safety-based heuristic for the objective function targeted for optimization. This paper presents the results of our proposed test generation technique in the 2021 IEEE Autonomous Driving AI Test Challenge. In order to demonstrate the efficiency and effectiveness of our approach, we also report the results from a baseline random generation technique. Our evaluation shows that the proposed evolutionary test case generator is more effective at generating failure-revealing test cases and provides higher diversity between the generated failures than the random baseline

Model-based data generation for the evaluation of functional reliability and resilience of distributed machine learning systems against abnormal cases

Future production technologies will comprise a multitude of systems whose core functionality is closely related to machine-learned models. Such systems require reliable components to ensure the safety of workers and their trust in the systems. The evaluation of the functional reliability and resilience of systems based on machine-learned models is generally challenging. For this purpose, appropriate test data must be available, which also includes abnormal cases. These abnormal cases can be unexpected usage scenarios, erroneous inputs, accidents during operation or even the failure of certain subcomponents. In this work, approaches to the model-based generation of an arbitrary abundance of data representing such abnormal cases are explored. Such computer-based generation requires domain-specific approaches, especially with respect to the nature and distribution of the data, protocols used, or domain-specific communication structures. In previous work, we found that different use cases impose different requirements on synthetic data, and the requirements in turn imply different generation methods [1]. Based on this, various use cases are identified and different methods for computer-based generation of realistic data, as well as for the quality assessment of such data, are explored. Ultimately we explore the use of Federated Learning (FL) to address data privacy and security challenges in Industrial Control Systems. FL enables local model training while keeping sensitive information decentralized and private to their owners. In detail, we investigate whether FL can benefit clients with limited knowledge by leveraging collaboratively trained models that aggregate client-specific knowledge distributions. We found that in such scenarios federated training results in a significant increase in classification accuracy by 31.3% compared to isolated local training. Furthermore, as we introduce Differential Privacy, the resulting model achieves on par accuracy of 99.62% to an idealized case where data is independent and identically distributed across clients

Testing Component-Based Real Time Systems

International audienceThis paper focuses on studying efficient solutions for modeling and deriving compositional tests for component-based real-time systems. In this work, we propose a coherent framework that does not require the computation of the synchronous product (composition) of components, and therefore avoids a major bottleneck in this class of test. For this framework, we introduce an approach and associated algorithm. In our approach, the overall behavior of the system is obtained by restricting free runs of components to those involving interactions between them. This restriction is achieved through the use of a particular component called assembly controller. For the generation algorithm, compositional test cases are derived from the assembly controller model using symbolic analysis. This reduces the state space size (a practical size) and enables the generation of sequences which cover all critical interaction scenarios

PlayTest: A Gamified Test Generator for Games

Games are usually created incrementally, requiring repeated testing of the same scenarios, which is a tedious and error-prone task for game developers. Therefore, we aim to alleviate this game testing process by encapsulating it into a game called Playtest, which transforms the tiring testing process into a competitive game with a purpose. Playtest automates the generation of valuable test cases based on player actions, without the players even realising it. We envision the use of Playtest to crowdsource the task of testing games by giving players access to the respective games through our tool in the playtesting phases during the development process.Comment: 4 pages with 4 figures, to be published in Proceedings of the 2nd International Workshop on Gamification in Software Development, Verification, and Validation 202

Practical Application Of Uml Activity Diagrams For The Generation Of Test Cases

Software testing and debugging represents around one third of total effort in development projects. Different factors which have influence on poor practices of testing have been identified through specific surveys. Amongst several, one of the most important is the lack of efficient methods to exploit development models for generating test cases. This paper presents a new method for automatically generating a complete set of functional test cases from UML activity diagrams complementing specification of use cases. Test cases are prioritized according to software risk information. Results from experiences with more than 70 software professionals/experts validate benefits of the method. Participants also confirm its interest and effectiveness for testing needs of industry