An extended abstract of "Metamorphic testing: testing the untestable"

This document is an extended abstract of an IEEE Software paper, "Metamorphic Testing: Testing the Untestable," presented as a J1C2 (Journal publication first, Conference presentation following) at the IEEE Computer Society signature conference on Computers, Software and Applications (COMPSAC 2019), hosted by Marquette University, Milwaukee, Wisconsin, USA. © 2019 IEEE

Metamorphic Testing of Autonomous Vehicles: a Case Study on Simulink

Autonomous Vehicles (AVs) will revolutionize the way people travel by car. However, in order to deploy autonomous vehicles, effective testing techniques are required. The driving quality of an AV should definitely be considered when testing such systems. However, as in other complex systems, determining the outcome of a test in the driving quality on an AV can be extremely complex. To solve this issue, in this paper we explore the application of Quality-of-Service (QoS) aware metamorphic testing to test AVs modeled in MATLAB/Simulink, one of the predominant modeling tools in the market. We first defined a set of QoS measures applied to AVs by considering as input a recent study. With them, we define metamorphic relations. Lastly we assess the approach in an AV modeled in Simulink by using mutation testing. The results suggests that our approach is effective at detecting faults

RMT: Rule-based Metamorphic Testing for Autonomous Driving Models

Deep neural network models are widely used for perception and control in autonomous driving. Recent work uses metamorphic testing but is limited to using equality-based metamorphic relations and does not provide expressiveness for defining inequality-based metamorphic relations. To encode real world traffic rules, domain experts must be able to express higher order relations e.g., a vehicle should decrease speed in certain ratio, when there is a vehicle x meters ahead and compositionality e.g., a vehicle must have a larger deceleration, when there is a vehicle ahead and when the weather is rainy and proportional compounding effect to the test outcome. We design RMT, a declarative rule-based metamorphic testing framework. It provides three components that work in concert:(1) a domain specific language that enables an expert to express higher-order, compositional metamorphic relations, (2) pluggable transformation engines built on a variety of image and graphics processing techniques, and (3) automated test generation that translates a human-written rule to a corresponding executable, metamorphic relation and synthesizes meaningful inputs.Our evaluation using three driving models shows that RMT can generate meaningful test cases on which 89% of erroneous predictions are found by enabling higher-order metamorphic relations. Compositionality provides further aids for generating meaningful, synthesized inputs-3012 new images are generated by compositional rules. These detected erroneous predictions are manually examined and confirmed by six human judges as meaningful traffic rule violations. RMT is the first to expand automated testing capability for autonomous vehicles by enabling easy mapping of traffic regulations to executable metamorphic relations and to demonstrate the benefits of expressivity, customization, and pluggability

Performance-Driven Metamorphic Testing of Cyber-Physical Systems

Cyber-physical systems (CPSs) are a new generation of systems, which integrate software with physical processes. The increasing complexity of these systems, combined with the un certainty in their interactions with the physical world, makes the definition of effective test oracles especially challenging, facing the well-known test oracle problem. Metamorphic testing has shown great potential to alleviate the test oracle problem by exploiting the relations among the inputs and outputs of different executions of the system, so-called metamorphic relations (MRs). In this article, we propose an MR pattern called PV for the identification of performance-driven MRs, and we show its applicability in two CPSs from different domains, which are automated navigation systems and elevator control systems. For the evaluation, we as sessed the effectiveness of this approach for detecting failures in an open-source simulation-based autonomous navigation system, as well as in an industrial case study from the elevation domain. We derive concrete MRs based on the PV pattern for both case studies, and we evaluate their effectiveness with seeded faults. Results show that the approach is effective at detecting over 88% of the seeded faults, while keeping the ratio of FPs at 4% or lower.European Union's Horizon 2020 Research and Innovation Programme (Grant Number: 871319)Junta de Andalucía US-1264651 (APOLO)Junta de Andalucía P18-FR-2895 (EKIPMENT-PLUS)Ministerio de Ciencia e Innovación RTI2018-101204-B-C21 (HORATIO)Mondragon Unibertsitatea IT1519-2

Exploratory datamorphic testing of classification applications

Testing has been widely recognised as difficult for AI applications. This paper proposes a set of testing strategies for testing machine learning applications in the framework of the datamorphism testing methodology. In these strategies, testing aims at exploring the data space of a classification or clustering application to discover the boundaries between classes that the machine learning application defines. This enables the tester to understand precisely the behaviour and function of the software under test. In the paper, three variants of exploratory strategies are presented with the algorithms as implemented in the automated datamorphic testing tool Morphy. The correctness of these algorithms are formally proved. The paper also reports the results of some controlled experiments with Morphy that study the factors that affect the test effectiveness of the strategies

Testing and Validation Framework for Autonomous Aerial Vehicles

Autonomous aerial vehicles (AAV) have the potential to have market disruptions for various industries such as ground delivery and aerial transportation. Hence, the USAF has called for increased level of autonomy. There has been a significant progress in artificial intelligence engines, complex and non-deterministic system components, which are at the core of the autonomous aerial platforms. Traditional testing and validation methods fall short of satisfying the requirement of testing such complex systems. Therefore, to achieve highly or fully autonomous capabilities, a major leap forward in the validation is required. The key challenges are the localization of problems, development of object models for perception and the creation of a safety measure. A similar challenge exists in ground autonomous vehicles (AVs), where there is a significant investment in recent years. However, there are important differences in the environmental and regulatory conditions between these two domains. In this paper, we present a validation framework that uses modeling and simulation and formal methods for solving the issues in the validation of AAVs. We define a novel abstraction stack using separation of concerns and create a testing plan using techniques such as constrained pseudo-random test generation, random walks and functional assertions. The system aims to assess the creation of an evolving safety measure and a licensing structure