Testing Autonomous Cars for Feature Interaction Failures using Many-Objective Search

Complex systems such as autonomous cars are typically built as a composition of features that are independent units of functionality. Features tend to interact and impact one another’s behavior in unknown ways. A challenge is to detect and manage feature interactions, in particular, those that violate system requirements, hence leading to failures. In this paper, we propose a technique to detect feature interaction failures by casting our approach into a search-based test generation problem. We define a set of hybrid test objectives (distance functions) that combine traditional coverage-based heuristics with new heuristics specifically aimed at revealing feature interaction failures. We develop a new search-based test generation algorithm, called FITEST, that is guided by our hybrid test objectives. FITEST extends recently proposed many-objective evolutionary algorithms to reduce the time required to compute fitness values. We evaluate our approach using two versions of an industrial self-driving system. Our results show that our hybrid test objectives are able to identify more than twice as many feature interaction failures as two baseline test objectives used in the software testing literature (i.e., coverage-based and failure-based test objectives). Further, the feedback from domain experts indicates that the detected feature interaction failures represent real faults in their systems that were not previously identified based on analysis of the system features and their requirements

An orthogonal framework for fault tolerance composition in software systems

Building reliable systems is one of the major challenges faced by software developers as society is becoming more dependent on software systems. The failure of any system can lead to a serious loss, for example serious injury or death in case of safety critical systems and significant financial loss in the case of business-critical systems. As a consequence, fault tolerance is considered as a solution to provide reliability, but the fault tolerance capability is associated with many challenges, such as the right development phase where it needs to be introduced, how it can be composed with the software, and the issues that arise from this composition such as complexity and potential undesirable feature interactions. This thesis presents an orthogonal fault tolerance framework for the composition of design diversity fault tolerance mechanism with the base system. It further ensures the separation of concerns between the ‘base’ system and the fault tolerance mechanisms that are composed with the base system. The composition in this framework is based on operational semantics that describe the behaviour of the underlying components when composed with the fault tolerance mechanisms. A custom-built pre-processor is based on these composition rules, and is used to automatically compose the system component and the fault tolerance mechanisms. The very introduction of different fault tolerance mechanisms to the system may cause interactions with other fault tolerance features or with system components. Logic properties written in CTL and LTL are used in NuSMV to analyse undesirable interactions. To illustrate its applicability, the framework has been applied to the Home Automation and Therac-25 software

Effective Testing Of Advanced Driver Assistance Systems Using Evolutionary Algorithms And Machine Learning

Improving road safety is a major concern for most car manufacturers. In recent years, the development of Advanced Driver Assistance Systems (ADAS) has subsequently seen a tremendous boost. The development of such systems requires complex testing to ensure vehicle’s safety and reliability. Performing road tests tends to be dangerous, time-consuming, and costly. Hence, a large part of testing for ADAS has to be carried out using physics-based simulation platforms, which are able to emulate a wide range of virtual traffic scenarios and road environments. The main difficulties with simulation-based testing of ADAS are: (1) the test input space is large and multidimensional, (2) simulation platforms provide no guidance to engineers as to which scenarios should be selected for testing, and hence, simulation is limited to a small number of scenarios hand-picked by engineers, and (3) test executions are computationally expensive because they often involve executing high-fidelity mathematical models capturing continuous dynamic behaviors of vehicles and their environment. The complexity of testing ADAS is further exacerbated when many ADAS are employed together in a self-driving system. In particular, when self-driving systems include many ADAS (i.e., features), they tend to interact and impact one another’s behavior in an unknown way and may lead to conflicting situations. The main challenge here is to detect and manage feature interactions, in particular, those that violate system safety requirements, hence leading to critical failures. In practice, once feature interaction failures are detected, engineers need to devise resolution strategies to resolve potential conflicts between features. Developing resolution strategies is a complex task and despite the extensive domain expertise, these resolution strategies can be erroneous and are too complex to be manually repaired. In this dissertation, in addition to testing individual ADAS, we focus on testing self-driving systems that include several ADAS. In this dissertation, we propose a set of approaches based on meta-heuristic search and machine learning techniques to automate ADAS testing and to repair feature interaction failures in self-driving systems. The work presented in this dissertation is motivated by ADAS testing needs at IEE, a world-leading part supplier to the automotive industry. In this dissertation, we focus on the problem of design time testing of ADAS in a simulated environment, relying on Simulink models. The main research contributions in this dissertation are: - A testing approach for ADAS that combines multi-objective search with surrogate models to guide testing towards the most critical behaviors of ADAS, and to explore a larger part of the input search space with less computational resources. - An automated testing algorithm that builds on learnable evolution models and uses classification decision trees to guide the generation of new test scenarios within complex and multidimensional input spaces and help engineers interpret test results. - An automated technique that detects feature interaction failures in the context of self-driving systems based on analyzing executable function models typically developed to specify system behaviors at early development stages. - An automated technique that uses a new many-objective search algorithm to localize and repair errors in the feature interaction resolution rules for self-driving systems

Runtime Resolution of Feature Interactions in Evolving Telecommunications Systems

Feature interactions in telecommunications is an active research area. Many approaches to solve the so-called feature interaction problem have been proposed. However, all these approaches consider feature interaction as a somewhat isolated problem, in particular it is not seen in the context of evolving legacy systems and third party features in a deregulated market environment. An exception is the approach by Marples and Magill [MM98, Mar00], which presents an interaction detection mechanism and an essentially manual resolution approach. We develop an automatic resolution approach that can be integrated with Marples and Magill's detection mechanism. We distinguish two key concepts, namely solutions and resolutions. The former are essentially possible behaviours of the system, they are not qualified as desirable or undesirable, the latter are the desirable solutions. Our approach allows for automatic removal of undesired behaviour and selection of the "best" desired behaviour. The correctness, complexity and suitability of our approach are analysed. Two case studies support these more theoretical considerations. Our approach is transferable to other areas, such as quality of service management, and is not restricted to network architectures with a single point of control

Families of Formal Requirements in Telephone Switching

We introduce to the idea of families of requirements documents. Requirements families are sets of requirements whose common properties are so extensive that it is advantageous to study the common properties in the requirements before analyzing individual members. We present how we can apply this idea with a formal description technique, how our approach helps to avoid feature interaction problems in telephone switching, and how it supports the detection of remaining problems