Efficient and Effective Generation of Test Cases for Pedestrian Detection - Search-based Software Testing of Baidu Apollo in SVL

With the growing capabilities of autonomous vehicles, there is a higher demand for sophisticated and pragmatic quality assurance approaches for machine learning-enabled systems in the automotive AI context. The use of simulation-based prototyping platforms provides the possibility for early-stage testing, enabling inexpensive testing and the ability to capture critical corner-case test scenarios. Simulation-based testing properly complements conventional on-road testing. However, due to the large space of test input parameters in these systems, the efficient generation of effective test scenarios leading to the unveiling of failures is a challenge. This paper presents a study on testing pedestrian detection and emergency braking system of the Baidu Apollo autonomous driving platform within the SVL simulator. We propose an evolutionary automated test generation technique that generates failure-revealing scenarios for Apollo in the SVL environment. Our approach models the input space using a generic and flexible data structure and benefits a multi-criteria safety-based heuristic for the objective function targeted for optimization. This paper presents the results of our proposed test generation technique in the 2021 IEEE Autonomous Driving AI Test Challenge. In order to demonstrate the efficiency and effectiveness of our approach, we also report the results from a baseline random generation technique. Our evaluation shows that the proposed evolutionary test case generator is more effective at generating failure-revealing test cases and provides higher diversity between the generated failures than the random baseline

Efficient and Effective Generation of Test Cases for Pedestrian Detection - Search-based Software Testing of Baidu Apollo in SVL

With the growing capabilities of autonomous vehicles, there is a higher demand for sophisticated and pragmatic quality assurance approaches for machine learning-enabled systems in the automotive AI context. The use of simulation-based prototyping platforms provides the possibility for early-stage testing, enabling inexpensive testing and the ability to capture critical corner-case test scenarios. Simulation-based testing properly complements conventional on-road testing. However, due to the large space of test input parameters in these systems, the efficient generation of effective test scenarios leading to the unveiling of failures is a challenge. This paper presents a study on testing pedestrian detection and emergency braking system of the Baidu Apollo autonomous driving platform within the SVL simulator. We propose an evolutionary automated test generation technique that generates failure-revealing scenarios for Apollo in the SVL environment. Our approach models the input space using a generic and flexible data structure and benefits a multi-criteria safety-based heuristic for the objective function targeted for optimization. This paper presents the results of our proposed test generation technique in the 2021 IEEE Autonomous Driving AI Test Challenge. In order to demonstrate the efficiency and effectiveness of our approach, we also report the results from a baseline random generation technique. Our evaluation shows that the proposed evolutionary test case generator is more effective at generating failure-revealing test cases and provides higher diversity between the generated failures than the random baseline

Light Commercial Vehicle ADAS-Oriented Modelling: An Optimization-Based Conversion Tool from Multibody to Real-Time Vehicle Dynamics Model

In the last few years, the number of Advanced Driver Assistance Systems (ADAS) on road vehicles has been increased with the aim of dramatically reducing road accidents. Therefore, the OEMs need to integrate and test these systems, to comply with the safety regulations. To lower the development cost, instead of experimental testing, many virtual simulation scenarios need to be tested for ADAS validation. The classic multibody vehicle approach, normally used to design and optimize vehicle dynamics performance, is not always suitable to cope with these new tasks; therefore, real-time lumped-parameter vehicle models implementation becomes more and more necessary. This paper aims at providing a methodology to convert experimentally validated light commercial vehicles (LCV) multibody models (MBM) into real-time lumped-parameter models (RTM). The proposed methodology involves the definition of the vehicle subsystems and the level of complexity required to achieve a good match between the simulation results obtained from the two models. Thus, an automatic vehicle model converter will be presented together with the assessment of its accuracy. An optimization phase is included into the conversion tool, to fine-tune uncertain vehicle parameters and to compensate for inherent modelling differences. The objective function of the optimization is based on typical performance indices used for vehicle longitudinal and lateral dynamics assessment. Finally, the simulation results from the original and converted models are compared during steady-state and transient tests, to prove the conversion fidelity

Perception architecture exploration for automotive cyber-physical systems

2022 Spring.Includes bibliographical references.In emerging autonomous and semi-autonomous vehicles, accurate environmental perception by automotive cyber physical platforms are critical for achieving safety and driving performance goals. An efficient perception solution capable of high fidelity environment modeling can improve Advanced Driver Assistance System (ADAS) performance and reduce the number of lives lost to traffic accidents as a result of human driving errors. Enabling robust perception for vehicles with ADAS requires solving multiple complex problems related to the selection and placement of sensors, object detection, and sensor fusion. Current methods address these problems in isolation, which leads to inefficient solutions. For instance, there is an inherent accuracy versus latency trade-off between one stage and two stage object detectors which makes selecting an enhanced object detector from a diverse range of choices difficult. Further, even if a perception architecture was equipped with an ideal object detector performing high accuracy and low latency inference, the relative position and orientation of selected sensors (e.g., cameras, radars, lidars) determine whether static or dynamic targets are inside the field of view of each sensor or in the combined field of view of the sensor configuration. If the combined field of view is too small or contains redundant overlap between individual sensors, important events and obstacles can go undetected. Conversely, if the combined field of view is too large, the number of false positive detections will be high in real time and appropriate sensor fusion algorithms are required for filtering. Sensor fusion algorithms also enable tracking of non-ego vehicles in situations where traffic is highly dynamic or there are many obstacles on the road. Position and velocity estimation using sensor fusion algorithms have a lower margin for error when trajectories of other vehicles in traffic are in the vicinity of the ego vehicle, as incorrect measurement can cause accidents. Due to the various complex inter-dependencies between design decisions, constraints and optimization goals a framework capable of synthesizing perception solutions for automotive cyber physical platforms is not trivial. We present a novel perception architecture exploration framework for automotive cyber- physical platforms capable of global co-optimization of deep learning and sensing infrastructure. The framework is capable of exploring the synthesis of heterogeneous sensor configurations towards achieving vehicle autonomy goals. As our first contribution, we propose a novel optimization framework called VESPA that explores the design space of sensor placement locations and orientations to find the optimal sensor configuration for a vehicle. We demonstrate how our framework can obtain optimal sensor configurations for heterogeneous sensors deployed across two contemporary real vehicles. We then utilize VESPA to create a comprehensive perception architecture synthesis framework called PASTA. This framework enables robust perception for vehicles with ADAS requiring solutions to multiple complex problems related not only to the selection and placement of sensors but also object detection, and sensor fusion as well. Experimental results with the Audi-TT and BMW Minicooper vehicles show how PASTA can intelligently traverse the perception design space to find robust, vehicle-specific solutions

OpenSBT: A Modular Framework for Search-based Testing of Automated Driving Systems

Search-based software testing (SBT) is an effective and efficient approach for testing automated driving systems (ADS). However, testing pipelines for ADS testing are particularly challenging as they involve integrating complex driving simulation platforms and establishing communication protocols and APIs with the desired search algorithm. This complexity prevents a wide adoption of SBT and thorough empirical comparative experiments with different simulators and search approaches. We present OpenSBT, an open-source, modular and extensible framework to facilitate the SBT of ADS. With OpenSBT, it is possible to integrate simulators with an embedded system under test, search algorithms and fitness functions for testing. We describe the architecture and show the usage of our framework by applying different search algorithms for testing Automated Emergency Braking Systems in CARLA as well in the high-fidelity Prescan simulator in collaboration with our industrial partner DENSO. OpenSBT is available at https://git.fortiss.org/opensbt

Advanced Driver-Assistance System with Traffic Sign Recognition for Safe and Efficient Driving

Advanced Driver-Assistance Systems (ADAS) coupled with traffic sign recognition could lead to safer driving environments. This study presents a sophisticated, yet robust and accurate traffic sign detection system using computer vision and ML, for ADAS. Unavailability of large local traffic sign datasets and the unbalances of traffic sign distribution are the key bottlenecks of this research.  Hence, we choose to work with support vector machines (SVM) with a custom-built unbalance dataset, to build a lightweight model with excellent classification accuracy.  The SVM model delivered optimum performance with the radial basis kernel, C=10, and gamma=0.0001. In the proposed method, same priority was given to processing time (testing time) and accuracy, as traffic sign identification is time critical. The final accuracy obtained was 87% (with confidence interval 84%-90%) with a processing time of 0.64s (with confidence interval of 0.57s-0.67s) for correct detection at testing, which emphasizes the effectiveness of the proposed method

Estimating Vehicle Suspension Characteristics for Digital Twin Creation with Genetic Algorithm

Usage of simulation techniques like Vehicle-in-the-Loop, Scenario-in-the-Loop, and other mixed-reality systems are becoming inevitable in autonomous vehicle development, particularly in testing and validation. These methods rely on using digital twins, realistic representations of real vehicles, and traffic in a carefully rebuilt virtual world. Recreating them precisely in a virtual ecosystem requires many parameters of real vehicles to follow their properties in a simulation. This is especially true for vehicle dynamics, where these parameters have high impact on the simulation results. The paper's objective is to provide a method that can help reverse engineering a real car's suspension characteristics with the help of a genetic algorithm. A detailed description of the method is presented, guiding the reader through the whole process, including the meta-heuristic function's settings and how it interfaces with IPG Carmaker. The paper also presents multiple measurements, which can be effortlessly recreated without expensive devices or the need to disassemble any vehicle parts. Measurements are reproduced in two separate simulation tools with special scenarios providing an efficient way to analyze and verify the results. The provided method creates vehicle suspension characteristics with adequate quality, opening up the possibility to use them in the creation of digital twins or creating virtual traffic with realistic vehicle dynamics for high-quality visualization. Results show satisfying accuracy when tested with OpenCRG

Adversarial attack on a deep model of pedestrian detection in CARLA simulator

L'assistència a la conducció i la conducció autònoma utilitzen models profunds de percepció per dur a terme tasques com la detecció i classificació d'objectes. Un cop entrenats els models, és important trobar casos en què aquests puguin fallar amb l'objectiu d'incloure'ls en un posterior reentrenament i aconseguir models més robusts. Una manera de trobar situacions no contemplades és a través de la simulació, gràcies a la qual es poden forçar casos extrems. En aquest context, aquest treball s'enfoca en la detecció de vianants per imatge utilitzant el simulador CARLA a partir de la creació automàtica d'escenaris per tal de validar les prestacions del detector. Mitjançant un algorisme genètic s'automatitza la cerca d'escenaris plausibles destinades a fer fallar el detector, mostrant-ne així les debilitats. Aquest procediment de validació del detector es considera un mètode d'atac adversari.La asistencia a la conducción y la conducción autónoma utilizan modelos profundos de percepción para realizar tareas como la detección y clasificación de objetos. Una vez entrenados los modelos, es importante encontrar casos en los que éstos puedan fallar con el objetivo de incluirlos en un posterior reentrenamiento y conseguir modelos más robustos. Una forma de encontrar situaciones no contempladas es a través de la simulación, gracias a la cual se pueden forzar casos extremos. En este contexto, este trabajo se enfoca en la detección de peatones por imagen utilizando el simulador CARLA a partir de la creación automática de escenarios para validar las prestaciones del detector. Mediante un algoritmo genético se automatiza la búsqueda de escenarios plausibles destinadas a hacer fallar el detector, mostrando así sus debilidades. Este procedimiento de validación del detector se considerará como un método de ataque adversario.Driving assistance and autonomous driving use deep perception models to perform tasks such as object detection and classification. Once the models have been trained, it is important to find cases where they can fail, with the aim of including them in subsequent retraining and achieving more robust models. One way to find unforeseen situations is through simulation, thanks to which corner cases can be forced. In this context, this project focuses on pedestrian detection by image using CARLA simulator, based on automatic creation of scenarios in order to validate the performance of the detector. One genetic algorithm automates the search for plausible scenarios designed to cause the detector to fail, thus exposing its weaknesses. This detector validation procedure is considered an adversarial attack method