Critical Scenario Identification for Testing of Autonomous Driving Systems

Background: Autonomous systems have received considerable attention from academia and are adopted by various industrial domains, such as automotive, avionics, etc. As many of them are considered safety-critical, testing is indispensable to verify their reliability and safety. However, there is no common standard for testing autonomous systems efficiently and effectively. Thus new approaches for testing such systems must be developed.Aim: The objective of this thesis is two-fold. First, we want to present an overview of software testing of autonomous systems, i.e., relevant concepts, challenges, and techniques available in academic research and industry practice. Second, we aim to establish a new approach for testing autonomous driving systems and demonstrate its effectiveness by using real autonomous driving systems from industry.Research Methodology: We conducted the research in three steps using the design science paradigm. First, we explored the existing literature and industry practices to understand the state of the art for testing of autonomous systems. Second, we focused on a particular sub-domain - autonomous driving - and proposed a systematic approach for critical test scenario identification. Lastly, we validated our approach and employed it for testing real autonomous driving systems by collaborating with Volvo Cars.Results: We present the results as four papers in this thesis. First, we conceptualized a definition of autonomous systems and classified challenges and approaches, techniques, and practices for testing autonomous systems in general. Second, we designed a systematic approach for critical test scenario identification. We employed the approach for testing two real autonomous driving systems from the industry and have effectively identified critical test scenarios. Lastly, we established a model for predicting the distribution of vehicle-pedestrian interactions for realistic test scenario generation for autonomous driving systems. Conclusion: Critical scenario identification is a favorable approach to generate test scenarios and facilitate the testing of autonomous driving systems in an efficient way. Future improvement of the approach includes (1) evaluating the effectiveness of the generated critical scenarios for testing; (2) extending the sub-components in this approach; (3) combining different testing approaches, and (4) exploring the application of the approach to test different autonomous systems

Assessing safety functionalities in the design and validation of driving automation

This paper aims to contribute to the comprehensive and systematic safety assessment of Automated Driving Systems (ADSs) by identifying unknown hazardous areas of operation. The current methodologies employed in this domain typically involve estimating the distributions of situational variables based on human-centered field test, crash databases, or expert knowledge of critical values. However, due to the lack of a-priori knowledge regarding the influential factors, their critical ranges, and their distributions, these approaches may not be entirely suitable for the assessment of emerging automated driving technologies. To deal with this challenging problem, here we propose a testing methodology incorporating realistic yet unobserved driving conditions, distinguished by numerous situational variables, so to encompass unknown unsafe conditions comprehensively. Our methodology utilizes stochastic simulation and uncertainty modeling techniques to account for the variability of realistic driving conditions and their impact on ADSs' performances. By doing so, we aim to identify unsafe operational regions and triggering conditions that can lead to hazardous behaviors, thus improving the development and safety of automated driving functions. For our purposes, the Latin Hypercube Sampling technique and the recently proposed PAWN density-based sensitivity analysis method are employed. We apply this methodology for the first time in the specific field of ADSs design and validation, using an exemplificative use case. We discuss and compare the results obtained from our approach with those obtained from a traditional approach

Methods and models for safety benefit assessment of advanced driver assistance systems in car-to-cyclist conflicts

To help drivers avoid or mitigate the severity of crashes, advanced driver assistance systems (ADAS) can be designed to provide warnings or interventions. Prospective safety assessment of ADAS is important to quantify and optimise their safety benefit. Such safety assessment methods include, for example, virtual simulations and test-track testing.Today, there are many components of virtual safety assessment simulations with models or methods that are missing or can be substantially improved. This is particularly true for simulations assessing ADASs that address crashes involving cyclists—a crash type that is not decreasing at the same rate as the overall number of road crashes in Europe. The specific methodological gaps that this work addresses are: a) computational driver models for car-to-cyclist overtaking, b) algorithms for model fitting and efficient calculation of ADAS intervention time, and c) a method for merging data from different data sources into the safety assessment.Specifically, for a), different driver models for everyday driver behaviour while overtaking cyclists in a naturalistic driving setting were derived and compared. For b), computationally efficient algorithms to fit driver models to data and compute ADAS intervention time were developed for different types of vehicle models. The algorithms can be included in ADAS both for offline use in virtual assessment simulations and online real-time use in in-vehicle ADAS. Lastly, for c), a method was developed that uses Bayesian statistics to combine results from different data sources, e.g., simulations and test-track data, for ADAS safety benefit assessment.In addition to presenting five peer-reviewed scientific publications, which address these issues, this compilation thesis discusses the use of different data sources; introduces the fundamentals of Bayesian inference, linear programming, and numerical root-finding algorithms; and provides the rationale for methodological choices made, where relevant. Finally, this thesis describes the relationships among the publications and places them into context with existing literature.This work developed driver models for the virtual simulations and methods for the reliable estimation of the prospective safety benefit, which together have the potential to improve the design and the evaluation of ADAS in general, and ADAS for the car-to-cyclist overtaking scenario in particular

Risk analysis of autonomous vehicle and its safety impact on mixed traffic stream

In 2016, more than 35,000 people died in traffic crashes, and human error was the reason for 94% of these deaths. Researchers and automobile companies are testing autonomous vehicles in mixed traffic streams to eliminate human error by removing the human driver behind the steering wheel. However, recent autonomous vehicle crashes while testing indicate the necessity for a more thorough risk analysis. The objectives of this study were (1) to perform a risk analysis of autonomous vehicles and (2) to evaluate the safety impact of these vehicles in a mixed traffic stream. The overall research was divided into two phases: (1) risk analysis and (2) simulation of autonomous vehicles. Risk analysis of autonomous vehicles was conducted using the fault tree method. Based on failure probabilities of system components, two fault tree models were developed and combined to predict overall system reliability. It was found that an autonomous vehicle system could fail 158 times per one-million miles of travel due to either malfunction in vehicular components or disruption from infrastructure components. The second phase of this research was the simulation of an autonomous vehicle, where change in crash frequency after autonomous vehicle deployment in a mixed traffic stream was assessed. It was found that average travel time could be reduced by about 50%, and 74% of conflicts, i.e., traffic crashes, could be avoided by replacing 90% of the human drivers with autonomous vehicles

Measurable Safety of Automated Driving Functions in Commercial Motor Vehicles

With the further development of automated driving, the functional performance increases resulting in the need for new and comprehensive testing concepts. This doctoral work aims to enable the transition from quantitative mileage to qualitative test coverage by aggregating the results of both knowledge-based and data-driven test platforms. The validity of the test domain can be extended cost-effectively throughout the software development process to achieve meaningful test termination criteria

Operational effectiveness of connected vehicle smartphone technology on a signalized corridor

Over the last decade, extensive research efforts have been placed on performance evaluation and the benefits of innovative CV applications. Findings indicate that CV technology can effectively mitigate the safety, mobility, and environmental challenges experienced on transportation networks. Most of research evaluated CV technology through simulation studies. However, a field study provides a more ideal method of assessing CV technology effectiveness. Therefore, a field study to obtain the actual effectiveness of CV technology was warranted, to validate previous findings, and to add to the body of knowledge surrounding this topic. This thesis presents both a field study and simulation evaluation of the effectiveness of CV smartphone technology on a 1.1 mile segment of State Road 121, containing five intersections, in Gainesville, Florida. Field observations were conducted using a CV application, developed by Connected Signals, Inc., that uses a smartphone application, called EnLighten, to communicate intersection information to driver’s smartphone, which serves as a vehicle on-board unit. Traffic operation and safety performance was evaluated using start-up lost time, discharge distribution model, and speed harmonization. Findings show that the CV smartphone technology improved intersection performance with a reduction in start-up lost time of approximately 86%. Additionally, driving safety improved with a reduction in speed variability by nearly 61% between vehicles in a specific lane for a 100% CV penetration rate. Cost analyses of deploying CV smartphone technology indicate that implementation may result in an average total economic cost savings associated with crashes of nearly $6.8 million at the study site, and approximately$5.6 billion statewide. Findings of the simulation evaluation revealed that the CV technology improved performance of intersections operating at a Level of Service (LOS) B or better, compared to lower operating levels. Operational performance improved at intersections operating at a LOS C with a 30% to 60% CV penetration rate

Non-Line of Sight Test Scenario Generation for Connected Autonomous Vehicle

Connected autonomous vehicles (CAV) level 4-5 use sensors to perceive their environment. These sensors are able to detect only up to a certain range and this range can be further constrained by the presence of obstacles in its path or as a result of the geometry of the road, for example, at a junction. This is termed as a non-line of sight (NLOS) scenario where the ego vehicle (system under test) is unable to detect an oncoming dynamic object due to obstacles or the geometry of the road. A large body of work now exist which proposes methods for extending the perception horizon of CAV’s using vehicular communication and incorporating this into CAV algorithms ranging from obstacle detection to path planning and beyond. Such proposed new algorithms and entire systems needs testing and validating, which can be conducted through primarily two ways, on road testing and simulation. On-road testing can be extremely expensive and time-consuming and may not cover all possible test scenarios. Testing through simulation is inexpensive and has a better scenario space coverage. However, there is currently a dearth in simulated testing techniques that provides the environment to test technologies and algorithms developed for NLOS scenarios. This thesis puts forward a novel end-to-end framework for testing the abilities of a CAV through simulated generation of NLOS scenarios. This has been achieved through following the development process of Functional, Logical and Concrete scenarios along the V-model-based development process in ISO 26262. The process begins with the representation of the NLOS environment (including the digital environment) knowledge as a scalable ontology where Functional and Logical scenarios stand for different abstraction levels. The proposed new ontology comprises of six layers: ‘Environment’, ‘Road User’, ‘Object Type’, ‘Communication Network’, ‘Scene’ and ‘Scenario’. The ontology is modelled and validated in protégé software and exported to OWL API where the logical scenarios are generated and validated. An innumerable number of “concrete” scenarios are generated as a result of the possible combinations of the values from the domains of each concept’s attributes. This research puts forward a novel genetic- algorithm (GA) approach to search through the scenario space and filter out safety critical test scenarios. A critical NLOS scenario is one where a collision is highly likely because the ego vehicle was unable to detect an obstacle in time due to obstructions present in the line-of-sight of the sensors or created due to the road geometry. The metric proposed to identify critical scenarios which also acts as the GA’s fitness function uses the time-to-collision (TTC) and total stopping time (TST) metric. These generated critical scenarios and proposed fitness function have been validated through MATLAB simulation. Furthermore, this research incorporates the relevant knowledge of vehicle-to-vehicle (V2V) communication technologies in the proposed ontology and uses the communication layer instances in the MATLAB simulation to support the testing of the increasing number of approaches that uses communications for alerting oncoming vehicles about imminent danger, or in other word, mitigating an otherwise critical scenario

Measurable Safety of Automated Driving Functions in Commercial Motor Vehicles - Technological and Methodical Approaches

Fahrerassistenzsysteme sowie automatisiertes Fahren leisten einen wesentlichen Beitrag zur Verbesserung der Verkehrssicherheit von Kraftfahrzeugen, insbesondere von Nutzfahrzeugen. Mit der Weiterentwicklung des automatisierten Fahrens steigt hierbei die funktionale Leistungsfähigkeit, woraus Anforderungen an neue, gesamtheitliche Erprobungskonzepte entstehen. Um die Absicherung höherer Stufen von automatisierten Fahrfunktionen zu garantieren, sind neuartige Verifikations- und Validierungsmethoden erforderlich. Ziel dieser Arbeit ist es, durch die Aggregation von Testergebnissen aus wissensbasierten und datengetriebenen Testplattformen den Übergang von einer quantitativen Kilometerzahl zu einer qualitativen Testabdeckung zu ermöglichen. Die adaptive Testabdeckung zielt somit auf einen Kompromiss zwischen Effizienz- und Effektivitätskriterien für die Absicherung von automatisierten Fahrfunktionen in der Produktentstehung von Nutzfahrzeugen ab. Diese Arbeit umfasst die Konzeption und Implementierung eines modularen Frameworks zur kundenorientierten Absicherung automatisierter Fahrfunktionen mit vertretbarem Aufwand. Ausgehend vom Konfliktmanagement für die Anforderungen der Teststrategie werden hochautomatisierte Testansätze entwickelt. Dementsprechend wird jeder Testansatz mit seinen jeweiligen Testzielen integriert, um die Basis eines kontextgesteuerten Testkonzepts zu realisieren. Die wesentlichen Beiträge dieser Arbeit befassen sich mit vier Schwerpunkten: * Zunächst wird ein Co-Simulationsansatz präsentiert, mit dem sich die Sensoreingänge in einem Hardware-in-the-Loop-Prüfstand mithilfe synthetischer Fahrszenarien simulieren und/ oder stimulieren lassen. Der vorgestellte Aufbau bietet einen phänomenologischen Modellierungsansatz, um einen Kompromiss zwischen der Modellgranularität und dem Rechenaufwand der Echtzeitsimulation zu erreichen. Diese Methode wird für eine modulare Integration von Simulationskomponenten, wie Verkehrssimulation und Fahrdynamik, verwendet, um relevante Phänomene in kritischen Fahrszenarien zu modellieren. * Danach wird ein Messtechnik- und Datenanalysekonzept für die weltweite Absicherung von automatisierten Fahrfunktionen vorgestellt, welches eine Skalierbarkeit zur Aufzeichnung von Fahrzeugsensor- und/ oder Umfeldsensordaten von spezifischen Fahrereignissen einerseits und permanenten Daten zur statistischen Absicherung und Softwareentwicklung andererseits erlaubt. Messdaten aus länderspezifischen Feldversuchen werden aufgezeichnet und zentral in einer Cloud-Datenbank gespeichert. * Anschließend wird ein ontologiebasierter Ansatz zur Integration einer komplementären Wissensquelle aus Feldbeobachtungen in ein Wissensmanagementsystem beschrieben. Die Gruppierung von Aufzeichnungen wird mittels einer ereignisbasierten Zeitreihenanalyse mit hierarchischer Clusterbildung und normalisierter Kreuzkorrelation realisiert. Aus dem extrahierten Cluster und seinem Parameterraum lassen sich die Eintrittswahrscheinlichkeit jedes logischen Szenarios und die Wahrscheinlichkeitsverteilungen der zugehörigen Parameter ableiten. Durch die Korrelationsanalyse von synthetischen und naturalistischen Fahrszenarien wird die anforderungsbasierte Testabdeckung adaptiv und systematisch durch ausführbare Szenario-Spezifikationen erweitert. * Schließlich wird eine prospektive Risikobewertung als invertiertes Konfidenzniveau der messbaren Sicherheit mithilfe von Sensitivitäts- und Zuverlässigkeitsanalysen durchgeführt. Der Versagensbereich kann im Parameterraum identifiziert werden, um die Versagenswahrscheinlichkeit für jedes extrahierte logische Szenario durch verschiedene Stichprobenverfahren, wie beispielsweise die Monte-Carlo-Simulation und Adaptive-Importance-Sampling, vorherzusagen. Dabei führt die geschätzte Wahrscheinlichkeit einer Sicherheitsverletzung für jedes gruppierte logische Szenario zu einer messbaren Sicherheitsvorhersage. Das vorgestellte Framework erlaubt es, die Lücke zwischen wissensbasierten und datengetriebenen Testplattformen zu schließen, um die Wissensbasis für die Abdeckung der Operational Design Domains konsequent zu erweitern. Zusammenfassend zeigen die Ergebnisse den Nutzen und die Herausforderungen des entwickelten Frameworks für messbare Sicherheit durch ein Vertrauensmaß der Risikobewertung. Dies ermöglicht eine kosteneffiziente Erweiterung der Validität der Testdomäne im gesamten Softwareentwicklungsprozess, um die erforderlichen Testabbruchkriterien zu erreichen

Collision and Avoidance Modelling of Autonomous Vehicles using Genetic Algorithm and Neural Network

This thesis is to study the optimisation problems in autonomous vehicles, especially the modelling and optimisation of collision avoidance, and to develop some optimisation algorithms based on genetic algorithms and neural networks to operate autonomous vehicles without any collision. Autonomous vehicles, also called self-driving vehicles or driverless vehicles are completely robotised driving frameworks to allow the vehicle to react to outside conditions within a bunch of calculations to play out the undertakings. This thesis summarised artificial intelligence and optimisation techniques for autonomous driving systems in the literature. The optimisation problems related to autonomous vehicles are categorised into four groups: lane change, motion planner, collision avoidance, and artificial intelligence. A chart had been developed to summarise those research and related optimisation methods to help future researchers in the selection of optimisation methods Collision Avoidance is one of streamlining issues in autonomous vehicles. Several sensors had been used to identify position and dangers and collision avoidance algorithms had been developed to analyse the dangers and to use vehicles to avoid a collision. In this thesis, the current research on collision avoidance has been reviewed and some challenges and future works were presented to select the research direction of this thesis, the aim of this research will be the development of optimisation methods to avoid collisions in a predefined environment. The contributions of this thesis are that (1) a simulation model had been developed using Matlab for collision avoidance and serval scenarios were proposed and experimented with. The sensors are used as the inputs to determine collision in the learning preparation of the algorithm; (2) a neural network was used for collision avoidance of autonomous vehicles; (3) a new method was proposed with the combination of genetic algorithm and neural network. In the proposed frame, the neural network is used for decision making and a genetic algorithm is used for the training of the neural network. The results and experimentation show that the proposed strategies are well in the designed environment

Ergonomics of intelligent vehicle braking systems

The present thesis examines the quantitative characteristics of driver braking and pedal operation and discusses the implications for the design of braking support systems for vehicles. After the current status of the relevant research is presented through a literature review, three different methods are employed to examine driver braking microscopically, supplemented by a fourth method challenging the potential to apply the results in an adaptive brake assist system. First, thirty drivers drove an instrumented vehicle for a day each. Pedal inputs were constantly monitored through force, position sensors and a video camera. Results suggested a range of normal braking inputs in terms of brake-pedal force, initial brake-pedal displacement and throttle-release (throttle-off) rate. The inter-personal and intra-personal variability on the main variables was also prominent. [Continues.