The immense effort required for the safety validation of an automated driving system of SAE level 3 or higher is known not to be feasible by real test drives alone. Therefore, simulation is key even for limited operational design domains for homologation of automated driving functions. Consequently, all simulation models used as tools for this purpose must be qualified beforehand. For this, in addition to their verification and validation, uncertainty quantification (VV&UQ) and prediction for the application domain are required for the credibility of the simulation model. To enable such VV&UQ, a particularly developed lidar sensor system simulation is utilized to present new metrics that can be used holistically to demonstrate the model credibility and -maturity for simulation models of active perception sensor systems.
The holistic process towards model credibility starts with the formulation of the requirements for the models. In this context, the threshold values of the metrics as acceptance criteria are quantifiable by the relevance analysis of the cause-effect chains prevailing in different scenarios, and should intuitively be in the same unit as the simulated metric for this purpose. These relationships can be inferred via the presented aligned methods “Perception Sensor Collaborative Effect and Cause Tree” (PerCollECT) and “Cause, Effect, and Phenomenon Relevance Analysis” (CEPRA). For sample validation, each experiment must be accompanied by reference measurements, as these then serve as simulation input. Since the reference data collection is subject to epistemic as well as aleatory uncertainty, which are both propagated through the simulation in the form of input data variation, this leads to several slightly different simulation results. In the simulation of measured signals and data over time considered here, this combination of uncertainties is best expressed as superimposed cumulative distribution functions. The metric must therefore be able to handle such so-called p-boxes as a result of the large set of simulations.
In the present work, the area validation metric (AVM) is selected by a detailed analysis as the best of the metrics already used and extended to be able to fulfill all the requirements. This results in the corrected AVM (CAVM), which quantifies the model scattering error with respect to the real scatter. Finally, the double validation metric (DVM) is elaborated as a double-vector of the former metric with the estimate for the model bias. The novel metric is exemplarily applied to the empirical cumulative distribution functions of lidar measurements and the p-boxes from their re-simulations. In this regard, aleatory and epistemic uncertainties are taken into account for the first time and the novel metrics are successfully established. The quantification of the uncertainties and error prediction of a sensor model based on the sample validation is also demonstrated for the first time
