Testing Method for Multi-UAV Conflict Resolution Using Agent-Based Simulation and Multi-Objective Search

A new approach to testing multi-UAV conflict resolution algorithms is presented. The problem is formulated as a multi-objective search problem with two objectives: finding air traffic encounters that 1) are able to reveal faults in conflict resolution algorithms and 2) are likely to happen in the real world. The method uses agent-based simulation and multi-objective search to automatically find encounters satisfying these objectives. It describes pairwise encounters in three-dimensional space using a parameterized geometry representation, which allows encounters involving multiple UAVs to be generated by combining several pairwise encounters. The consequences of the encounters, given the conflict resolution algorithm, are explored using a fast-time agent-based simulator. To find encounters meeting the two objectives, a genetic algorithm approach is used. The method is applied to test ORCA-3D, a widely cited open-source multi-UAV conflict resolution algorithm, and the method’s performance is compared with a plausible random testing approach. The results show that the method can find the required encounters more efficiently than the random search. The identified safety incidents are then the starting points for understanding limitations of the conflict resolution algorithm

Supporting Validation of UAV Sense-and-Avoid Algorithms with Agent-Based Simulation and Evolutionary Search

A Sense-and-Avoid (SAA) capability is required for the safe integration of Unmanned Aerial Vehicles (UAVs) into civilian airspace. Given their safety-critical nature, SAA algorithms must undergo rigorous verification and validation before deployment. The validation of UAV SAA algorithms requires identifying challenging situations that the algorithms have difficulties in handling. By building on ideas from Search-Based Software Testing, this thesis proposes an evolutionary-search-based approach that automatically identifies such situations to support the validation of SAA algorithms. Specifically, in the proposed approach, the behaviours of UAVs under the control of selected SAA algorithms are examined with agent-based simulations. Evolutionary search is used to guide the simulations to focus on increasingly challenging situations in a large search space defined by (the variations of) parameters that configure the simulations. An open-source tool has been developed to support the proposed approach so that the process can be partially automated. Positive results were achieved in a preliminary evaluation of the proposed approach using a simple two-dimensional SAA algorithm. The proposed approach was then further demonstrated and evaluated using two case studies, applying it to a prototype of an industry-level UAV collision avoidance algorithm (specifically, ACAS XU) and a multi-UAV conflict resolution algorithm (specifically, ORCA-3D). In the case studies, the proposed evolutionary-search-based approach was empirically compared with some plausible rivals (specifically, random-search-based approaches and a deterministic-global-search-based approach). The results show that the proposed approach can identify the required challenging situations more effectively and efficiently than the random-search-based approaches. The results also show that even though the proposed approach is a little less competitive than the deterministic-global-search-based approach in terms of effectiveness in relatively easy cases, it is more effective and efficient in more difficult cases, especially when the objective function becomes highly discontinuous. Thus, the proposed evolutionary-search-based approach has the potential to be used for supporting the validation of UAV SAA algorithms although it is not possible to show that it is the best approach

The Discovery and Quantification of Risk in High Dimensional Search Spaces

{kester.clegg l rob.alexander