Unmanned and Autonomous Systems of Systems Test and Evaluation: Challenges and Opportunities

The introduction of Unmanned and Autonomous Systems (UAS) brings substantial, interesting, and in many cases, new challenges to the Department of Defense’s Test and Evaluation community. The test and evaluation of UASs becomes significantly more complicated than traditional systems, especially as we approach more fully autonomous systems and need to test integrated systems of systems in joint military operational testing environments. Compounding the multi-faceted considerations involved in test and evaluation, systems have continuously increasing complexity and capabilities and can be at different maturity levels. Emergent properties, particularly those that are unplanned and undesired, also need to be considered. Challenges identified by the Unmanned and Autonomous Systems Test community and related to the test and evaluation of the UASs are discussed. This paper presents various approaches for addressing these challenges including an innovative Prescriptive and Adaptive Testing Framework and decision support system, PATFrame

Aircraft-sized anechoic chambers for electronic warfare, radar and other electromagnetic engineering evaluation

This paper considers capabilities and benefits of aircraft-sized radio/radar frequency anechoic chambers for Test and Evaluation (T&E) of Electronic Warfare (EW), radar and other electromagnetics aspects of air and ground platforms. There are few such chambers worldwide. Initially developed to reduce costs, timescales and risks associated with open-air range flight testing of EW systems, their utility has expanded to most areas of platforms’ electromagnetics’ T&E. A key feature is the ability to conduct T&E of nationally sensitive equipment and systems, fully installed on platforms, in absolute privacy. Chambers’ capabilities and uses are described, with emphasis on key infrastructure and instrumentation. Non-EW uses are identified and selected topics elaborated. Operation and maintenance are discussed, based on experiential knowledge from international use and the authors’ 30 years’ involvement with BAE Systems’ EW Test Facility. A view is provided of trends and challenges whose resolution could further increase chamber utility. National affordability challenges also suggest utility expansion to support continuing moves, from expensive and difficult to repeat flight test and operational evaluation trials, towards an affordability-driven optimal balance between modelling and simulation, and real-world testing of platforms

Acquisition Challenges of Autonomous Systems

The Department of Defense has stated publicly that future defense capabilities will depend strongly on autonomous systems;systems that make sophisticated judgments about the world and choose appropriate courses of action, and perhaps even adapt and learn over time. Developing and deploying such systems poses more than just a technical challenge in robotics and artificial intelligence;it also poses many challenges to the acquisition process and workforce. From cost estimation to sustainment planning, every aspect of acquisition will be affected. Test and evaluation, in particular, may require not only novel methodologies and resources, but organizational and process changes as well.Naval Postgraduate School Acquisition Research Progra

Full-Scaled Advanced Systems Testbed: Ensuring Success of Adaptive Control Research Through Project Lifecycle Risk Mitigation

The National Aeronautics and Space Administration's Dryden Flight Research Center completed flight testing of adaptive controls research on the Full-Scale Advance Systems Testbed (FAST) in January of 2011. The research addressed technical challenges involved with reducing risk in an increasingly complex and dynamic national airspace. Specific challenges lie with the development of validated, multidisciplinary, integrated aircraft control design tools and techniques to enable safe flight in the presence of adverse conditions such as structural damage, control surface failures, or aerodynamic upsets. The testbed is an F-18 aircraft serving as a full-scale vehicle to test and validate adaptive flight control research and lends a significant confidence to the development, maturation, and acceptance process of incorporating adaptive control laws into follow-on research and the operational environment. The experimental systems integrated into FAST were designed to allow for flexible yet safe flight test evaluation and validation of modern adaptive control technologies and revolve around two major hardware upgrades: the modification of Production Support Flight Control Computers (PSFCC) and integration of two, fourth-generation Airborne Research Test Systems (ARTS). Post-hardware integration verification and validation provided the foundation for safe flight test of Nonlinear Dynamic Inversion and Model Reference Aircraft Control adaptive control law experiments. To ensure success of flight in terms of cost, schedule, and test results, emphasis on risk management was incorporated into early stages of design and flight test planning and continued through the execution of each flight test mission. Specific consideration was made to incorporate safety features within the hardware and software to alleviate user demands as well as into test processes and training to reduce human factor impacts to safe and successful flight test. This paper describes the research configuration, experiment functionality, overall risk mitigation, flight test approach and results, and lessons learned of adaptive controls research of the Full-Scale Advanced Systems Testbed

An expert system for checking the correctness of memory systems using simulation and metamorphic testing

During the last few years, computer performance has reached a turning point where computing power is no longer the only important concern. This way, the emphasis is shifting from an exclusive focus on the optimisation of the computing system to optimising other systems, like the memory system. Broadly speaking, testing memory systems entails two main challenges: the oracle problem and the reliable test set problem. The former consists in deciding if the outputs of a test suite are correct. The latter refers to providing an appropriate test suite for determining the correctness of the system under test. In this paper we propose an expert system for checking the correctness of memory systems. In order to face these challenges, our proposed system combines two orthogonal techniques – simulation and metamorphic testing – enabling the automatic generation of appropriate test cases and deciding if their outputs are correct. In contrast to conventional expert systems, our system includes a factual database containing the results of previous simulations, and a simulation platform for computing the behaviour of memory systems. The knowledge of the expert is represented in the form of metamorphic relations, which are properties of the analysed system involving multiple inputs and their outputs. Thus, the main contribution of this work is two-fold: a method to automatise the testing process of memory systems, and a novel expert system design focusing on increasing the overall performance of the testing process. To show the applicability of our system, we have performed a thorough evaluation using 500 memory configurations and 4 di erent memory management algorithms, which entailed the execution of more than one million of simulations. The evaluation used mutation testing, injecting faults in the memory management algorithms. The developed expert system was able to detect over 99% of the critical injected faults, hence obtaining very promising results, and outperforming other standard techniques like random testingThis work was supported by the Spanish Ministerio de Economía, Industria y Competitividad, Gobierno de España/FEDER (grant numbers DArDOS, TIN2015-65845-C3-1-R and FAME, RTI2018-093608-B-C31) and the Comunidad de Madrid project FORTE under Grant S2018/TCS-4314. The first author is also supported by the Universidad Complutense de Madrid - Santander Universidades grant (CT17/17-CT18/17

Development, Test and Evaluation of Autonomous Unmanned Aerial Systems in A Simulated Wide Area Search Scenario: An Implementation of the Autonomous Systems Reference Architecture

The implementation and testing of autonomous and cooperative unmanned systems is challenging due to the inherent design complexity, infinite test spaces, and lack of autonomy specific measures. These challenges are limiting the USAF\u27s ability to deploy and take advantage of tactical and strategic advantages offered by these systems. This research instantiates an Autonomous System Reference Architecture (ASRA) on a Wide Area Search (WAS) scenario as a test bed for rapid prototyping and evaluation of autonomous and cooperative systems. This research aims to pro- vide a framework to evaluate the system’s ability to achieve mission and autonomy objectives, develop reusable autonomous behaviors, and develop reusable cooperative decision making algorithms. For this research and application to the WAS mission, metrics of autonomy were derived from literature requirements for autonomous systems implementing reactive architectures and control: responsiveness, robustness, and perception accuracy. Autonomous behaviors, to include more complex behaviors combining simple (atomic) behaviors were developed, and a variety of cooperative decision rules were defined. The subsequent evaluation implemented a face centered cubic design of experiments over four scenarios including a single vehicle, and three levels of cooperation between two vehicles. Following a rigorous test plan, the tests were conducted in simulation implementing automated testing and expedited analysis. The test results were used to create a response surface model to characterize the system and conduct multiple response optimization to determine an optimal configuration that maximizes area searched, percent detected, and perception accuracy in a given target density