SCALING REINFORCEMENT LEARNING THROUGH FEUDAL MULTI-AGENT HIERARCHY

Militaries conduct wargames for training, planning, and research purposes. Artificial intelligence (AI) can improve military wargaming by reducing costs, speeding up the decision-making process, and offering new insights. Previous researchers explored using reinforcement learning (RL) for wargaming based on the successful use of RL for other human competitive games. While previous research has demonstrated that an RL agent can generate combat behavior, those experiments have been limited to small-scale wargames. This thesis investigates the feasibility and acceptability of -scaling hierarchical reinforcement learning (HRL) to support integrating AI into large military wargames. Additionally, this thesis also investigates potential complications that arise when replacing the opposing force with an intelligent agent by exploring the ways in which an intelligent agent can cause a wargame to fail. The resources required to train a feudal multi-agent hierarchy (FMH) and a standard RL agent and their effectiveness are compared in increasingly complicated wargames. While FMH fails to demonstrate the performance required for large wargames, it offers insight for future HRL research. Finally, the Department of Defense verification, validation, and accreditation process is proposed as a method to ensure that any future AI application applied to wargames are suitable.Lieutenant Colonel, United States ArmyApproved for public release. Distribution is unlimited

Game Theory and Prescriptive Analytics for Naval Wargaming Battle Management Aids

NPS NRP Technical ReportThe Navy is taking advantage of advances in computational technologies and data analytic methods to automate and enhance tactical decisions and support warfighters in highly complex combat environments. Novel automated techniques offer opportunities to support the tactical warfighter through enhanced situational awareness, automated reasoning and problem-solving, and faster decision timelines. This study will investigate how game theory and prescriptive analytics methods can be used to develop real-time wargaming capabilities to support warfighters in their ability to explore and evaluate the possible consequences of different tactical COAs to improve tactical missions. This study will develop a conceptual design of a real-time tactical wargaming capability. This study will explore data analytic methods including game theory, prescriptive analytics, and artificial intelligence (AI) to evaluate their potential to support real-time wargaming.N2/N6 - Information WarfareThis research is supported by funding from the Naval Postgraduate School, Naval Research Program (PE 0605853N/2098). https://nps.edu/nrpChief of Naval Operations (CNO)Approved for public release. Distribution is unlimited.

APPLICATION OF AN ARTIFICIAL INTELLIGENCE-ENABLED REAL-TIME WARGAMING SYSTEM FOR NAVAL TACTICAL OPERATIONS

The Navy is taking advantage of advances in computational technologies and data analytic methods to automate and enhance tactical decisions to support warfighters in highly complex combat environments. Novel automated techniques offer opportunities for tactical warfighter support through enhanced situational awareness, automated reasoning and problem-solving, and faster decision timelines. This capstone project investigated the use of artificial Intelligence and game theory to develop real-time wargaming capabilities to enhance warfighters in their ability to explore and evaluate the possible consequences of different tactical COAs to improve tactical missions. This project applied a systems analysis approach and developed a conceptual design of a wargaming real-time Artificial Intelligence decision-aid (WRAID) system capability to support the future tactical warfighter. An operational scenario was developed and used to conduct an operational analysis of the WRAID capability. The project identified requirements for the future WRAID capabilities and studied implementation challenges (including ethical) that will need to be addressedNPS Naval Research ProgramThis project was funded in part by the NPS Naval Research Program.Civilian, DoD, NUWCNPTCivilian, Department of the NavyCivilian, Department of the NavyCivilian, Department of the NavyApproved for public release. Distribution is unlimited

Simulation in Military Training: Recent Developments

Training is considered to be one of the key factors in achieving military goal. Growing concernover the increasing training costs, time, risk of life and paucity of training ranges has forced peopleto adopt newer technologies like computer simulation models, simulators and computer wargamesin military training. With the advancement of computer and communication technologies along withthe advent of other newer technologies, these tools have emerged effective and also havesignificantly less operational cost. It is also becoming possible to integrate simulators, simulationand live exercise through networking, resulting into an effective training tool. This paper highlightsthe advancement of simulation technology in military training and also highlights its applications inIndia

Knowledge Acquisition Analytical Games: games for cognitive systems design

Knowledge discovery from data and knowledge acquisition from experts are steps of paramount importance when designing cognitive systems. The literature discusses extensively on the issues related to current knowledge acquisition techniques. In this doctoral work we explore the use of gaming approaches as a knowledge acquisition tools, capitalising on aspects such as engagement, ease of use and ability to access tacit knowledge. More specifically, we explore the use of analytical games for this purpose. Analytical game for decision making is not a new class of games, but rather a set of platform independent simulation games, designed not for entertainment, whose main purpose is research on decision-making, either in its complete dynamic cycle or a portion of it (i.e. Situational Awareness). Moreover, the work focuses on the use of analytical games as knowledge acquisition tools. To this end, the Knowledge Acquisition Analytical Game (K2AG) method is introduced. K2AG is an innovative game framework for supporting the knowledge acquisition task. The framework introduced in this doctoral work was born as a generalisation of the Reliability Game, which on turn was inspired by the Risk Game. More specifically, K2AGs aim at collecting information and knowledge to be used in the design of cognitive systems and their algorithms. The two main aspects that characterise those games are the use of knowledge cards to render information and meta-information to the players and the use of an innovative data gathering method that takes advantage of geometrical features of simple shapes (e.g. a triangle) to easily collect players\u2019 beliefs. These beliefs can be mapped to subjective probabilities or masses (in evidence theory framework) and used for algorithm design purposes. However, K2AGs might use also different means of conveying information to the players and to collect data. Part of the work has been devoted to a detailed articulation of the design cycle of K2AGs. More specifically, van der Zee\u2019s simulation gaming design framework has been extended in order to account for the fact that the design cycle steps should be modified to include the different kinds of models that characterise the design of simulation games and simulations in general, namely a conceptual model (platform independent), a design model (platform independent) and one or more implementation models (platform dependent). In addition, the processes that lead from one model to the other have been mapped to design phases of analytical wargaming. Aspects of game validation and player experience evaluation have been addressed in this work. Therefore, based on the literature a set of validation criteria for K2AG has been proposed and a player experience questionnaire for K2AGs has been developed. This questionnaire extends work proposed in the literature, but a validation has not been possible at the time of writing. Finally, two instantiations of the K2AG framework, namely the Reliability Game and the MARISA Game, have been designed and analysed in details to validate the approach and show its potentialities

ENLISTING AI IN COURSE OF ACTION ANALYSIS AS APPLIED TO NAVAL FREEDOM OF NAVIGATION OPERATIONS

Navy Planning Process (NPP) Course of Action (COA) analysis requires time and subject matter experts (SMEs) to function properly. Independent steamers (lone destroyers) can soon find themselves lacking time or more than 1–2 SMEs or both. Artificial Intelligence (AI) techniques implemented in real-time strategy (RTS) wargames can be applied to military wargaming to aid military decision-makers’ COA analysis. Using a deep-Q network (DQN) and the ATLATL wargaming framework, I was able to train AI agents that could operate as the opposing force (OPFOR) commander at both satisfactory and near-optimal levels of performance, after less than 24 hours of training or 500000–learning steps. I also show that under 6 hours or 150000–learning steps does not result in a satisfactory AI admiral capable of playing the role as the OPFOR commander in a similarly sized freedom of navigation operation (FONOP) scenario. Applying these AI techniques can save both time onboard and time for reachback personnel. Training AI admirals as quality OPFOR commanders can enhance the NPP for the entire Navy without adding additional strain and without creating analysis paralysis. The meaningful insights and localized flashpoints revealed through hundreds of thousands of constructive operations and experienced by the crew in live simulation or simulation replays will lead to real world, combat-ready naval forces capable of deterring aggression and maintaining freedom of the seas.Lieutenant, United States NavyApproved for public release. Distribution is unlimited