SCALING REINFORCEMENT LEARNING THROUGH FEUDAL MULTI-AGENT HIERARCHY

Militaries conduct wargames for training, planning, and research purposes. Artificial intelligence (AI) can improve military wargaming by reducing costs, speeding up the decision-making process, and offering new insights. Previous researchers explored using reinforcement learning (RL) for wargaming based on the successful use of RL for other human competitive games. While previous research has demonstrated that an RL agent can generate combat behavior, those experiments have been limited to small-scale wargames. This thesis investigates the feasibility and acceptability of -scaling hierarchical reinforcement learning (HRL) to support integrating AI into large military wargames. Additionally, this thesis also investigates potential complications that arise when replacing the opposing force with an intelligent agent by exploring the ways in which an intelligent agent can cause a wargame to fail. The resources required to train a feudal multi-agent hierarchy (FMH) and a standard RL agent and their effectiveness are compared in increasingly complicated wargames. While FMH fails to demonstrate the performance required for large wargames, it offers insight for future HRL research. Finally, the Department of Defense verification, validation, and accreditation process is proposed as a method to ensure that any future AI application applied to wargames are suitable.Lieutenant Colonel, United States ArmyApproved for public release. Distribution is unlimited

ENLISTING AI IN COURSE OF ACTION ANALYSIS AS APPLIED TO NAVAL FREEDOM OF NAVIGATION OPERATIONS

Navy Planning Process (NPP) Course of Action (COA) analysis requires time and subject matter experts (SMEs) to function properly. Independent steamers (lone destroyers) can soon find themselves lacking time or more than 1–2 SMEs or both. Artificial Intelligence (AI) techniques implemented in real-time strategy (RTS) wargames can be applied to military wargaming to aid military decision-makers’ COA analysis. Using a deep-Q network (DQN) and the ATLATL wargaming framework, I was able to train AI agents that could operate as the opposing force (OPFOR) commander at both satisfactory and near-optimal levels of performance, after less than 24 hours of training or 500000–learning steps. I also show that under 6 hours or 150000–learning steps does not result in a satisfactory AI admiral capable of playing the role as the OPFOR commander in a similarly sized freedom of navigation operation (FONOP) scenario. Applying these AI techniques can save both time onboard and time for reachback personnel. Training AI admirals as quality OPFOR commanders can enhance the NPP for the entire Navy without adding additional strain and without creating analysis paralysis. The meaningful insights and localized flashpoints revealed through hundreds of thousands of constructive operations and experienced by the crew in live simulation or simulation replays will lead to real world, combat-ready naval forces capable of deterring aggression and maintaining freedom of the seas.Lieutenant, United States NavyApproved for public release. Distribution is unlimited

On Wargaming

Wargames are as old as civilization—and perhaps older. In his informative and entertaining Public Broadcasting series Connections, James Burke argued that the first invention, the one that enabled all later inventions, was the plow. It allowed agriculture, and as agriculture permitted denser populations, the frequency of inventions increased, due either to “connecting” with new applications or combining with other inventions to create one that was greater than the sum of its parts.https://digital-commons.usnwc.edu/usnwc-newport-papers/1043/thumbnail.jp

Wargames: Non-Linear Experiential Modes of Historical Knowledge

This project is on wargames, a type of board and digital game concerned with the practice of war, leadership, command, strategy, tactics, and decision-making. The goal of the project is to explore whether wargames, as non-linear experiential forms of historical knowledge have any value in understanding how the public views, understands, and relates to the historical past. Moreover, the goal of the project is to help historical academia understand what wargames are as they have not been studied in any substantial way and open the door for further research projects regarding wargames, or in fact use wargames as a research aid

Knowledge Acquisition Analytical Games: games for cognitive systems design

Knowledge discovery from data and knowledge acquisition from experts are steps of paramount importance when designing cognitive systems. The literature discusses extensively on the issues related to current knowledge acquisition techniques. In this doctoral work we explore the use of gaming approaches as a knowledge acquisition tools, capitalising on aspects such as engagement, ease of use and ability to access tacit knowledge. More specifically, we explore the use of analytical games for this purpose. Analytical game for decision making is not a new class of games, but rather a set of platform independent simulation games, designed not for entertainment, whose main purpose is research on decision-making, either in its complete dynamic cycle or a portion of it (i.e. Situational Awareness). Moreover, the work focuses on the use of analytical games as knowledge acquisition tools. To this end, the Knowledge Acquisition Analytical Game (K2AG) method is introduced. K2AG is an innovative game framework for supporting the knowledge acquisition task. The framework introduced in this doctoral work was born as a generalisation of the Reliability Game, which on turn was inspired by the Risk Game. More specifically, K2AGs aim at collecting information and knowledge to be used in the design of cognitive systems and their algorithms. The two main aspects that characterise those games are the use of knowledge cards to render information and meta-information to the players and the use of an innovative data gathering method that takes advantage of geometrical features of simple shapes (e.g. a triangle) to easily collect players\u2019 beliefs. These beliefs can be mapped to subjective probabilities or masses (in evidence theory framework) and used for algorithm design purposes. However, K2AGs might use also different means of conveying information to the players and to collect data. Part of the work has been devoted to a detailed articulation of the design cycle of K2AGs. More specifically, van der Zee\u2019s simulation gaming design framework has been extended in order to account for the fact that the design cycle steps should be modified to include the different kinds of models that characterise the design of simulation games and simulations in general, namely a conceptual model (platform independent), a design model (platform independent) and one or more implementation models (platform dependent). In addition, the processes that lead from one model to the other have been mapped to design phases of analytical wargaming. Aspects of game validation and player experience evaluation have been addressed in this work. Therefore, based on the literature a set of validation criteria for K2AG has been proposed and a player experience questionnaire for K2AGs has been developed. This questionnaire extends work proposed in the literature, but a validation has not been possible at the time of writing. Finally, two instantiations of the K2AG framework, namely the Reliability Game and the MARISA Game, have been designed and analysed in details to validate the approach and show its potentialities

Game Theory and Prescriptive Analytics for Naval Wargaming Battle Management Aids

NPS NRP Technical ReportThe Navy is taking advantage of advances in computational technologies and data analytic methods to automate and enhance tactical decisions and support warfighters in highly complex combat environments. Novel automated techniques offer opportunities to support the tactical warfighter through enhanced situational awareness, automated reasoning and problem-solving, and faster decision timelines. This study will investigate how game theory and prescriptive analytics methods can be used to develop real-time wargaming capabilities to support warfighters in their ability to explore and evaluate the possible consequences of different tactical COAs to improve tactical missions. This study will develop a conceptual design of a real-time tactical wargaming capability. This study will explore data analytic methods including game theory, prescriptive analytics, and artificial intelligence (AI) to evaluate their potential to support real-time wargaming.N2/N6 - Information WarfareThis research is supported by funding from the Naval Postgraduate School, Naval Research Program (PE 0605853N/2098). https://nps.edu/nrpChief of Naval Operations (CNO)Approved for public release. Distribution is unlimited.

Rerolling Boardgames: Essays on Themes, Systems, Experiences and Ideologies (Studies in Gaming)

Despite the advent and explosion of videogames, boardgames--from fast-paced party games to intensely strategic titles--have in recent years become more numerous and more diverse in terms of genre, ethos and content. The growth of gaming events and conventions such as Essen Spiel, Gen Con and the UK Games EXPO, as well as crowdfunding through sites like Kickstarter, has diversified the evolution of game development, which is increasingly driven by fans, and boardgames provide an important glue to geek culture. In academia, boardgames are used in a practical sense to teach elements of design and game mechanics. Game studies is also recognizing the importance of expanding its focus beyond the digital. As yet, however, no collected work has explored the many different approaches emerging around the critical challenges that boardgaming represents. In this collection, game theorists analyze boardgame play and player behavior, and explore the complex interactions between the sociality, conflict, competition and cooperation that boardgames foster. Game designers discuss the opportunities boardgame system designs offer for narrative and social play. Cultural theorists discuss boardgames' complex history as both beautiful physical artifacts and special places within cultural experiences of play

Developing a Serious Game to Explore Joint All Domain Command and Control

Changes in the geopolitical landscape and increasing technological complexity have prompted the U.S. Military to coin Multi-Domain Operations (MDO) and Joint All-Domain Command and Control as terms to describe an over-arching strategy that frames the complexity of warfare across both traditional and emerging warfighting domains. Teaching new and advanced concepts associated with these terms requires both innovation as well as distinct education and training tools in order to realize the cultural change advocated by senior military leaders. BSN, a Collectible Card Game, was developed to teach concepts integral to MDO and initiate discussion on military strategy