Virtual player design using self-learning via competitive coevolutionary algorithms

The Google Artificial Intelligence (AI) Challenge is an international contest the objective of which is to program the AI in a two-player real time strategy (RTS) game. This AI is an autonomous computer program that governs the actions that one of the two players executes during the game according to the state of play. The entries are evaluated via a competition mechanism consisting of two-player rounds where each entry is tested against others. This paper describes the use of competitive coevolutionary (CC) algorithms for the automatic generation of winning game strategies in Planet Wars, the RTS game associated with the 2010 contest. Three different versions of a prime algorithm have been tested. Their common nexus is not only the use of a Hall-of-Fame (HoF) to keep note of the winners of past coevolutions but also the employment of an archive of experienced players, termed the hall-of-celebrities (HoC), that puts pressure on the optimization process and guides the search to increase the strength of the solutions; their differences come from the periodical updating of the HoF on the basis of quality and diversity metrics. The goal is to optimize the AI by means of a self-learning process guided by coevolutionary search and competitive evaluation. An empirical study on the performance of a number of variants of the proposed algorithms is described and a statistical analysis of the results is conducted. In addition to the attainment of competitive bots we also conclude that the incorporation of the HoC inside the primary algorithm helps to reduce the effects of cycling caused by the use of HoF in CC algorithms.This work is partially supported by Spanish MICINN under Project ANYSELF (TIN2011-28627-C04-01),3 by Junta de Andalucía under Project P10-TIC-6083 (DNEMESIS) and by Universidad de Málaga, Campus de Excelencia Internacional Andalucía Tech

Procedural Content Generation for Real-Time Strategy Games

Videogames are one of the most important and profitable sectors in the industry of entertainment. Nowadays, the creation of a videogame is often a large-scale endeavor and bears many similarities with, e.g., movie production. On the central tasks in the development of a videogame is content generation, namely the definition of maps, terrains, non-player characters (NPCs) and other graphical, musical and AI-related components of the game. Such generation is costly due to its complexity, the great amount of work required and the need of specialized manpower. Hence the relevance of optimizing the process and alleviating costs. In this sense, procedural content generation (PCG) comes in handy as a means of reducing costs by using algorithmic techniques to automatically generate some game contents. PCG also provides advantages in terms of player experience since the contents generated are typically not fixed but can vary in different playing sessions, and can even adapt to the player herself. For this purpose, the underlying algorithmic technique used for PCG must be also flexible and adaptable. This is the case of computational intelligence in general and evolutionary algorithms in particular. In this work we shall provide an overview of the use of evolutionary intelligence for PCG, with special emphasis on its use within the context of real-time strategy games. We shall show how these techniques can address both playability and aesthetics, as well as improving the game AI

Coevolutionary Approaches to Generating Robust Build-Orders for Real-Time Strategy Games

We aim to find winning build-orders for Real-Time Strategy games. Real-Time Strategy games provide a variety of challenges, from short-term control to longer term planning. We focus on a longer-term planning problem; which units to build and in what order to produce the units so a player successfully defeats the opponent. Plans which address unit construction scheduling problems in Real-Time Strategy games are called build-orders. A robust build-order defeats many opponents, while a strong build-order defeats opponents quickly. However, no single build-order defeats all other build-orders, and build-orders that defeat many opponents may still lose against a specific opponent. Other researchers have only investigated generating build-orders that defeat a specific opponent, rather than finding robust, strong build-orders. Additionally, previous research has not applied coevolutionary algorithms towards generating build-orders. In contrast, our research has three main contributions towards finding robust, strong build-orders. First, we apply a coevolutionary algorithm towards finding robust build-orders. Compared to exhaustive search, a genetic algorithm finds the strongest build-orders while a coevolutionary algorithm finds more robust build-orders. Second, we show that case-injection enables coevolution to learn from specific opponents while maintaining robustness. Build-orders produced with coevolution and case-injection learn to defeat or play like the injected build-orders. Third, we show that coevolved build-orders benefit from a representation which includes branches and loops. Coevolution will utilize multiple branches and loops to create build-orders that are stronger than build-orders without loops and branches. We believe this work provides evidence that coevolutionary algorithms may be a viable approach to creating robust, strong build-orders for Real-Time Strategy games

Game Artificial Intelligence: Challenges for the Scientific Community

This paper discusses some of the most interesting challenges to which the games research community members may face in the área of the application of arti cial or computational intelligence techniques to the design and creation of video games. The paper focuses on three lines that certainly will in uence signi cantly the industry of game development in the near future, speci cally on the automatic generation of content, the a ective computing applied to video games and the generation of behaviors that manage the decisions of entities not controlled by the human player.Universidad de Málaga. Campus de Excelencia Internacional Andalucía Tech

Coevolutionary algorithms for the optimization of strategies for red teaming applications

Red teaming (RT) is a process that assists an organization in finding vulnerabilities in a system whereby the organization itself takes on the role of an “attacker” to test the system. It is used in various domains including military operations. Traditionally, it is a manual process with some obvious weaknesses: it is expensive, time-consuming, and limited from the perspective of humans “thinking inside the box”. Automated RT is an approach that has the potential to overcome these weaknesses. In this approach both the red team (enemy forces) and blue team (friendly forces) are modelled as intelligent agents in a multi-agent system and the idea is to run many computer simulations, pitting the plan of the red team against the plan of blue team. This research project investigated techniques that can support automated red teaming by conducting a systematic study involving a genetic algorithm (GA), a basic coevolutionary algorithm and three variants of the coevolutionary algorithm. An initial pilot study involving the GA showed some limitations, as GAs only support the optimization of a single population at a time against a fixed strategy. However, in red teaming it is not sufficient to consider just one, or even a few, opponent‟s strategies as, in reality, each team needs to adjust their strategy to account for different strategies that competing teams may utilize at different points. Coevolutionary algorithms (CEAs) were identified as suitable algorithms which were capable of optimizing two teams simultaneously for red teaming. The subsequent investigation of CEAs examined their performance in addressing the characteristics of red teaming problems, such as intransitivity relationships and multimodality, before employing them to optimize two red teaming scenarios. A number of measures were used to evaluate the performance of CEAs and in terms of multimodality, this study introduced a novel n-peak problem and a new performance measure based on the Circular Earth Movers‟ Distance. Results from the investigations involving an intransitive number problem, multimodal problem and two red teaming scenarios showed that in terms of the performance measures used, there is not a single algorithm that consistently outperforms the others across the four test problems. Applications of CEAs on the red teaming scenarios showed that all four variants produced interesting evolved strategies at the end of the optimization process, as well as providing evidence of the potential of CEAs in their future application in red teaming. The developed techniques can potentially be used for red teaming in military operations or analysis for protection of critical infrastructure. The benefits include the modelling of more realistic interactions between the teams, the ability to anticipate and to counteract potentially new types of attacks as well as providing a cost effective solution

Algoritmos coevolutivos competitivos para la programación de inteligencia artificial en videojuegos

Los juegos que han sido objeto de experimentación en esta investigación son del género estrategia en tiempo real (RTS, por sus siglas en inglés: Real Time Strategy Games). Se escogió este género porque ofrece una amplia variedad de problemas muy interesantes en el ámbito de la investigación de la IA aplicada, tales como, la planificación de tareas y recursos en tiempo real teniendo en cuenta al adversario y la toma de decisiones con incertidumbre. Es por esto que es uno de los géneros de juego que más se ha empleado en las investigaciones de IA publicadas en la bibliografía científica. Son dos los juegos RTS sobre los que se ha experimentado en esta tesis. En ambos se han probado los dos enfoques diseñados en esta investigación. Uno de ellos tiene como objetivo la creación de estrategias de IA para manejar los bot y/o los jugadores virtuales (NPCs, por sus siglas en inglés: Non-Player Character) de los juegos RTS antes mencionados. Y el otro, tiene un objetivo mucho más complejo, el de generar de forma simultánea, estrategias de IA y contenido para videojuegos. Este segundo modelo representa el aporte más novedoso de esta investigación, pues hasta donde hemos podido investigar en la bibliografía relacionada, es la primera vez que se logra la coevolución de lo que vendrían siendo (teniendo en cuenta la teoría de la evolución) dos especies tan distintas. Además, constituye un paso importante hacia la creación procedural de juegos completos a nivel de Diseño. Nuestros algoritmos pueden verse como métodos procedurales de generación de contenidos que permiten crear inteligencias artificiales (o sea, comportamientos) de los NPCs y también contenidos (en forma de mapas) del juego. Esta tesis presenta el primer algoritmo conocido en la literatura científica (hasta donde nuestros conocimientos alcanzan) que es capaz de crear proceduralmente ambos contenidos a la vez; y propone un esquema genérico que puede aplicarse a otros videojuegos (no sólo a RTS). A lo largo de la memoria son explicadas detalladamente las propuestas de soluciones que han sido definidas en esta tesis y se aporta el correspondiente análisis de sus resultados para poder valorar seriamente su idoneidad.Esta investigación trata sobre la aplicación del enfoque competitivo de la Coevolución en la programación de soluciones de Inteligencia Artificial (IA) para videojuegos y propone diferentes modelos para abordarlo. La motivación principal de esta tesis es que la Coevolución resulta un enfoque muy adecuado para explotarlo como método de búsqueda y optimización en videojuegos ya que estos son de por sí escenarios intrínsecamente competitivos. Está demostrado que en los contextos donde resulta complejo definir una medida de calidad de las soluciones, aplicar la Coevolución es muy conveniente porque se podría calcular el fitness de los individuos a partir de sus resultados en las competiciones contra las poblaciones adversarias. Este mecanismo de evaluación es tradicional en los videojuegos basados en competición, donde la calidad de un jugador está determinada por su desempeño frente a los oponentes. Por otra parte, además de las muchas muestras que existen de que la Coevolución es un método de optimización muy adecuado en videojuegos, también se han dedicado varios trabajos en la comunidad científica a explicar las “patologías” que afectan a los modelos coevolutivos. Incluso se han propuesto “remedios” para combatirlas, pero a pesar de eso, la complejidad que genera la dinámica de estos modelos hace que sea difícil evadir esos fallos. Este problema es también una de las motivaciones que han guiado esta investigación pues los algoritmos que hemos diseñado pretenden disminuir el riesgo de padecer algunos de los fallos típicos de un proceso coevolutivo. Nuestra apuesta en este sentido ha sido el empleo de los conocidos métodos de archivo y el diseño de tres enfoques que mejoran los resultados del modo de uso tradicional de dichos métodos

Co-evolutionary and Reinforcement Learning Techniques Applied to Computer Go players

The objective of this thesis is model some processes from the nature as evolution and co-evolution, and proposing some techniques that can ensure that these learning process really happens and useful to solve some complex problems as Go game. The Go game is ancient and very complex game with simple rules which still is a challenge for the Artificial Intelligence. This dissertation cover some approaches that were applied to solve this problem, proposing solve this problem using competitive and cooperative co-evolutionary learning methods and other techniques proposed by the author. To study, implement and prove these methods were used some neural networks structures, a framework free available and coded many programs. The techniques proposed were coded by the author, performed many experiments to find the best configuration to ensure that co-evolution is progressing and discussed the results. Using co-evolutionary learning processes can be observed some pathologies which could impact co-evolution progress. In this dissertation is introduced some techniques to solve pathologies as loss of gradients, cycling dynamics and forgetting. According to some authors, one solution to solve these co-evolution pathologies is introduce more diversity in populations that are evolving. In this thesis is proposed some techniques to introduce more diversity and some diversity measurements for neural networks structures to monitor diversity during co-evolution. The genotype diversity evolved were analyzed in terms of its impact to global fitness of the strategies evolved and their generalization. Additionally, it was introduced a memory mechanism in the network neural structures to reinforce some strategies in the genes of the neurons evolved with the intention that some good strategies learned are not forgotten. In this dissertation is presented some works from other authors in which cooperative and competitive co-evolution has been applied. The Go board size used in this thesis was 9x9, but can be easily escalated to more bigger boards.The author believe that programs coded and techniques introduced in this dissertation can be used for other domains

Bayesian opponent modeling in adversarial game environments.

This thesis investigates the use of Bayesian analysis upon an opponent¿s behaviour in order to determine the desired goals or strategy used by a given adversary. A terrain analysis approach utilising the A* algorithm is investigated, where a probability distribution between discrete behaviours of an opponent relative to a set of possible goals is generated. The Bayesian analysis of agent behaviour accurately determines the intended goal of an opponent agent, even when the opponent¿s actions are altered randomly. The environment of Poker is introduced and abstracted for ease of analysis. Bayes¿ theorem is used to generate an effective opponent model, categorizing behaviour according to its similarity with known styles of opponent. The accuracy of Bayes¿ rule yields a notable improvement in the performance of an agent once an opponent¿s style is understood. A hybrid of the Bayesian style predictor and a neuroevolutionary approach is shown to lead to effective dynamic play, in comparison to agents that do not use an opponent model. The use of recurrence in evolved networks is also shown to improve the performance and generalizability of an agent in a multiplayer environment. These strategies are then employed in the full-scale environment of Texas Hold¿em, where a betting round-based approach proves useful in determining and counteracting an opponent¿s play. It is shown that the use of opponent models, with the adaptive benefits of neuroevolution aid the performance of an agent, even when the behaviour of an opponent does not necessarily fit within the strict definitions of opponent ¿style¿.Engineering and Physical Sciences Research Council (EPSRC

A Comparison of Self-Play Algorithms Under a Generalized Framework

Throughout scientific history, overarching theoretical frameworks have allowed researchers to grow beyond personal intuitions and culturally biased theories. They allow to verify and replicate existing findings, and to link is connected results. The notion of self-play, albeit often cited in multiagent Reinforcement Learning, has never been grounded in a formal model. We present a formalized framework, with clearly defined assumptions, which encapsulates the meaning of self-play as abstracted from various existing self-play algorithms. This framework is framed as an approximation to a theoretical solution concept for multiagent training. On a simple environment, we qualitatively measure how well a subset of the captured self-play methods approximate this solution when paired with the famous PPO algorithm. We also provide insights on interpreting quantitative metrics of performance for self-play training. Our results indicate that, throughout training, various self-play definitions exhibit cyclic policy evolutions