StarCraft Bots and Competitions

International audienceDefinition Real-Time Strategy (RTS) games is a sub-genre of strategy games where players need to build an economy (gathering resources and building a base) and military power (training units and researching technologies) in order to defeat their opponents (destroying their army and base). Artificial Intelligence (AI) problems related to RTS games deal with the behavior of an artificial player. Since 2010, many international competitions have been organized to match AIs, or bots, playing to the RTS game StarCraft. This chapter presents a review of all major international competitions from 2010 until 2015, and details some competing StarCraft bots. State of the Art Bots for StarCraft Thanks to the recent organization of international game AI competitions fo-cused around the popular StarCraft game, several groups have been working on integrating many of the techniques developed for RTS game AI into complete "bots", capable of playing complete StarCraft games. In this chapter we will overview some of the currently available top bots, and their results of recent competitions

Q-learnings in RTs game's micro-management

Treballs Finals de Grau d'Enginyeria Informàtica, Facultat de Matemàtiques, Universitat de Barcelona, Any: 2015, Director: Jesús Cerquides BuenoThe purpose of this Project is to implement the one-step Q-Learning algorithm and a similar version using linear function approximation in a combat scenario in the Real-Time Strategy game Starcraft: BroodwarTM. First, there is a brief description of Real-Time Strategy games, and particularly about Starcraft, and some of the work done in the field of Reinforcement Learning. After the introduction and previous work are covered, a description of the Reinforcement Learning problem in Real-Time Strategy games is shown. Then, the development of the Reinforcement Learning agents using Q-Learning and Approximate Q-Learning is explained. It is divided into three phases: the first phase consists of defining the task that the agents must solve as a Markov Decision Process and implementing the Reinforcement Learning agents. The second phase is the training period: the agents have to learn how to destroy the rival units and avoid being destroyed in a set of training maps. This will be done through exploration because the agents have no prior knowledge of the outcome of the available actions. The third and last phase is testing the agents’ knowledge acquired in the training period in a different set of maps, observing the results and finally comparing which agent has performed better. The expected behavior is that both Q-Learning agents will learn how to kite (attack and flee) in any combat scenario. Ultimately, this behavior could become the micro-management portion of a new Bot or could be added to an existing bot

MCTS library for unit movement planning in real-time strategy game StarCraft

Existuje společnost vývojářů umělé inteligence, kteří zkouší své nápady a pilně pracují, aby vytvořili neporazitelného protivníka pro živou strategickou hru Starcraft, což dokázali v Šachách a Go. Tato práce předvádí využití knihovny pro Monte CarloTree Search Considering durations algorytmus, který byl prvně navrhnut Albertem Uriarte a Santagem Ontańon z Drexelské univerzity. Tento algorytmus prokazuje vynikající výsledky v řízení armády v živých strategických hrách. Jako menší náhradu přidáme do knihovny vyhledávání Negamax. Naše využití algorytmu je vypracováno jako statická knihovna S++, která může být připojena k jakémukoli možnému botovi. Instalace je velmi jednoduchá a nenáročná. V průběhu práce vyhodnocujeme algoritmy, porovnáváme je a demonstrujeme jejich využití. Tyto algoritmy jsou založeny a testovány na platformě UAlberta bot.There is a live community of AI developers that are trying their ideas and putting effort to create an unbeatable rival for real-time strategy game Starcraft, as it was done with Chess and Go. This work presents an implementation of the library for the Monte Carlo Tree Search Considering Durations algorithm, that was recently proposed by Alberto Uriarte and Santiago Onta~n´on from Drexel University. It appears to bring outstanding results in real-time strategy army control. As a smaller substitute, we add a Negamax search to the library. Our implementation of the algorithms is designed as a static C++ library, which could be easily plugged in-to any possible bot. The setup is simple and intuitive. During the course of the work we evaluate the algorithms, compare them and demonstrate their usage. The algorithms are based and tested on UAlberta bot framework

Using influence maps with heuristic search to craft sneak-attacks in Starcraft

Real-Time Strategy (RTS) games have consistently been popular among AI researchers over the past couple of decades due to their complexity and difficulty to play for both humans and AI. A popular strategy in RTS games is a “Sneak-Attack,” where one player tries to maneuver some of their units into the base of their enemy without being seen for as long as possible to surprise their enemy and deal massive damage to their economy. This thesis introduces a novel method for finding Sneak-Attack paths in StarCraft: Brood War by combining influence maps with heuristic search. The combined system creates paths that can guide units effectively - and automatically - into the enemy’s base, by avoiding enemy unit vision and minimizing both travel distance and unit damage. For StarCraft, this involves guiding a loaded transport ship to the enemy’s base to drop off units for attack. Our results show that the new system performs better than direct paths across a variety of maps in terms of total transport deaths, total damage taken, as well as the total time spent by the transport within enemy vision. We then utilize this new system to demonstrate an alternate use: a proof of concept for calculating building placements to defend against enemy sneak-attacks

Redes neuronales que expresan múltiples estrategias en el videojuego StarCraft 2.

ilustracionesUsing neural networks and supervised learning, we have created models capable of solving problems at a superhuman level. Nevertheless, this training process results in models that learn policies that average the plethora of behaviors usually found in datasets. In this thesis we present and study the Behavioral Repetoires Imitation Learning (BRIL) technique. In BRIL, the user designs a behavior space, the user then projects this behavior space into low coordinates and uses these coordinates as input to the model. Upon deployment, the user can adjust the model to express a behavior by specifying fixed coordinates for these inputs. The main research question ponders on the relationship between the Dimension Reduction algorithm and how much the trained models are able to replicate behaviors. We study three different Dimensionality Reduction algorithms: Principal Component Analysis (PCA), Isometric Feature Mapping (Isomap) and Uniform Manifold Approximation and Projection (UMAP); we design and embed a behavior space in the video game StarCraft 2, we train different models for each embedding and we test the ability of each model to express multiple strategies. Results show that with BRIL we are able to train models that are able to express the multiple behaviors present in the dataset. The geometric structure these methods preserve induce different separations of behaviors, and these separations are reflected in the models' conducts. (Tomado de la fuente)Usando redes neuronales y aprendizaje supervisado, hemos creado modelos capaces de solucionar problemas a nivel súperhumano. Sin embargo, el proceso de entrenamiento de estos modelos es tal que el resultado es una política que promedia todos los diferentes comportamientos presentes en el conjunto de datos. En esta tesis presentamos y estudiamos la técnica Aprendizaje por Imitación de Repertorios de Comportamiento (BRIL), la cual permite entrenar modelos que expresan múltiples comportamientos de forma ajustable. En BRIL, el usuario diseña un espacio de comportamientos, lo proyecta a bajas dimensiones y usa las coordenadas resultantes como entradas del modelo. Para poder expresar cierto comportamiento a la hora de desplegar la red, basta con fijar estas entradas a las coordenadas del respectivo comportamiento. La pregunta principal que investigamos es la relación entre el algoritmo de reducción de dimensionalidad y la capacidad de los modelos entrenados para replicar y expresar las estrategias representadas. Estudiamos tres algoritmos diferentes de reducción de dimensionalidad: Análisis de Componentes Principales (PCA), Mapeo de Características Isométrico (Isomap) y Aproximación y Proyección de Manifolds Uniformes (UMAP); diseñamos y proyectamos un espacio de comportamientos en el videojuego StarCraft 2, entrenamos diferentes modelos para cada embebimiento y probamos la capacidad de cada modelo de expresar múltiples estrategias. Los resultados muestran que, usando BRIL, logramos entrenar modelos que pueden expresar los múltiples comportamientos presentes en el conjunto de datos. La estructura geométrica preservada por cada método de reducción induce diferentes separaciones de los comportamientos, y estas separaciones se ven reflejadas en las conductas de los modelos. (Tomado de la fuente)Maestrí

Component-action deep Q-learning for real-time strategy game AI

Real-time Strategy (RTS) games provide a challenging environment for AI research, due to their large state and action spaces, hidden information, and real-time gameplay. The RTS game StarCraft II has become a new test-bed for deep reinforcement learning (RL) systems using the StarCraft II Learning Environment (SC2LE). Recently the full game of StarCraft II has been approached with a complex multi-agent RL system only possible with extremely large financial investments. In this thesis we will describe existing work in RTS AI and motivate our work adapting the deep Q-learning (DQN) RL algorithm to accommodate the multi-dimensional action-space of the SC2LE. We then present the results of our experiments using custom combat scenarios. First, we compare methods for calculating DQN training loss with action components. Second, we show that policies trained with component-action DQN for five hours perform comparably to scripted policies in smaller scenarios and outperform them in larger scenarios. Third, we explore several ways to transfer policies between scenarios, and show that it is a viable method to reduce training time. We show that policies trained on scenarios with fewer units can be applied to larger scenarios and to scenarios with different unit types with only a small loss in performance