Predictive Models and Monte Carlo Tree Search: A Pipeline for Believable Agents

Developing and assessing believable agents remains a sought out challenge. Recently, research has approached this problem by treating and assessing believability as a time-continuous phenomenon, learning from collected data to predict believability of games and game states. Our study will build on this work: by integrating this believability model with a game agent to affect its behaviour. In this short paper, we first describe our methodology and then the results obtained from our user study, which suggests that this methodology can help creating more believable agents, opening the possibility of integrating this type of models into game development. We also discuss the limitations of this approach, possible variants to tackle these, and ideas for future work to extend this preliminary work

Balancing Wargames through Predicting Unit Point Costs

In tactical wargames, such as Warhammer 40K, two or more players control asymmetrical armies that include multiple units of different types and strengths. In these type of games, unit are assigned point costs, which are used to ensure that all players will control armies of similar strength. Players are provided with a total budget of points they can spend to purchase units that will be part of their army lists. Calculating the point value of individual units is a tedious manual process, which often requires long play-testing sessions and iterations of adjustments. In this paper, we propose an automated way of predicting these point costs using a linear regression approach. We use a multi-unit, turn-based, non-balanced game that has three asymmetric armies. We use Monte Carlo Tree Search agents to simulate the players, using different heuristics to emulate playing strategies. We present six different variants of our unit-point prediction algorithm, and we show how our best variant is able to almost reduce the unbalanced nature of the game by half

General Video Game AI: Learning from screen capture

General Video Game Artificial Intelligence is a general game playing framework for Artificial General Intelligence research in the video-games domain. In this paper, we propose for the first time a screen capture learning agent for General Video Game AI framework. A Deep Q-Network algorithm was applied and improved to develop an agent capable of learning to play different games in the framework. After testing this algorithm using various games of different categories and difficulty levels, the results suggest that our proposed screen capture learning agent has the potential to learn many different games using only a single learning algorithm

Fingerprinting Tabletop Games

We present some initial work on characterizing games using a visual 'fingerprint' generated from several independent optimisation runs over the parameters used in Monte Carlo Tree Search (MCTS). This 'fingerprint' provides a useful tool to compare games, as well as highlighting the relative sensitivity of a specific game to algorithmic variants of MCTS. The exploratory work presented here shows that in some games there is a major change in the optimal MCTS parameters when we move from 2-players to 3 or 4-players

Learning on a Budget via Teacher Imitation

Deep Reinforcement Learning (RL) techniques can benefit greatly from leveraging prior experience, which can be either self-generated or acquired from other entities. Action advising is a framework that provides a flexible way to transfer such knowledge in the form of actions between teacher-student peers. However, due to the realistic concerns, the number of these interactions is limited with a budget; therefore, it is crucial to perform these in the most appropriate moments. There have been several promising studies recently that address this problem setting especially from the student's perspective. Despite their success, they have some shortcomings when it comes to the practical applicability and integrity as an overall solution to the learning from advice challenge. In this paper, we extend the idea of advice reusing via teacher imitation to construct a unified approach that addresses both advice collection and advice utilisation problems. We also propose a method to automatically tune the relevant hyperparameters of these components on-the-fly to make it able to adapt to any task with minimal human intervention. The experiments we performed in 5 different Atari games verify that our algorithm either surpasses or performs on-par with its top competitors while being far simpler to be employed. Furthermore, its individual components are also found to be providing significant advantages alone

Bandit-based Random Mutation Hill-Climbing

The Random Mutation Hill-Climbing algorithm is a direct search technique mostly used in discrete domains. It repeats the process of randomly selecting a neighbour of a best-so-far solution and accepts the neighbour if it is better than or equal to it. In this work, we propose to use a novel method to select the neighbour solution using a set of independent multi-armed bandit-style selection units which results in a bandit-based Random Mutation Hill-Climbing algorithm. The new algorithm significantly outperforms Random Mutation Hill-Climbing in both OneMax (in noise-free and noisy cases) and Royal Road problems (in the noise-free case). The algorithm shows particular promise for discrete optimisation problems where each fitness evaluation is expensive

Diversity maintenance using a population of repelling random-mutation hill climbers

A novel evolutionary algorithm, which can be viewed as an extension to the simple, yet effective, approach of the Random-Mutation Hill Climber (RMHC), is presented. The algorithm addresses the shortcomings of RMHC and its multi-individual parallel version through the introduction of a penalty term into the fitness function, which penalizes individuals in the population for being too similar, hence maintaining population diversity. The performance of the algorithm is evaluated on the deceptive trap and a set of SAT problems, comparing them to the Crowding EA. The results show that at a small cost of solution speed on simpler problems, the algorithm gains better capabilities of dealing with the issues of local maxima

Population seeding techniques for Rolling Horizon Evolution in General Video Game Playing

While Monte Carlo Tree Search and closely related methods have dominated General Video Game Playing, recent research has demonstrated the promise of Rolling Horizon Evolutionary Algorithms as an interesting alternative. However, there is little attention paid to population initialization techniques in the setting of general real-time video games. Therefore, this paper proposes the use of population seeding to improve the performance of Rolling Horizon Evolution and presents the results of two methods, One Step Look Ahead and Monte Carlo Tree Search, tested on 20 games of the General Video Game AI corpus with multiple evolution parameter values (population size and individual length). An in-depth analysis is carried out between the results of the seeding methods and the vanilla Rolling Horizon Evolution. In addition, the paper presents a comparison to a Monte Carlo Tree Search algorithm. The results are promising, with seeding able to boost performance significantly over baseline evolution and even match the high level of play obtained by the Monte Carlo Tree Search

Automatic Goal Discovery in Subgoal Monte Carlo Tree Search

Monte Carlo Tree Search (MCTS) is a heuristic search algorithm that can play a wide range of games without requiring any domain-specific knowledge. However, MCTS tends to struggle in very complicated games due to an exponentially increasing branching factor. A promising solution for this problem is to focus the search only on a small fraction of states. Subgoal Monte Carlo Tree Search (S-MCTS) achieves this by using a predefined subgoal-predicate that detects promising states called subgoals. However, not only does this make S-MCTS domain-dependent, but also it is often difficult to define a good predicate. In this paper, we propose using quality diversity (QD) algorithms to detect subgoals in real-time. Furthermore, we show how integrating QD-algorithms into S-MCTS significantly improves its performance in the Physical Travelling Salesmen Problem without requiring any domain-specific knowledge