A simulation study of Texas hold 'em poker: what Taylor Swift understands and James Bond doesn't

Recent years have seen a large increase in the popularity of Texas hold ’em poker. It is now the most commonly played variant of the game, both in casinos and through online platforms. In this paper, we present a simulation study for games of Texas hold ’em with between two and 23 players. From these simulations, we estimate the probabilities of each player having been dealt the winning hand. These probabilities are calculated conditional on both partial information (that is, the player only having knowledge of his/her cards) and also on fuller information (that is, the true probabilities of each player winning given knowledge of the cards dealt to each player). Where possible, our estimates are compared to exact analytic results and are shown to have converged to three significant figures. With these results, we assess the poker strategies described in two recent pieces of popular culture. In comparing the ideas expressed in Taylor Swift’s song, New Romantics, and the betting patterns employed by James Bond in the 2006 film, Casino Royale, we conclude that Ms Swift demonstrates a greater understanding of the true probabilities of winning a game of Texas hold ’em poker. doi:10.1017/S144618111800015

A scheme for creating digital entertainment with substance

Computer games constitute a major branch of the entertainment industry nowadays. The financial and research potentials of making games more appealing (or else more interesting) are more than impressive. Interactive and cooperative characters can generate more realism in games and satisfaction for the player. Moreover, on-line (while play) machine learning techniques are able to produce characters with intelligent capabilities useful to any game’s context. On that basis, richer human-machine interaction through real-time entertainment, player and emotional modeling may provide means for effective adjustment of the non-player characters’ behavior in order to obtain games of substantial entertainment. This paper introduces a research scheme for creating NPCs that generate entertaining games which is based interdisciplinary on the aforementioned areas of research and is foundationally supported by several pilot studies on testbed games. Previous work and recent results are presented within this framework.peer-reviewe

Using a high-level language to build a poker playing agent

Tese de mestrado integrado. Engenharia Informática e Computação. Faculdade de Engenharia. Universidade do Porto. 200

Opponent modeling and exploitation in poker using evolved recurrent neural networks

As a classic example of imperfect information games, poker, in particular, Heads-Up No-Limit Texas Holdem (HUNL), has been studied extensively in recent years. A number of computer poker agents have been built with increasingly higher quality. While agents based on approximated Nash equilibrium have been successful, they lack the ability to exploit their opponents effectively. In addition, the performance of equilibrium strategies cannot be guaranteed in games with more than two players and multiple Nash equilibria. This dissertation focuses on devising an evolutionary method to discover opponent models based on recurrent neural networks. A series of computer poker agents called Adaptive System for Hold’Em (ASHE) were evolved for HUNL. ASHE models the opponent explicitly using Pattern Recognition Trees (PRTs) and LSTM estimators. The default and board-texture-based PRTs maintain statistical data on the opponent strategies at different game states. The Opponent Action Rate Estimator predicts the opponent’s moves, and the Hand Range Estimator evaluates the showdown value of ASHE’s hand. Recursive Utility Estimation is used to evaluate the expected utility/reward for each available action. Experimental results show that (1) ASHE exploits opponents with high to moderate level of exploitability more effectively than Nash-equilibrium-based agents, and (2) ASHE can defeat top-ranking equilibrium-based poker agents. Thus, the dissertation introduces an effective new method to building high-performance computer agents for poker and other imperfect information games. It also provides a promising direction for future research in imperfect information games beyond the equilibrium-based approach.Computer Science

Opponent Modelling in Multi-Agent Systems

Reinforcement Learning (RL) formalises a problem where an intelligent agent needs to learn and achieve certain goals by maximising a long-term return in an environment. Multi-agent reinforcement learning (MARL) extends traditional RL to multiple agents. Many RL algorithms lose convergence guarantee in non-stationary environments due to the adaptive opponents. Partial observation caused by agents’ different private observations introduces high variance during the training which exacerbates the data inefficiency. In MARL, training an agent to perform well against a set of opponents often leads to bad performance against another set of opponents. Non-stationarity, partial observation and unclear learning objective are three critical problems in MARL which hinder agents’ learning and they all share a cause which is the lack of knowledge of the other agents. Therefore, in this thesis, we propose to solve these problems with opponent modelling methods. We tailor our solutions by combining opponent modelling with other techniques according to the characteristics of problems we face. Specifically, we first propose ROMMEO, an algorithm inspired by Bayesian inference, as a solution to alleviate the non-stationarity in cooperative games. Then we study the partial observation problem caused by agents’ private observation and design an implicit communication training method named PBL. Lastly, we investigate solutions to the non-stationarity and unclear learning objective problems in zero-sum games. We propose a solution named EPSOM which aims for finding safe exploitation strategies to play against non-stationary opponents. We verify our proposed methods by varied experiments and show they can achieve the desired performance. Limitations and future works are discussed in the last chapter of this thesis

Characterizing the Value and Effect of Perceptiveness in Various Game-Theoretic Settings

This thesis investigates the value and effect that perceptiveness has in three game-theoretic settings. I consider a player to be expert if they know the value of a particular payoff-relevant parameter in the models I study. If the player does not know such value, I consider the player to be inexpert. A player is perceptive if they know with certainty whether their opponent is expert. Otherwise, the player is imperceptive. The goal of this thesis is to provide insight regarding the potential value and effect that perceptiveness has in the game-theoretic settings I study. The first model I consider emulates a two-player, one-round game of poker. The second model I investigate is a two-player market-entry game. The third model I study depicts a two-player market-entry game that is influenced by an information designer who aims to maximize producer surplus. In each model, I consider distinct information structures, which vary in terms of the players\u27 levels of expertise and perceptiveness. In the first two models, I solve for the Bayesian Nash equilibria of each game and compute each agent\u27s expected payoff. Then, by comparing the equilibrium action and expected payoff of an agent when perceptive to that when imperceptive, holding all else constant, I determine the agent\u27s value of perceptiveness and the effect that perceptiveness has on the agent\u27s equilibrium strategy. In the third model, I solve for the information designer\u27s attainable decision rules, then determine which of the attainable decision rules maximizes producer surplus. Among other insights, I find that perceptiveness is generally valuable, whether that be from the perspective of a poker player, a player considering market-entry, or an information designer in a market-entry game. Furthermore, under an equilibrium that treats the market-entry players as symmetrically as possible, the value of perceptiveness is positive when both players have a sufficiently high probability of being expert; whereas, the value of perceptiveness is zero when either player is inexpert with a sufficiently high probability. Also, perceptiveness is generally less beneficial to players considering market entry than it is to players playing poker

Learning under uncertainty: a model-based approach for understanding gambling behaviour

Gamblers in the real world have been found to successfully navigate complex multivariate problems such as those of poker and the racetrack but also to misunderstand elementary problems such as those of roulette and dice. An account of gambling behaviour must accommodate both the strengths and weaknesses of decision making and yet neither of the dominating decision making traditions of heuristics and biases or Bayesian rational inference does. This thesis presents evidence supporting a model-based approach for studying gambling behaviour. The account is built on the premise that decision making agents hold a highly structured mental representation of the problem that is then refined through adjustments made by evaluating incoming evidence. In Study 1, roulette games played at a casino illustrate the range of tactics beyond simple data-driven strategies that are used in chance-based games. In Study 2, an experimental manipulation of the framing of a chance-based dice game highlights the role of prior beliefs about underlying outcome-generating processes. Studies 3 and 4 examine the impact of prior beliefs on subsequent information processing, using a laboratory-based slot machine paradigm. To complement these findings on a computational level, a modelling exercise in Study 5 shows indirectly that assuming a similarity mechanism of judgment is insufficient for predicting the impact of prior beliefs over time. Studies 6 and 7 used racetrack and poker betting experimental paradigms to show that, although priors were integrated into decisions without evaluation, incoming evidence underwent information search and hypothesis and data evaluation processes. Implications for users of gambling research and for future directions of the field are discussed

AI in Computer Games: Generating Interesting Interactive Opponents by the use of Evolutionary Computation

Institute of Perception, Action and BehaviourWhich features of a computer game contribute to the player’s enjoyment of it? How can we automatically generate interesting and satisfying playing experiences for a given game? These are the two key questions addressed in this dissertation. Player satisfaction in computer games depends on a variety of factors; here the focus is on the contribution of the behaviour and strategy of game opponents in predator/prey games. A quantitative metric of the ‘interestingness’ of opponent behaviours is defined based on qualitative considerations of what is enjoyable in such games, and a mathematical formulation grounded in observable data is derived. Using this metric, neural-network opponent controllers are evolved for dynamic game environments where limited inter-agent communication is used to drive spatial coordination of opponent teams. Given the complexity of the predator task, cooperative team behaviours are investigated. Initial candidates are generated using off-line learning procedures operating on minimal neural controllers with the aim of maximising opponent performance. These example controllers are then adapted using on-line (i.e. during play) learning techniques to yield opponents that provide games of high interest. The on-line learning methodology is evaluated using two dissimilar predator/prey games with a number of different computer player strategies. It exhibits generality across the two game test-beds and robustness to changes of player, initial opponent controller selected, and complexity of the game field. The interest metric is also evaluated by comparison with human judgement of game satisfaction in an experimental survey. A statistically significant number of players were asked to rank game experiences with a test-bed game using perceived interestingness and their ranking was compared with that of the proposed interest metric. The results show that the interest metric is consistent with human judgement of game satisfaction. Finally, the generality, limitations and potential of the proposed methodology and techniques are discussed, and other factors affecting the player’s satisfaction, such as the player’s own strategy, are briefly considered. Future directions building on the work described herein are presented and discussed