Poker Learner: Reinforcement Learning Applied to Texas Hold'em Poker

Bibliografia: p. 61-66Tese de Mestrado Integrado. Engenharia Informática e Computação. Universidade do Porto. Faculdade de Engenharia.. 201

Using a high-level language to build a poker playing agent

Tese de mestrado integrado. Engenharia Informática e Computação. Faculdade de Engenharia. Universidade do Porto. 200

Rule based strategies for large extensive-form games: A specification language for No-Limit Texas Hold'em agents

Poker is used to measure progresses in extensive-form games research due to its unique characteristics: it is a game where playing agents have to deal with incomplete information and stochastic scenarios and a large number of decision points. The development of Poker agents has seen significant advances in one-on-one matches but there are still no consistent results in multiplayer and in games against human experts. In order to allow for experts to aid the improvement of the agents' performance, we have created a high-level strategy specification language. To support strategy definition, we have also developed an intuitive graphical tool. Additionally, we have also created a strategy inferring system, based on a dynamically weighted Euclidean distance. This approach was validated through the creation of simple agents and by successfully inferring strategies from 10 human players. The created agents were able to beat previously developed mid-level agents by a good profit margin

API para desenvolvimento de agentes de póquer online

A construção de agentes que simulem o comportamento de jogadores humanos para jogos que se baseiam em informação escondida e de natureza não determinista são uma área muito ativa na investigação em Inteligência Artificial. A variante Texas Hold'em do jogo de Poker fornece um contexto para o estudo da eficácia de várias técnicas de implementação e teorias por causa das propriedades que esta possui bem como a necessidade de lidar com informação escondida e aleatoriedade.A forma mais efetiva de testar, avaliar e comparar as metodologias utilizadas na construção do agente de poker seria poder colocar esse agente a jogar contra humanos em situação de igualdade jogando em ambiente online, em tempo real e com as mesmas restrições de tempo e de dinheiro. Desta forma, esta dissertação consiste no desenvolvimento de uma API que forneça aos agentes as estruturas necessária para jogar poker online, providenciando como input ao agente o estado do jogo e efetuando as jogadas de acordo com o output recebido do agente.Com o estudo do estado da arte realizado, algumas abordagens e técnicas de visão por computador, poderão servir de base e fornecem promissoras esperanças na concretização dos objetivos pretendidos.The creation of agents that simulate human player's behaviour for games that are based on hidden information and non-deterministic nature is a very active area on Artificial Intelligence research domain. The Texas Hold'em variant of poker provides a context for studying the effectiveness of various implementation techniques and theories because of the properties it possesses such as the need to deal with hidden information and stochasticity.The most effective way to test, evaluate and compare methodologies used in the creation of poker agents would be the ability to put those agents playing against humans on an equal footing in a online playing environment at real time and with the same time and money constraints. The focus of this dissertation follows this idea, the creation of an API that provides the agents the necessary structures to play poker online, providing the state of the game as input to the agent and making the moves according to the output received from that agent.With the study of the state of the art, some approaches and techniques of computer vision, could form the basis and provide promising hopes on achieving the intended goals

A Profitable Online Poker Agent

Jogos de informação incompleta tais como poker são uma fonte contínua de estudo e pesquisa no âmbito da inteligência artificial. No poker problemas como: modelação de oponentes; gestão de riscos e detecção de bluffs representam um desafio. O desenvolvimento de agentes capazes de considerar esses problemas e realizar cálculos probabilísticos é considerado como uma tarefa árdua de se realizar, uma vez que é exigida uma adaptação dinâmica para que seja criado um agente de poker robusto. Esta tese irá focar-se no desenvolvimento de um agente de poker capaz de jogar contra jogadores humanos e alcançar a adaptação dinâmica necessária para superar alguns jogadores humanos de poker online. Algo que será possível usando um conjunto de informações sobre cada jogador que o agente enfrenta. Utilizando como auxílio o Holdem Manager, uma ferramenta que regista mãos jogadas em salas de poker online, é possível obter estatísticas sobre todos os jogadores que o agente enfrenta nas mesas. O agente é capaz de explorar algumas destas estatísticas de maneira que possa decidir melhor sobre a acção a tomar. Alguns factores como quão agressivo é um adversário, a posição ocupada na mesa, quantos jogadores estão envolvidos, quanto dinheiro está em causa, e o par de cartas que o agente recebe são uma pequena porção do conjunto de informações utilizadas na determinação do comportamento do agente. Este agente foi desenvolvido baseando-se numa estratégia "short stack", e modelando adversários com o auxílio do conjunto de informações reunido através do Holdem Manager. Pela primeira vez na literatura do Computer Poker, são apresentados resultados de jogos de poker online, num ambiente controlado, contra jogadores humanos sem estes saberem que estão em jogo contra um agente. O agente é capaz de jogar poker online ao vivo contra jogadores humanos, e apresenta um pequeno lucro na vertente Texas Hold'em em micro limites6 de apostas, nomeadamente 0.01 e 0.02 cêntimos.Games of incomplete information, such as poker, are a continuous source of research and study in the area of artificial intelligence. Poker presents challenging problems such as opponent modeling, risk management and bluff detection. The development of agents capable of probabilistic calculations considering those problems is considered to be difficult to achieve, since dynamic adaption is required in order to create a robust computer poker player. This thesis focuses on the development of a poker agent able to play against human players and aiming to achieve the dynamic adaptation needed to beat some human players online. This will be achieved by using some sets of information about each player the agent plays against. Using Holdem Manager, a tool that registers the hands played in an online poker room; it is possible to obtain statistics about every player the agent is playing against. The agent is able to explore some of these statistics so that it can better decide on which action to take. Some factors like how aggressive an opponent is, the position held at the table, how many players are involved, how much money is involved, and the hand dealt to the agent are a few portions of the information sets used to compute the agent's behavior. This agent was developed based on a short-stack strategy, and through the use of the sets of information provided by the Holdem Manager. For the first time in the Computer Poker literature, results on online Poker agent games versus human players in a controlled environment are presented, and without the players being aware their opponent was a computer agent. The agent is able to play live online poker versus human players, and presents a small profit in the No-Limit Texas Hold'em poker game at micro stakes, namely 0.02 and 0.01 cents

Computing card probabilities in Texas Hold'em

Developing Poker agents that can compete at the level of a human expert can be a challenging endeavor, since agents' strategies must be capable of dealing with hidden information, deception and risk management. A way of addressing this issue is to model opponents' behavior in order to estimate their game plan and make decisions based on such estimations. In this paper, several hand evaluation and classification techniques are compared and conclusions on their respective applicability and scope are drawn. Also, we suggest improvements on current techniques through Monte Carlo sampling. The current methods to deal with risk management were found to be pertinent concerning the agent's decision-making process; nevertheless future integration of these methods with opponent modeling techniques can greatly improve overall Poker agents' performance