Hierarchical Deep Counterfactual Regret Minimization

Imperfect Information Games (IIGs) offer robust models for scenarios where decision-makers face uncertainty or lack complete information. Counterfactual Regret Minimization (CFR) has been one of the most successful family of algorithms for tackling IIGs. The integration of skill-based strategy learning with CFR could potentially enhance learning performance for complex IIGs. For this, a hierarchical strategy needs to be learnt, wherein low-level components represent specific skills and the high-level component manages the transition between skills. This hierarchical approach also enhances interpretability, helping humans pinpoint scenarios where the agent is struggling and intervene with targeted expertise. This paper introduces the first hierarchical version of Deep CFR (HDCFR), an innovative method that boosts learning efficiency in tasks involving extensively large state spaces and deep game trees. A notable advantage of HDCFR over previous research in this field is its ability to facilitate learning with predefined (human) expertise and foster the acquisition of transferable skills that can be applied to similar tasks. To achieve this, we initially construct our algorithm on a tabular setting, encompassing hierarchical CFR updating rules and a variance-reduced Monte-Carlo sampling extension, and offer its essential theoretical guarantees. Then, to adapt our algorithm for large-scale applications, we employ neural networks as function approximators and suggest deep learning objectives that coincide with those in the tabular setting while maintaining the theoretical outcomes

Deep Counterfactual Regret Minimization in Continuous Action Space

Counterfactual regret minimization based algorithms are used as the state-of-the-art solutions for various problems within imperfect-information games. Deep learning has seen a multitude of uses in recent years. Recently deep learning has been combined with counterfactual regret minimization to increase the generality of the counterfactual regret minimization algorithms. This thesis proposes a new way of increasing the generality of the counterfactual regret minimization algorithms even further by increasing the role of neural networks. In addition, to combat the variance caused by the use of neural networks, a new way of sampling is introduced to reduce the variance. These proposed modifications were compared against baseline algorithms. The proposed way of reducing variance improved the performance of counterfactual regret minimization, while the method for increasing generality was found to be lacking especially when scaling the baseline model. Possible reasons for this are discussed and future research ideas are offered

Survey of Artificial Intelligence for Card Games and Its Application to the Swiss Game Jass

In the last decades we have witnessed the success of applications of Artificial Intelligence to playing games. In this work we address the challenging field of games with hidden information and card games in particular. Jass is a very popular card game in Switzerland and is closely connected with Swiss culture. To the best of our knowledge, performances of Artificial Intelligence agents in the game of Jass do not outperform top players yet. Our contribution to the community is two-fold. First, we provide an overview of the current state-of-the-art of Artificial Intelligence methods for card games in general. Second, we discuss their application to the use-case of the Swiss card game Jass. This paper aims to be an entry point for both seasoned researchers and new practitioners who want to join in the Jass challenge