The role of information in multi-agent learning

This paper aims to contribute to the study of auction design within the domain of agent-based computational economics. In particular, we investigate the efficiency of different auction mechanisms in a bounded-rationality setting where heterogeneous artificial agents learn to compete for the supply of a homogeneous good. Two different auction mechanisms are compared: the uniform and the discriminatory pricing rules. Demand is considered constant and inelastic to price. Four learning algorithms representing different models of bounded rationality, are considered for modeling agents' learning capabilities. Results are analyzed according to two game-theoretic solution concepts, i.e., Nash equilibria and Pareto optima, and three performance metrics. Different computational experiments have been performed in different game settings, i.e., self-play and mixed-play competition with two, three and four market participants. This methodological approach permits to highlight properties which are invariant to the different market settings considered. The main economic result is that, irrespective of the learning model considered, the discriminatory pricing rule is a more e±cient market mechanism than the uniform one in the two and three players games, whereas identical outcomes are obtained in four players competitions. Important insights are also given for the use of multi-agent learning as a framework for market design.multi-agent learning; auction markets; design economics; agent-based computational economics

The role of information in multi-agent learning

This paper aims to contribute to the study of auction design within the domain of agent-based computational economics. In particular, we investigate the efficiency of different auction mechanisms in a bounded-rationality setting where heterogeneous artificial agents learn to compete for the supply of a homogeneous good. Two different auction mechanisms are compared: the uniform and the discriminatory pricing rules. Demand is considered constant and inelastic to price. Four learning algorithms representing different models of bounded rationality, are considered for modeling agents' learning capabilities. Results are analyzed according to two game-theoretic solution concepts, i.e., Nash equilibria and Pareto optima, and three performance metrics. Different computational experiments have been performed in different game settings, i.e., self-play and mixed-play competition with two, three and four market participants. This methodological approach permits to highlight properties which are invariant to the different market settings considered. The main economic result is that, irrespective of the learning model considered, the discriminatory pricing rule is a more e±cient market mechanism than the uniform one in the two and three players games, whereas identical outcomes are obtained in four players competitions. Important insights are also given for the use of multi-agent learning as a framework for market design

COMBINATORIAL AUCTIONS WITH TRANSPORTATION COST

Abstrac

Many-agent Reinforcement Learning

Multi-agent reinforcement learning (RL) solves the problem of how each agent should behave optimally in a stochastic environment in which multiple agents are learning simultaneously. It is an interdisciplinary domain with a long history that lies in the joint area of psychology, control theory, game theory, reinforcement learning, and deep learning. Following the remarkable success of the AlphaGO series in single-agent RL, 2019 was a booming year that witnessed significant advances in multi-agent RL techniques; impressive breakthroughs have been made on developing AIs that outperform humans on many challenging tasks, especially multi-player video games. Nonetheless, one of the key challenges of multi-agent RL techniques is the scalability; it is still non-trivial to design efficient learning algorithms that can solve tasks including far more than two agents ($N \gg 2$), which I name by \emph{many-agent reinforcement learning} (MARL\footnote{I use the world of ``MARL" to denote multi-agent reinforcement learning with a particular focus on the cases of many agents; otherwise, it is denoted as ``Multi-Agent RL" by default.}) problems. In this thesis, I contribute to tackling MARL problems from four aspects. Firstly, I offer a self-contained overview of multi-agent RL techniques from a game-theoretical perspective. This overview fills the research gap that most of the existing work either fails to cover the recent advances since 2010 or does not pay adequate attention to game theory, which I believe is the cornerstone to solving many-agent learning problems. Secondly, I develop a tractable policy evaluation algorithm -- $\alpha^\alpha$-Rank -- in many-agent systems. The critical advantage of $\alpha^\alpha$-Rank is that it can compute the solution concept of $\alpha$-Rank tractably in multi-player general-sum games with no need to store the entire pay-off matrix. This is in contrast to classic solution concepts such as Nash equilibrium which is known to be $PPAD$-hard in even two-player cases. $\alpha^\alpha$-Rank allows us, for the first time, to practically conduct large-scale multi-agent evaluations. Thirdly, I introduce a scalable policy learning algorithm -- mean-field MARL -- in many-agent systems. The mean-field MARL method takes advantage of the mean-field approximation from physics, and it is the first provably convergent algorithm that tries to break the curse of dimensionality for MARL tasks. With the proposed algorithm, I report the first result of solving the Ising model and multi-agent battle games through a MARL approach. Fourthly, I investigate the many-agent learning problem in open-ended meta-games (i.e., the game of a game in the policy space). Specifically, I focus on modelling the behavioural diversity in meta-games, and developing algorithms that guarantee to enlarge diversity during training. The proposed metric based on determinantal point processes serves as the first mathematically rigorous definition for diversity. Importantly, the diversity-aware learning algorithms beat the existing state-of-the-art game solvers in terms of exploitability by a large margin. On top of the algorithmic developments, I also contribute two real-world applications of MARL techniques. Specifically, I demonstrate the great potential of applying MARL to study the emergent population dynamics in nature, and model diverse and realistic interactions in autonomous driving. Both applications embody the prospect that MARL techniques could achieve huge impacts in the real physical world, outside of purely video games

Agendas for multi-agent learning

Shoham et al. [1] identify several important agendas which can help direct research in multi-agent learning. We propose two additional agendas— called “modelling ” and “design”—which cover the problems we need to consider before our agents can start learning. We then consider research goals for modelling, design, and learning, and identify the problem of finding learning algorithms that guarantee convergence to Pareto-dominant equilibria against a wide range of opponents. Finally, we conclude with an example: starting from an informally-specified multi-agent learning problem, we illustrate how one might formalize and solve it by stepping through the tasks of modelling, design, and learning. This report is an extended version of a paper which will appear in a special issue of Artificial Intelligence Journal [2]; in addition to the topics covered in that paper, this report contains several appendices providing extra details on various algorithms, definitions, and examples

Agendas for multi-agent learning

Shoham et al. [1] identify several important agendas which can help direct research in multi-agent learning. We propose two additional agendas— called “modelling ” and “design”—which cover the problems we need to consider before our agents can start learning. We then consider research goals for modelling, design, and learning, and identify the problem of finding learning algorithms that guarantee convergence to Pareto-dominant equilibria against a wide range of opponents. Finally, we conclude with an example: starting from an informally-specified multi-agent learning problem, we illustrate how one might formalize and solve it by stepping through the tasks of modelling, design, and learning. This report is an extended version of a paper which will appear in a special issue of Artificial Intelligence Journal [2]; in addition to the topics covered in that paper, this report contains several appendices providing extra details on various algorithms, definitions, and examples