Learning with Opponent-Learning Awareness

Multi-agent settings are quickly gathering importance in machine learning. This includes a plethora of recent work on deep multi-agent reinforcement learning, but also can be extended to hierarchical RL, generative adversarial networks and decentralised optimisation. In all these settings the presence of multiple learning agents renders the training problem non-stationary and often leads to unstable training or undesired final results. We present Learning with Opponent-Learning Awareness (LOLA), a method in which each agent shapes the anticipated learning of the other agents in the environment. The LOLA learning rule includes a term that accounts for the impact of one agent's policy on the anticipated parameter update of the other agents. Results show that the encounter of two LOLA agents leads to the emergence of tit-for-tat and therefore cooperation in the iterated prisoners' dilemma, while independent learning does not. In this domain, LOLA also receives higher payouts compared to a naive learner, and is robust against exploitation by higher order gradient-based methods. Applied to repeated matching pennies, LOLA agents converge to the Nash equilibrium. In a round robin tournament we show that LOLA agents successfully shape the learning of a range of multi-agent learning algorithms from literature, resulting in the highest average returns on the IPD. We also show that the LOLA update rule can be efficiently calculated using an extension of the policy gradient estimator, making the method suitable for model-free RL. The method thus scales to large parameter and input spaces and nonlinear function approximators. We apply LOLA to a grid world task with an embedded social dilemma using recurrent policies and opponent modelling. By explicitly considering the learning of the other agent, LOLA agents learn to cooperate out of self-interest. The code is at github.com/alshedivat/lola

It pays to pay - Big Five personality influences on co-operative behaviour in an incentivized and hypothetical prisoner's dilemma game

The authors investigated how the presence or absence of monetary incentives in a prisoner's dilemma game may influence research outcomes. Specifically, the predictive power of the Big Five personality traits on decisions in an incentivized (N = 60) or hypothetical (N = 60) prisoner's dilemma game was investigated. Participants were less generous in the incentivized game. More importantly, personality predicted decisions only in the incentivized game, with low Neuroticism and high Openness to Experience predicting more cooperative transfers. The influence of Neuroticism on behaviour in the incentivized game was mediated by risk attitude. The results are consistent with other results suggesting that the Big Five are relevant predictors of moral behaviour, and with results suggesting that the determinants of hypothetical decisions are different from the determinants of real decisions, with the latter being more revealing of one's true preferences. The authors argue that psychologists, contrary to prevailing praxis, should consider making their participants' decisions more real. This could allow psychologists to more convincingly generalize laboratory findings into contexts outside of the laboratory.Big Five, Prisoner's dilemma, Social dilemma, Moral behaviour, Incentives, Stake size

Can We Build Behavioral Game Theory?

The way economists and other social scientists model how people make interdependent decisions is through the theory of games. Psychologists and behavioral economists, however, have established many deviations from the predictions of game theory. In response to these findings, a broad movement has arisen to salvage the core of game theory. Extant models of interdependent decision-making try to improve their explanatory domain by adding some corrective terms or limits. We will make the argument that this approach is misguided. For this approach to work, the deviations would have to be consistent. Drawing in part on our experimental results, we will argue that deviations from classical models are not consistent for any individual from one task to the next or between individuals for the same task. In turn, the problem of finding an equilibrium strategy is not easier but rather is exponentially more difficult. It does not seem that game theory can be repaired by adding corrective terms (such as consideration of personal characteristics, social norms, heuristic or bias terms, or cognitive limits on choice and learning). In what follows, we describe new methods for investigating interdependent decision-making. Our experimental results show that people do not choose consistently, do not hold consistent beliefs, and do not in general align actions and beliefs. We will show that experimental choices are inconsistent in ways that prevent us from drawing general characterizations of an individual’s choices or beliefs or of the general population\u27s choices and beliefs. A general behavioral game theory seems a distant and, at present, unfulfilled hope

Learning in Repeated Games: Human Versus Machine

While Artificial Intelligence has successfully outperformed humans in complex combinatorial games (such as chess and checkers), humans have retained their supremacy in social interactions that require intuition and adaptation, such as cooperation and coordination games. Despite significant advances in learning algorithms, most algorithms adapt at times scales which are not relevant for interactions with humans, and therefore the advances in AI on this front have remained of a more theoretical nature. This has also hindered the experimental evaluation of how these algorithms perform against humans, as the length of experiments needed to evaluate them is beyond what humans are reasonably expected to endure (max 100 repetitions). This scenario is rapidly changing, as recent algorithms are able to converge to their functional regimes in shorter time-scales. Additionally, this shift opens up possibilities for experimental investigation: where do humans stand compared with these new algorithms? We evaluate humans experimentally against a representative element of these fast-converging algorithms. Our results indicate that the performance of at least one of these algorithms is comparable to, and even exceeds, the performance of people

Collective behavior and evolutionary games - An introduction

This is an introduction to the special issue titled "Collective behavior and evolutionary games" that is in the making at Chaos, Solitons & Fractals. The term collective behavior covers many different phenomena in nature and society. From bird flocks and fish swarms to social movements and herding effects, it is the lack of a central planner that makes the spontaneous emergence of sometimes beautifully ordered and seemingly meticulously designed behavior all the more sensational and intriguing. The goal of the special issue is to attract submissions that identify unifying principles that describe the essential aspects of collective behavior, and which thus allow for a better interpretation and foster the understanding of the complexity arising in such systems. As the title of the special issue suggests, the later may come from the realm of evolutionary games, but this is certainly not a necessity, neither for this special issue, and certainly not in general. Interdisciplinary work on all aspects of collective behavior, regardless of background and motivation, and including synchronization and human cognition, is very welcome.Comment: 6 two-column pages, 1 figure; accepted for publication in Chaos, Solitons & Fractals [the special issue is available at http://www.sciencedirect.com/science/journal/09600779/56

An exploration of the motivational basis of take-some and give-some games

Surprisingly little research has investigated the particular motives that underlie choice behavior in social dilemma situations. The main aim of the present research was to ask whether behavior in take-some games (such as the multiple-person Commons Dilemma Game and the two-person Bandit Game) and give-some games (such as the multiple-person Public Goods Dilemma Game and the two-person Dictator Game) is differently affected by proself and prosocial motives. Two experimental studies were conducted. Our first experiment used a trait-based assessment of the motives, whereas in our second experiment the motives were measured as state variables. The results of both experiments revealed that proself and prosocial motives did not explain much difference between taking and giving when comparing the Commons Dilemma Game and the Public Goods Dilemma Game. Yet, our second experiment revealed that these motives did differentiate choices in the Bandit Game and the Dictator Game. More specifically, prosocial motives are more strongly related to giving behavior in the Dictator Game than to taking behavior in the Bandit Game. As such, it can be concluded that in dyadic games (but not in multiple-person games) prosocial motives (but not proself motives) predict choice behavior in a game-specific way

When a precedent of donation favors defection in the Prisoner's dilemma

In this paper we examine the question of wether a collective activity can influence cooperation in a subsequent repeated one shot prisoner's dilemma (PD) game. We conduct two series of experiments. The first consists of control experiments in which 30 periods of a PD game are played, with a random re-matching of the pairs in every period. In a second series of experiments, subjects first play a donation game and then the PD game. In the donation game they collectively discuss the amount of a donation to a given charity, before putting the question to an individual and anonymous vote. Cooperation levels in the PD games preceded by the donation game are signficantly lower than those observed in the control experiment.DONATION;COOPERATION;DEFECTION;REPEATED ONE SHOT PRISONER'S DILEMMA;EXPERIMENT