Inequity aversion improves cooperation in intertemporal social dilemmas

Groups of humans are often able to find ways to cooperate with one another in complex, temporally extended social dilemmas. Models based on behavioral economics are only able to explain this phenomenon for unrealistic stateless matrix games. Recently, multi-agent reinforcement learning has been applied to generalize social dilemma problems to temporally and spatially extended Markov games. However, this has not yet generated an agent that learns to cooperate in social dilemmas as humans do. A key insight is that many, but not all, human individuals have inequity averse social preferences. This promotes a particular resolution of the matrix game social dilemma wherein inequity-averse individuals are personally pro-social and punish defectors. Here we extend this idea to Markov games and show that it promotes cooperation in several types of sequential social dilemma, via a profitable interaction with policy learnability. In particular, we find that inequity aversion improves temporal credit assignment for the important class of intertemporal social dilemmas. These results help explain how large-scale cooperation may emerge and persist.Comment: 15 pages, 8 figure

Differentiable Game Mechanics

Deep learning is built on the foundational guarantee that gradient descent on an objective function converges to local minima. Unfortunately, this guarantee fails in settings, such as generative adversarial nets, that exhibit multiple interacting losses. The behavior of gradient-based methods in games is not well understood and is becoming increasingly important as adversarial and multi-objective architectures proliferate. In this paper, we develop new tools to understand and control the dynamics in n-player differentiable games. The key result is to decompose the game Jacobian into two components. The first, symmetric component, is related to potential games, which reduce to gradient descent on an implicit function. The second, antisymmetric component, relates to Hamiltonian games, a new class of games that obey a conservation law akin to conservation laws in classical mechanical systems. The decomposition motivates Symplectic Gradient Adjustment (SGA), a new algorithm for finding stable fixed points in differentiable games. Basic experiments show SGA is competitive with recently proposed algorithms for finding stable fixed points in GANs – while at the same time being applicable to, and having guarantees in, much more general cases

Partner Selection for the Emergence of Cooperation in Multi-Agent Systems Using Reinforcement Learning

Social dilemmas have been widely studied to explain how humans are able to cooperate in society. Considerable effort has been invested in designing artificial agents for social dilemmas that incorporate explicit agent motivations that are chosen to favor coordinated or cooperative responses. The prevalence of this general approach points towards the importance of achieving an understanding of both an agent's internal design and external environment dynamics that facilitate cooperative behavior. In this paper, we investigate how partner selection can promote cooperative behavior between agents who are trained to maximize a purely selfish objective function. Our experiments reveal that agents trained with this dynamic learn a strategy that retaliates against defectors while promoting cooperation with other agents resulting in a prosocial society.Comment:

Do Equity Preferences Matter in Climate Negotiations? An Experimental Investigation

This paper investigates in how far equity preferences may matter for climate negotiations. For this purposes we conducted a simple experiment with people who have been involved in international climate policy. The experiment, which was run via the Internet, consisted of two simple non-strategic games suited to measure the parameters of inequity aversion in a Fehr and Schmidt (1999) utility function. We find that our participants show aversion against advantageous as well as disadvantageous inequity to a considerable amount. Moreover, the degree of inequity aversion is higher compared to that of students in the similar study of Dannenberg et al. (2007). Regarding the geographical variety in our sample, we cannot confirm significant differences in the degree of inequity aversion between different regions in the world, which is in line with previous findings from the experimental literature. This finding lends support to the hypothesis that equity preferences are "hard-wired" and not much influenced by socio-economic or cultural circumstances. --individual preferences,inequity aversion,climate policy,experimental economics,public goods