Continuous Strategy Replicator Dynamics for Multi--Agent Learning

The problem of multi-agent learning and adaptation has attracted a great deal of attention in recent years. It has been suggested that the dynamics of multi agent learning can be studied using replicator equations from population biology. Most existing studies so far have been limited to discrete strategy spaces with a small number of available actions. In many cases, however, the choices available to agents are better characterized by continuous spectra. This paper suggests a generalization of the replicator framework that allows to study the adaptive dynamics of Q-learning agents with continuous strategy spaces. Instead of probability vectors, agents strategies are now characterized by probability measures over continuous variables. As a result, the ordinary differential equations for the discrete case are replaced by a system of coupled integral--differential replicator equations that describe the mutual evolution of individual agent strategies. We derive a set of functional equations describing the steady state of the replicator dynamics, examine their solutions for several two-player games, and confirm our analytical results using simulations.Comment: 12 pages, 15 figures, accepted for publication in JAAMA

Information Geometry and Evolutionary Game Theory

The Shahshahani geometry of evolutionary game theory is realized as the information geometry of the simplex, deriving from the Fisher information metric of the manifold of categorical probability distributions. Some essential concepts in evolutionary game theory are realized information-theoretically. Results are extended to the Lotka-Volterra equation and to multiple population systems.Comment: Added reference

Coupled Replicator Equations for the Dynamics of Learning in Multiagent Systems

Starting with a group of reinforcement-learning agents we derive coupled replicator equations that describe the dynamics of collective learning in multiagent systems. We show that, although agents model their environment in a self-interested way without sharing knowledge, a game dynamics emerges naturally through environment-mediated interactions. An application to rock-scissors-paper game interactions shows that the collective learning dynamics exhibits a diversity of competitive and cooperative behaviors. These include quasiperiodicity, stable limit cycles, intermittency, and deterministic chaos--behaviors that should be expected in heterogeneous multiagent systems described by the general replicator equations we derive.Comment: 4 pages, 3 figures, http://www.santafe.edu/projects/CompMech/papers/credlmas.html; updated references, corrected typos, changed conten

Evolutionary Game Dynamics for Two Interacting Populations under Environmental Feedback

We study the evolutionary dynamics of games under environmental feedback using replicator equations for two interacting populations. One key feature is to consider jointly the co-evolution of the dynamic payoff matrices and the state of the environment: the payoff matrix varies with the changing environment and at the same time, the state of the environment is affected indirectly by the changing payoff matrix through the evolving population profiles. For such co-evolutionary dynamics, we investigate whether convergence will take place, and if so, how. In particular, we identify the scenarios where oscillation offers the best predictions of long-run behavior by using reversible system theory. The obtained results are useful to describe the evolution of multi-community societies in which individuals' payoffs and societal feedback interact.Comment: 7 pages, submitted to a conferenc

Chaotic provinces in the kingdom of the Red Queen

The interplay between parasites and their hosts is found in all kinds of species and plays an important role in understanding the principles of evolution and coevolution. Usually, the different genotypes of hosts and parasites oscillate in their abundances. The well-established theory of oscillatory Red Queen dynamics proposes an ongoing change in frequencies of the different types within each species. So far, it is unclear in which way Red Queen dynamics persists with more than two types of hosts and parasites. In our analysis, an arbitrary number of types within two species are examined in a deterministic framework with constant or changing population size. This general framework allows for analytical solutions for internal fixed points and their stability. For more than two species, apparently chaotic dynamics has been reported. Here we show that even for two species, once more than two types are considered per species, irregular dynamics in their frequencies can be observed in the long run. The nature of the dynamics depends strongly on the initial configuration of the system; the usual regular Red Queen oscillations are only observed in some parts of the parameter region

Evolutionary Poisson Games for Controlling Large Population Behaviors

Emerging applications in engineering such as crowd-sourcing and (mis)information propagation involve a large population of heterogeneous users or agents in a complex network who strategically make dynamic decisions. In this work, we establish an evolutionary Poisson game framework to capture the random, dynamic and heterogeneous interactions of agents in a holistic fashion, and design mechanisms to control their behaviors to achieve a system-wide objective. We use the antivirus protection challenge in cyber security to motivate the framework, where each user in the network can choose whether or not to adopt the software. We introduce the notion of evolutionary Poisson stable equilibrium for the game, and show its existence and uniqueness. Online algorithms are developed using the techniques of stochastic approximation coupled with the population dynamics, and they are shown to converge to the optimal solution of the controller problem. Numerical examples are used to illustrate and corroborate our results