Hybrid Mean Field Learning in Large-Scale Dynamic Robust Games

Applied and Computational Mathematics, ISSN 1683-3511One of the objectives in distributed interacting multi-player systems is to enable a collection of different players to achieve a desirable objective. There are two overriding challenges to achieving this objective: The first one is related to the complexity of finding optimal solution. A centralized algorithm may be prohibitively complex when there are large number of interacting players. This motivates the use of adaptive methods that enable players to self-organize into suitable, if not optimal, alternative solutions. The second challenge is limited information. Players may have limited knowledge about the status of other players, except perhaps for a small subset of neighboring players. The limitations in term of information induce robust stochastic optimization, bounded rationality and inconsistent beliefs. In this work, we investigate asymptotic pseudo-trajectories of large-scale dynamic robust games under various COmbined fully DIstributed PAyoff and Strategy Reinforcement Learning (CODIPAS-RL) under outdated noisy measurement and random updates. Extension to continuous action space is discussed

Crowd-Averse Robust Mean-Field Games: Approximation via State Space Extension

We consider a population of dynamic agents, also referred to as players. The state of each player evolves according to a linear stochastic differential equation driven by a Brownian motion and under the influence of a control and an adversarial disturbance. Every player minimizes a cost functional which involves quadratic terms on state and control plus a crosscoupling mean-field term measuring the congestion resulting from the collective behavior, which motivates the term “crowdaverse”. Motivations for this model are analyzed and discussed in three main contexts: a stock market application, a production engineering example, and a dynamic demand management problem in power systems. For the problem in its abstract formulation, we illustrate the paradigm of robust mean-field games. Main contributions involve first the formulation of the problem as a robust mean-field game; second, the development of a new approximate solution approach based on the extension of the state space; third, a relaxation method to minimize the approximation error. Further results are provided for the scalar case, for which we establish performance bounds, and analyze stochastic stability of both the microscopic and the macroscopic dynamics

Crowd-Averse Cyber-Physical Systems: The Paradigm of Robust Mean Field Games

For a networked controlled system we illustrate the paradigm of robust mean-field games. This is a modeling framework at the interface of differential game theory, mathematical physics, and H1-optimal control that tries to capture the mutual influence between a crowd and its individuals. First, we establish a mean-field system for such games including the effects of adversarial disturbances. Second, we identify the optimal response of the individuals for a given population behavior. Third, we provide an analysis of equilibria and their stability

Consensus via multi-population robust mean-field games

In less prescriptive environments where individuals are told ‘what to do’ but not ‘how to do’, synchronization can be a byproduct of strategic thinking, prediction, and local interactions. We prove this in the context of multipopulation robust mean-field games. The model sheds light on a multi-scale phenomenon involving fast synchronization within the same population and slow inter-cluster oscillation between different populations

Approximate solutions for crowd-averse robust mean-field games

We consider a population of dynamic agents, also referred to as players. The state of each player evolves according to a linear stochastic differential equation driven by a Brownian motion and under the influence of a control and an adversarial disturbance. Every player minimizes a cost functional which involves quadratic terms on state and control plus a cross-coupling mean-field term measuring the congestion resulting from the collective behavior, which motivates the term 'crowd-averse'. For this game we first illustrate the paradigm of robust mean-field games. Second, we provide a new approximate solution approach based on the extension of the state space and prove the existence of equilibria and their stability properties

Oscillatory Dynamics in Rock-Paper-Scissors Games with Mutations

We study the oscillatory dynamics in the generic three-species rock-paper-scissors games with mutations. In the mean-field limit, different behaviors are found: (a) for high mutation rate, there is a stable interior fixed point with coexistence of all species; (b) for low mutation rates, there is a region of the parameter space characterized by a limit cycle resulting from a Hopf bifurcation; (c) in the absence of mutations, there is a region where heteroclinic cycles yield oscillations of large amplitude (not robust against noise). After a discussion on the main properties of the mean-field dynamics, we investigate the stochastic version of the model within an individual-based formulation. Demographic fluctuations are therefore naturally accounted and their effects are studied using a diffusion theory complemented by numerical simulations. It is thus shown that persistent erratic oscillations (quasi-cycles) of large amplitude emerge from a noise-induced resonance phenomenon. We also analytically and numerically compute the average escape time necessary to reach a (quasi-)cycle on which the system oscillates at a given amplitude.Comment: 25 pages, 9 figures. To appear in the Journal of Theoretical Biolog

Approximate Dynamic Programming for a Mean-field Game of Traffic Flow: Existence and Uniqueness

Highway vehicular traffic is an inherently multi-agent problem. Traffic jams can appear and disappear mysteriously. We develop a method for traffic flow control that is applied at the vehicular level via mean-field games. We begin this work with a microscopic model of vehicles subject to control input, disturbances, noise, and a speed limit. We formulate a discounted-cost infinite-horizon robust mean-field game on the vehicles, and obtain the associated dynamic programming (DP) PDE system. We then perform approximate dynamic programming (ADP) using these equations to obtain a sub-optimal control for the traffic density adaptively. The sub-optimal controls are subject to an ODE-PDE system. We show that the ADP ODE-PDE system has a unique weak solution in a suitable Hilbert space using semigroup and successive approximation methods. We additionally give a numerical simulation, and interpret the results.Comment: 42 pages, 5 figure