Nonlinear and evolutionary phenomena in deterministic growing economies

We discuss the implications of nonlinearity in competitive models of optimal endogenous growth. Departing from a simple representative agent setup with convex risk premium and investment adjustment costs, we define an open economy dynamic optimization problem and show that the optimal control solution is given by an autonomous nonlinear vector field in <3 with multiple equilibria and no optimal stable solutions. We give a thorough analytical and numerical analysis of this system qualitative dynamics and show the existence of local singularities, such as fold (saddle-node), Hopf and Fold-Hopf bifurcations of equilibria. Finally, we discuss the policy implications of global nonlinear phenomena. We focus on dynamic scenarios arising in the vicinity of Fold-Hopf bifurcations and demonstrate the existence of global dynamic phenomena arising from the complex organization of the invariant manifolds of this system. We then consider this setup in a non-cooperative differential game environment, where asymmetric players choose open loop no feedback strategies and dynamics are coupled by an aggregate risk premium mechanism. When only convex risk premium is considered, we show that these games have a specific state-separability property, where players have optimal, but naive, beliefs about the evolution of the state of the game. We argue that the existence of optimal beliefs in this fashion, provides a unique framework to study the implications of the self-confirming equilibrium (SCE) hypothesis in a dynamic game setup. We propose to answer the following question. Are players able to concur on a SCE, where their expectations are self-fulfilling? To evaluate this hypothesis we consider a simple conjecture. If beliefs bound the state-space of the game asymptotically and strategies are Lipschitz continuous, then it is possible to describe SCE solutions and evaluate the qualitative properties of equilibrium. If strategies are not smooth, which is likely in environments where belief-based solutions require players to learn a SCE, then asymptotic dynamics can be evaluated numerically as a Hidden Markov Model (HMM). We discuss this topic for a class of games where players lack the relevant information to pursue their optimal strategies and have to base their decisions on subjective beliefs. We set up one of the games proposed as a multi-objective optimization problem under uncertainty and evaluate its asymptotic solution as a multi-criteria HMM.We show that under a simple linear learning regime there is convergence to a SCE and portray strong emergence phenomena as a result of persistent uncertainty

Mean-Field-Type Games in Engineering

A mean-field-type game is a game in which the instantaneous payoffs and/or the state dynamics functions involve not only the state and the action profile but also the joint distributions of state-action pairs. This article presents some engineering applications of mean-field-type games including road traffic networks, multi-level building evacuation, millimeter wave wireless communications, distributed power networks, virus spread over networks, virtual machine resource management in cloud networks, synchronization of oscillators, energy-efficient buildings, online meeting and mobile crowdsensing.Comment: 84 pages, 24 figures, 183 references. to appear in AIMS 201

Many-agent Reinforcement Learning

Multi-agent reinforcement learning (RL) solves the problem of how each agent should behave optimally in a stochastic environment in which multiple agents are learning simultaneously. It is an interdisciplinary domain with a long history that lies in the joint area of psychology, control theory, game theory, reinforcement learning, and deep learning. Following the remarkable success of the AlphaGO series in single-agent RL, 2019 was a booming year that witnessed significant advances in multi-agent RL techniques; impressive breakthroughs have been made on developing AIs that outperform humans on many challenging tasks, especially multi-player video games. Nonetheless, one of the key challenges of multi-agent RL techniques is the scalability; it is still non-trivial to design efficient learning algorithms that can solve tasks including far more than two agents ($N \gg 2$), which I name by \emph{many-agent reinforcement learning} (MARL\footnote{I use the world of ``MARL" to denote multi-agent reinforcement learning with a particular focus on the cases of many agents; otherwise, it is denoted as ``Multi-Agent RL" by default.}) problems. In this thesis, I contribute to tackling MARL problems from four aspects. Firstly, I offer a self-contained overview of multi-agent RL techniques from a game-theoretical perspective. This overview fills the research gap that most of the existing work either fails to cover the recent advances since 2010 or does not pay adequate attention to game theory, which I believe is the cornerstone to solving many-agent learning problems. Secondly, I develop a tractable policy evaluation algorithm -- $\alpha^\alpha$-Rank -- in many-agent systems. The critical advantage of $\alpha^\alpha$-Rank is that it can compute the solution concept of $\alpha$-Rank tractably in multi-player general-sum games with no need to store the entire pay-off matrix. This is in contrast to classic solution concepts such as Nash equilibrium which is known to be $PPAD$-hard in even two-player cases. $\alpha^\alpha$-Rank allows us, for the first time, to practically conduct large-scale multi-agent evaluations. Thirdly, I introduce a scalable policy learning algorithm -- mean-field MARL -- in many-agent systems. The mean-field MARL method takes advantage of the mean-field approximation from physics, and it is the first provably convergent algorithm that tries to break the curse of dimensionality for MARL tasks. With the proposed algorithm, I report the first result of solving the Ising model and multi-agent battle games through a MARL approach. Fourthly, I investigate the many-agent learning problem in open-ended meta-games (i.e., the game of a game in the policy space). Specifically, I focus on modelling the behavioural diversity in meta-games, and developing algorithms that guarantee to enlarge diversity during training. The proposed metric based on determinantal point processes serves as the first mathematically rigorous definition for diversity. Importantly, the diversity-aware learning algorithms beat the existing state-of-the-art game solvers in terms of exploitability by a large margin. On top of the algorithmic developments, I also contribute two real-world applications of MARL techniques. Specifically, I demonstrate the great potential of applying MARL to study the emergent population dynamics in nature, and model diverse and realistic interactions in autonomous driving. Both applications embody the prospect that MARL techniques could achieve huge impacts in the real physical world, outside of purely video games

Linear Regression Models Applied to Imperfect Information Spacecraft Pursuit-evasion Differential Games

Within satellite rendezvous and proximity operations lies pursuit-evasion differential games between two spacecraft. The extent of possible outcomes can be mathematically bounded by differential games where each player employs optimal strategies. A linear regression model is developed from a large data set of optimal control solutions. The model is shown to map pursuer relative starting positions to final capture positions and estimate capture time. The model is 3.8 times faster than the indirect heuristic method for arbitrary pursuer starting positions on an initial relative orbit about the evader. The linear regression model is shown to be well suited for on-board implementation for autonomous mission planning

A survey of random processes with reinforcement

The models surveyed include generalized P\'{o}lya urns, reinforced random walks, interacting urn models, and continuous reinforced processes. Emphasis is on methods and results, with sketches provided of some proofs. Applications are discussed in statistics, biology, economics and a number of other areas.Comment: Published at http://dx.doi.org/10.1214/07-PS094 in the Probability Surveys (http://www.i-journals.org/ps/) by the Institute of Mathematical Statistics (http://www.imstat.org