Malthusian Reinforcement Learning

Here we explore a new algorithmic framework for multi-agent reinforcement learning, called Malthusian reinforcement learning, which extends self-play to include fitness-linked population size dynamics that drive ongoing innovation. In Malthusian RL, increases in a subpopulation's average return drive subsequent increases in its size, just as Thomas Malthus argued in 1798 was the relationship between preindustrial income levels and population growth. Malthusian reinforcement learning harnesses the competitive pressures arising from growing and shrinking population size to drive agents to explore regions of state and policy spaces that they could not otherwise reach. Furthermore, in environments where there are potential gains from specialization and division of labor, we show that Malthusian reinforcement learning is better positioned to take advantage of such synergies than algorithms based on self-play.Comment: 9 pages, 2 tables, 4 figure

Reinforced Galton-Watson processes I: Malthusian exponents

In a reinforced Galton-Watson process with reproduction law $\boldsymbol{\nu}$ and memory parameter $q\in(0,1)$, the number of children of a typical individual either, with probability $q$, repeats that of one of its forebears picked uniformly at random, or, with complementary probability $1-q$, is given by an independent sample from $\boldsymbol{\nu}$. We estimate the average size of the population at a large generation, and in particular, we determine explicitly the Malthusian growth rate in terms of $\boldsymbol{\nu}$ and $q$. Our approach via the analysis of transport equations owns much to works by Flajolet and co-authors.Comment: Several precisions added to the singularity analysis in Section 5 ; some additional results obtained in Section

Is the Political Economy Stable or Chaotic?

Recent events in the global economy have caused many writers to argue that the market is driven by animal spirits, by irrational exuberance or speculation. At the same time, the economic downturn has apparently caused many voters in the United States, and other countries, to change their opinion about the the proper role of government. Unfortunately, there does not exist a general equilibrium model of the political economy, combining a formal model of the existence, and convergence to a price equilibrium, as well as an equilibrium model of political choice. One impediment to such a theory is the so-called chaos theorem which suggests that existence of a political equilibrium is non-generic. This paper surveys results in the theory of dynamical systems, emphasizing the role of structural stability and chaos. We consider models of celestial mechanics where the notion of chaos first developed, and then examine applications in models of climate change and economics. There is discussion of the past influ ences of climate on human society, and particularly how agriculture developed during the “holocene,” the past ten thousand years of benign climate. The recent period of globalization is likened to the holocene, and the question is raised whether future climate change may bring economic and political chaos.Economic uncertainty, climate change, political disorder

Grounding Artificial Intelligence in the Origins of Human Behavior

Recent advances in Artificial Intelligence (AI) have revived the quest for agents able to acquire an open-ended repertoire of skills. However, although this ability is fundamentally related to the characteristics of human intelligence, research in this field rarely considers the processes that may have guided the emergence of complex cognitive capacities during the evolution of the species. Research in Human Behavioral Ecology (HBE) seeks to understand how the behaviors characterizing human nature can be conceived as adaptive responses to major changes in the structure of our ecological niche. In this paper, we propose a framework highlighting the role of environmental complexity in open-ended skill acquisition, grounded in major hypotheses from HBE and recent contributions in Reinforcement learning (RL). We use this framework to highlight fundamental links between the two disciplines, as well as to identify feedback loops that bootstrap ecological complexity and create promising research directions for AI researchers