Intrinsic fluctuations of reinforcement learning promote cooperation

In this work, we ask for and answer what makes classical reinforcement learning cooperative. Cooperating in social dilemma situations is vital for animals, humans, and machines. While evolutionary theory revealed a range of mechanisms promoting cooperation, the conditions under which agents learn to cooperate are contested. Here, we demonstrate which and how individual elements of the multi-agent learning setting lead to cooperation. Specifically, we consider the widely used temporal-difference reinforcement learning algorithm with epsilon-greedy exploration in the classic environment of an iterated Prisoner's dilemma with one-period memory. Each of the two learning agents learns a strategy that conditions the following action choices on both agents' action choices of the last round. We find that next to a high caring for future rewards, a low exploration rate, and a small learning rate, it is primarily intrinsic stochastic fluctuations of the reinforcement learning process which double the final rate of cooperation to up to 80\%. Thus, inherent noise is not a necessary evil of the iterative learning process. It is a critical asset for the learning of cooperation. However, we also point out the trade-off between a high likelihood of cooperative behavior and achieving this in a reasonable amount of time. Our findings are relevant for purposefully designing cooperative algorithms and regulating undesired collusive effects.Comment: 9 pages, 4 figure

From math to metaphors and back again: Social-ecological resilience from a multi-agent-environment perspective

Science and policy stand to benefit from reconnecting the many notions of social-ecological resilience to their roots in complexity sciences.We propose several ways of moving towards operationalization through the classification of modern concepts of resilience based on a multi-agent-environment perspective. Social-ecological resilience underlies popular sustainability concepts that have been influential in formulating the United Nations Sustainable Development Goals (SDGs), such as the Planetary Boundaries and Doughnut Economics. Scientific investigation of these concepts is supported by mathematical models of planetary biophysical and societal dynamics, both of which call for operational measures of resilience. However, current quantitative descriptions tend to be restricted to the foundational form of the concept: persistence resilience. We propose a classification of modern notions of social-ecological resilience from a multi-agent-environment perspective. This aims at operationalization in a complex systems framework, including the persistence, adaptation and transformation aspects of resilience, normativity related to desirable system function, first- vs. second-order and specific vs. general resilience. For example, we discuss the use of the Topology of Sustainable Management Framework. Developing the mathematics of resilience along these lines would not only make social-ecological resilience more applicable to data and models, but could also conceptually advance resilience thinking

When optimization for governing human environment tipping elements is neither sustainable nor safe

Optimizing economic welfare in environmental governance has been criticized for delivering short-term gains at the expense of long-term environmental degradation. Different from economic optimization, the concepts of sustainability and the more recent safe operating space have been used to derive policies in environmental governance. However, a formal comparison between these three policy paradigms is still missing, leaving policy makers uncertain which paradigm to apply. Here, we develop a better understanding of their interrelationships, using a stylized model of human-environment tipping elements. We find that no paradigm guarantees fulfilling requirements imposed by another paradigm and derive simple heuristics for the conditions under which these trade-offs occur. We show that the absence of such a master paradigm is of special relevance for governing real-world tipping systems such as climate, fisheries, and farming, which may reside in a parameter regime where economic optimization is neither sustainable nor safe.The authors are grateful for financial support from the Heinrich-Böll-Foundation, the Stordalen Foundation (via the Planetary Boundaries Research Network PB.net), the Earth League’s EarthDoc program, the Leibniz Association (project DOMINOES) and the Swedish Research Council Formas (Project Grant 2014-589)

Sustainable use of renewable resources in a stylized social–ecological network model under heterogeneous resource distribution

Human societies depend on the resources ecosystems provide. Particularly since the last century, human activities have transformed the relationship between nature and society at a global scale. We study this coevolutionary relationship by utilizing a stylized model of private resource use and social learning on an adaptive network. The latter process is based on two social key dynamics beyond economic paradigms: boundedly rational imitation of resource use strategies and homophily in the formation of social network ties. The private and logistically growing resources are harvested with either a sustainable (small) or non-sustainable (large) effort. We show that these social processes can have a profound influence on the environmental state, such as determining whether the private renewable resources collapse from overuse or not. Additionally, we demonstrate that heterogeneously distributed regional resource capacities shift the critical social parameters where this resource extraction system collapses. We make these points to argue that, in more advanced coevolutionary models of the planetary social–ecological system, such socio-cultural phenomena as well as regional resource heterogeneities should receive attention in addition to the processes represented in established Earth system and integrated assessment model

Parsimonious modeling with information filtering networks

We introduce a methodology to construct parsimonious probabilistic models. This method makes use of information filtering networks to produce a robust estimate of the global sparse inverse covariance from a simple sum of local inverse covariances computed on small subparts of the network. Being based on local and low-dimensional inversions, this method is computationally very efficient and statistically robust, even for the estimation of inverse covariance of high-dimensional, noisy, and short time series. Applied to financial data our method results are computationally more efficient than state-of-the-art methodologies such as Glasso producing, in a fraction of the computation time, models that can have equivalent or better performances but with a sparser inference structure. We also discuss performances with sparse factor models where we notice that relative performances decrease with the number of factors. The local nature of this approach allows us to perform computations in parallel and provides a tool for dynamical adaptation by partial updating when the properties of some variables change without the need of recomputing the whole model. This makes this approach particularly suitable to handle big data sets with large numbers of variables. Examples of practical application for forecasting, stress testing, and risk allocation in financial systems are also provided

Earth system modeling with endogenous and dynamic human societies: the copan:CORE open World-Earth modeling framework

Analysis of Earth system dynamics in the Anthropocene requires to explicitly take into account the increasing magnitude of processes operating in human societies, their cultures, economies and technosphere and their growing feedback entanglement with those in the physical, chemical and biological systems of the planet. However, current state-of-the-art Earth System Models do not represent dynamic human societies and their feedback interactions with the biogeophysical Earth system and macroeconomic Integrated Assessment Models typically do so only with limited scope. This paper (i) proposes design principles for constructing World-Earth Models (WEM) for Earth system analysis of the Anthropocene, i.e., models of social (World) - ecological (Earth) co-evolution on up to planetary scales, and (ii) presents the copan:CORE open simulation modeling framework for developing, composing and analyzing such WEMs based on the proposed principles. The framework provides a modular structure to flexibly construct and study WEMs. These can contain biophysical (e.g. carbon cycle dynamics), socio-metabolic/economic (e.g. economic growth) and socio-cultural processes (e.g. voting on climate policies or changing social norms) and their feedback interactions, and are based on elementary entity types, e.g., grid cells and social systems. Thereby, copan:CORE enables the epistemic flexibility needed for contributions towards Earth system analysis of the Anthropocene given the large diversity of competing theories and methodologies used for describing socio-metabolic/economic and socio-cultural processes in the Earth system by various fields and schools of thought. To illustrate the capabilities of the framework, we present an exemplary and highly stylized WEM implemented in copan:CORE that illustrates how endogenizing socio-cultural processes and feedbacks could fundamentally change macroscopic model outcomes

Perspectives on adaptive dynamical systems

Adaptivity is a dynamical feature that is omnipresent in nature, socio-economics, and technology. For example, adaptive couplings appear in various real-world systems, such as the power grid, social, and neural networks, and they form the backbone of closed-loop control strategies and machine learning algorithms. In this article, we provide an interdisciplinary perspective on adaptive systems. We reflect on the notion and terminology of adaptivity in different disciplines and discuss which role adaptivity plays for various fields. We highlight common open challenges and give perspectives on future research directions, looking to inspire interdisciplinary approaches

Perspectives on adaptive dynamical systems

Adaptivity is a dynamical feature that is omnipresent in nature, socio-economics, and technology. For example, adaptive couplings appear in various real-world systems like the power grid, social, and neural networks, and they form the backbone of closed-loop control strategies and machine learning algorithms. In this article, we provide an interdisciplinary perspective on adaptive systems. We reflect on the notion and terminology of adaptivity in different disciplines and discuss which role adaptivity plays for various fields. We highlight common open challenges, and give perspectives on future research directions, looking to inspire interdisciplinary approaches.Comment: 46 pages, 9 figure