315 research outputs found
Intrinsic fluctuations of reinforcement learning promote cooperation
In this work, we ask for and answer what makes classical reinforcement
learning cooperative. Cooperating in social dilemma situations is vital for
animals, humans, and machines. While evolutionary theory revealed a range of
mechanisms promoting cooperation, the conditions under which agents learn to
cooperate are contested. Here, we demonstrate which and how individual elements
of the multi-agent learning setting lead to cooperation. Specifically, we
consider the widely used temporal-difference reinforcement learning algorithm
with epsilon-greedy exploration in the classic environment of an iterated
Prisoner's dilemma with one-period memory. Each of the two learning agents
learns a strategy that conditions the following action choices on both agents'
action choices of the last round. We find that next to a high caring for future
rewards, a low exploration rate, and a small learning rate, it is primarily
intrinsic stochastic fluctuations of the reinforcement learning process which
double the final rate of cooperation to up to 80\%. Thus, inherent noise is not
a necessary evil of the iterative learning process. It is a critical asset for
the learning of cooperation. However, we also point out the trade-off between a
high likelihood of cooperative behavior and achieving this in a reasonable
amount of time. Our findings are relevant for purposefully designing
cooperative algorithms and regulating undesired collusive effects.Comment: 9 pages, 4 figure
Recommended from our members
From math to metaphors and back again: Social-ecological resilience from a multi-agent-environment perspective
Science and policy stand to benefit from reconnecting the many notions of social-ecological resilience to their roots in complexity sciences.We propose several ways of moving towards operationalization through the classification of modern concepts of resilience based on a multi-agent-environment perspective.
Social-ecological resilience underlies popular sustainability concepts that have been influential in formulating the United Nations Sustainable Development Goals (SDGs), such as the Planetary Boundaries and Doughnut Economics. Scientific investigation of these concepts is supported by mathematical models of planetary biophysical and societal dynamics, both of which call for operational measures of resilience. However, current quantitative descriptions tend to be restricted to the foundational form of the concept: persistence resilience. We propose a classification of modern notions of social-ecological resilience from a multi-agent-environment perspective. This aims at operationalization in a complex systems framework, including the persistence, adaptation and transformation aspects of resilience, normativity related to desirable system function, first- vs. second-order and specific vs. general resilience. For example, we discuss the use of the Topology of Sustainable Management Framework. Developing the mathematics of resilience along these lines would not only make social-ecological resilience more applicable to data and models, but could also conceptually advance resilience thinking
When optimization for governing human environment tipping elements is neither sustainable nor safe
Optimizing economic welfare in environmental governance has been criticized for delivering short-term gains at the expense of long-term environmental degradation. Different from economic optimization, the concepts of sustainability and the more recent safe operating space have been used to derive policies in environmental governance. However, a formal comparison between these three policy paradigms is still missing, leaving policy makers uncertain which paradigm to apply. Here, we develop a better understanding of their interrelationships, using a stylized model of human-environment tipping elements. We find that no paradigm guarantees fulfilling requirements imposed by another paradigm and derive simple heuristics for the conditions under which these trade-offs occur. We show that the absence of such a master paradigm is of special relevance for governing real-world tipping systems such as climate, fisheries, and farming, which may reside in a parameter regime where economic optimization is neither sustainable nor safe.The
authors are grateful for financial support from the Heinrich-Böll-Foundation, the
Stordalen Foundation (via the Planetary Boundaries Research Network PB.net), the Earth
League’s EarthDoc program, the Leibniz Association (project DOMINOES) and the
Swedish Research Council Formas (Project Grant 2014-589)
Recommended from our members
Sustainable use of renewable resources in a stylized social–ecological network model under heterogeneous resource distribution
Human societies depend on the resources ecosystems provide. Particularly since the last century,
human activities have transformed the relationship between nature and society at a global scale. We study this
coevolutionary relationship by utilizing a stylized model of private resource use and social learning on an adaptive
network. The latter process is based on two social key dynamics beyond economic paradigms: boundedly
rational imitation of resource use strategies and homophily in the formation of social network ties. The private
and logistically growing resources are harvested with either a sustainable (small) or non-sustainable (large) effort.
We show that these social processes can have a profound influence on the environmental state, such as
determining whether the private renewable resources collapse from overuse or not. Additionally, we demonstrate
that heterogeneously distributed regional resource capacities shift the critical social parameters where this
resource extraction system collapses. We make these points to argue that, in more advanced coevolutionary
models of the planetary social–ecological system, such socio-cultural phenomena as well as regional resource
heterogeneities should receive attention in addition to the processes represented in established Earth system and
integrated assessment model
Parsimonious modeling with information filtering networks
We introduce a methodology to construct parsimonious probabilistic models. This method makes use of information filtering networks to produce a robust estimate of the global sparse inverse covariance from a simple sum of local inverse covariances computed on small subparts of the network. Being based on local and low-dimensional inversions, this method is computationally very efficient and statistically robust, even for the estimation of inverse covariance of high-dimensional, noisy, and short time series. Applied to financial data our method results are computationally more efficient than state-of-the-art methodologies such as Glasso producing, in a fraction of the computation time, models that can have equivalent or better performances but with a sparser inference structure. We also discuss performances with sparse factor models where we notice that relative performances decrease with the number of factors. The local nature of this approach allows us to perform computations in parallel and provides a tool for dynamical adaptation by partial updating when the properties of some variables change without the need of recomputing the whole model. This makes this approach particularly suitable to handle big data sets with large numbers of variables. Examples of practical application for forecasting, stress testing, and risk allocation in financial systems are also provided
Earth system modeling with endogenous and dynamic human societies: the copan:CORE open World-Earth modeling framework
Analysis of Earth system dynamics in the Anthropocene requires to explicitly
take into account the increasing magnitude of processes operating in human
societies, their cultures, economies and technosphere and their growing
feedback entanglement with those in the physical, chemical and biological
systems of the planet. However, current state-of-the-art Earth System Models do
not represent dynamic human societies and their feedback interactions with the
biogeophysical Earth system and macroeconomic Integrated Assessment Models
typically do so only with limited scope. This paper (i) proposes design
principles for constructing World-Earth Models (WEM) for Earth system analysis
of the Anthropocene, i.e., models of social (World) - ecological (Earth)
co-evolution on up to planetary scales, and (ii) presents the copan:CORE open
simulation modeling framework for developing, composing and analyzing such WEMs
based on the proposed principles. The framework provides a modular structure to
flexibly construct and study WEMs. These can contain biophysical (e.g. carbon
cycle dynamics), socio-metabolic/economic (e.g. economic growth) and
socio-cultural processes (e.g. voting on climate policies or changing social
norms) and their feedback interactions, and are based on elementary entity
types, e.g., grid cells and social systems. Thereby, copan:CORE enables the
epistemic flexibility needed for contributions towards Earth system analysis of
the Anthropocene given the large diversity of competing theories and
methodologies used for describing socio-metabolic/economic and socio-cultural
processes in the Earth system by various fields and schools of thought. To
illustrate the capabilities of the framework, we present an exemplary and
highly stylized WEM implemented in copan:CORE that illustrates how endogenizing
socio-cultural processes and feedbacks could fundamentally change macroscopic
model outcomes
Perspectives on adaptive dynamical systems
Adaptivity is a dynamical feature that is omnipresent in nature, socio-economics, and technology. For example, adaptive couplings appear in various real-world systems, such as the power grid, social, and neural networks, and they form the backbone of closed-loop control strategies and machine learning algorithms. In this article, we provide an interdisciplinary perspective on adaptive systems. We reflect on the notion and terminology of adaptivity in different disciplines and discuss which role adaptivity plays for various fields. We highlight common open challenges and give perspectives on future research directions, looking to inspire interdisciplinary approaches
Perspectives on adaptive dynamical systems
Adaptivity is a dynamical feature that is omnipresent in nature,
socio-economics, and technology. For example, adaptive couplings appear in
various real-world systems like the power grid, social, and neural networks,
and they form the backbone of closed-loop control strategies and machine
learning algorithms. In this article, we provide an interdisciplinary
perspective on adaptive systems. We reflect on the notion and terminology of
adaptivity in different disciplines and discuss which role adaptivity plays for
various fields. We highlight common open challenges, and give perspectives on
future research directions, looking to inspire interdisciplinary approaches.Comment: 46 pages, 9 figure
- …