11,812 research outputs found
Biases for Emergent Communication in Multi-agent Reinforcement Learning
We study the problem of emergent communication, in which language arises
because speakers and listeners must communicate information in order to solve
tasks. In temporally extended reinforcement learning domains, it has proved
hard to learn such communication without centralized training of agents, due in
part to a difficult joint exploration problem. We introduce inductive biases
for positive signalling and positive listening, which ease this problem. In a
simple one-step environment, we demonstrate how these biases ease the learning
problem. We also apply our methods to a more extended environment, showing that
agents with these inductive biases achieve better performance, and analyse the
resulting communication protocols.Comment: Accepted at NeurIPS 201
Adaptive Load Balancing: A Study in Multi-Agent Learning
We study the process of multi-agent reinforcement learning in the context of
load balancing in a distributed system, without use of either central
coordination or explicit communication. We first define a precise framework in
which to study adaptive load balancing, important features of which are its
stochastic nature and the purely local information available to individual
agents. Given this framework, we show illuminating results on the interplay
between basic adaptive behavior parameters and their effect on system
efficiency. We then investigate the properties of adaptive load balancing in
heterogeneous populations, and address the issue of exploration vs.
exploitation in that context. Finally, we show that naive use of communication
may not improve, and might even harm system efficiency.Comment: See http://www.jair.org/ for any accompanying file
Learning with Opponent-Learning Awareness
Multi-agent settings are quickly gathering importance in machine learning.
This includes a plethora of recent work on deep multi-agent reinforcement
learning, but also can be extended to hierarchical RL, generative adversarial
networks and decentralised optimisation. In all these settings the presence of
multiple learning agents renders the training problem non-stationary and often
leads to unstable training or undesired final results. We present Learning with
Opponent-Learning Awareness (LOLA), a method in which each agent shapes the
anticipated learning of the other agents in the environment. The LOLA learning
rule includes a term that accounts for the impact of one agent's policy on the
anticipated parameter update of the other agents. Results show that the
encounter of two LOLA agents leads to the emergence of tit-for-tat and
therefore cooperation in the iterated prisoners' dilemma, while independent
learning does not. In this domain, LOLA also receives higher payouts compared
to a naive learner, and is robust against exploitation by higher order
gradient-based methods. Applied to repeated matching pennies, LOLA agents
converge to the Nash equilibrium. In a round robin tournament we show that LOLA
agents successfully shape the learning of a range of multi-agent learning
algorithms from literature, resulting in the highest average returns on the
IPD. We also show that the LOLA update rule can be efficiently calculated using
an extension of the policy gradient estimator, making the method suitable for
model-free RL. The method thus scales to large parameter and input spaces and
nonlinear function approximators. We apply LOLA to a grid world task with an
embedded social dilemma using recurrent policies and opponent modelling. By
explicitly considering the learning of the other agent, LOLA agents learn to
cooperate out of self-interest. The code is at github.com/alshedivat/lola
Multi-lingual agents through multi-headed neural networks
This paper considers cooperative Multi-Agent Reinforcement Learning, focusing
on emergent communication in settings where multiple pairs of independent
learners interact at varying frequencies. In this context, multiple distinct
and incompatible languages can emerge. When an agent encounters a speaker of an
alternative language, there is a requirement for a period of adaptation before
they can efficiently converse. This adaptation results in the emergence of a
new language and the forgetting of the previous language. In principle, this is
an example of the Catastrophic Forgetting problem which can be mitigated by
enabling the agents to learn and maintain multiple languages. We take
inspiration from the Continual Learning literature and equip our agents with
multi-headed neural networks which enable our agents to be multi-lingual. Our
method is empirically validated within a referential MNIST based communication
game and is shown to be able to maintain multiple languages where existing
approaches cannot.Comment: Cooperative AI workshop NeurIPS 202
Towards More Human-like AI Communication: A Review of Emergent Communication Research
In the recent shift towards human-centric AI, the need for machines to
accurately use natural language has become increasingly important. While a
common approach to achieve this is to train large language models, this method
presents a form of learning misalignment where the model may not capture the
underlying structure and reasoning humans employ in using natural language,
potentially leading to unexpected or unreliable behavior. Emergent
communication (Emecom) is a field of research that has seen a growing number of
publications in recent years, aiming to develop artificial agents capable of
using natural language in a way that goes beyond simple discriminative tasks
and can effectively communicate and learn new concepts. In this review, we
present Emecom under two aspects. Firstly, we delineate all the common
proprieties we find across the literature and how they relate to human
interactions. Secondly, we identify two subcategories and highlight their
characteristics and open challenges. We encourage researchers to work together
by demonstrating that different methods can be viewed as diverse solutions to a
common problem and emphasize the importance of including diverse perspectives
and expertise in the field. We believe a deeper understanding of human
communication is crucial to developing machines that can accurately use natural
language in human-machine interactions.Comment: 25 pages, 9 figures, 2 table
- …