22 research outputs found
Deep Reinforcement Learning for Swarm Systems
Recently, deep reinforcement learning (RL) methods have been applied
successfully to multi-agent scenarios. Typically, these methods rely on a
concatenation of agent states to represent the information content required for
decentralized decision making. However, concatenation scales poorly to swarm
systems with a large number of homogeneous agents as it does not exploit the
fundamental properties inherent to these systems: (i) the agents in the swarm
are interchangeable and (ii) the exact number of agents in the swarm is
irrelevant. Therefore, we propose a new state representation for deep
multi-agent RL based on mean embeddings of distributions. We treat the agents
as samples of a distribution and use the empirical mean embedding as input for
a decentralized policy. We define different feature spaces of the mean
embedding using histograms, radial basis functions and a neural network learned
end-to-end. We evaluate the representation on two well known problems from the
swarm literature (rendezvous and pursuit evasion), in a globally and locally
observable setup. For the local setup we furthermore introduce simple
communication protocols. Of all approaches, the mean embedding representation
using neural network features enables the richest information exchange between
neighboring agents facilitating the development of more complex collective
strategies.Comment: 31 pages, 12 figures, version 3 (published in JMLR Volume 20
Adversarial Search and Tracking with Multiagent Reinforcement Learning in Sparsely Observable Environment
We study a search and tracking (S&T) problem where a team of dynamic search
agents must collaborate to track an adversarial, evasive agent. The
heterogeneous search team may only have access to a limited number of past
adversary trajectories within a large search space. This problem is challenging
for both model-based searching and reinforcement learning (RL) methods since
the adversary exhibits reactionary and deceptive evasive behaviors in a large
space leading to sparse detections for the search agents. To address this
challenge, we propose a novel Multi-Agent RL (MARL) framework that leverages
the estimated adversary location from our learnable filtering model. We show
that our MARL architecture can outperform all baselines and achieves a 46%
increase in detection rate.Comment: Accepted by IEEE International Symposium on Multi-Robot & Multi-Agent
Systems (MRS) 202
Hierarchical Imitation Learning for Stochastic Environments
Many applications of imitation learning require the agent to generate the
full distribution of behaviour observed in the training data. For example, to
evaluate the safety of autonomous vehicles in simulation, accurate and diverse
behaviour models of other road users are paramount. Existing methods that
improve this distributional realism typically rely on hierarchical policies.
These condition the policy on types such as goals or personas that give rise to
multi-modal behaviour. However, such methods are often inappropriate for
stochastic environments where the agent must also react to external factors:
because agent types are inferred from the observed future trajectory during
training, these environments require that the contributions of internal and
external factors to the agent behaviour are disentangled and only internal
factors, i.e., those under the agent's control, are encoded in the type.
Encoding future information about external factors leads to inappropriate agent
reactions during testing, when the future is unknown and types must be drawn
independently from the actual future. We formalize this challenge as
distribution shift in the conditional distribution of agent types under
environmental stochasticity. We propose Robust Type Conditioning (RTC), which
eliminates this shift with adversarial training under randomly sampled types.
Experiments on two domains, including the large-scale Waymo Open Motion
Dataset, show improved distributional realism while maintaining or improving
task performance compared to state-of-the-art baselines.Comment: Published at IROS'2
Measuring Policy Distance for Multi-Agent Reinforcement Learning
Diversity plays a crucial role in improving the performance of multi-agent
reinforcement learning (MARL). Currently, many diversity-based methods have
been developed to overcome the drawbacks of excessive parameter sharing in
traditional MARL. However, there remains a lack of a general metric to quantify
policy differences among agents. Such a metric would not only facilitate the
evaluation of the diversity evolution in multi-agent systems, but also provide
guidance for the design of diversity-based MARL algorithms. In this paper, we
propose the multi-agent policy distance (MAPD), a general tool for measuring
policy differences in MARL. By learning the conditional representations of
agents' decisions, MAPD can computes the policy distance between any pair of
agents. Furthermore, we extend MAPD to a customizable version, which can
quantify differences among agent policies on specified aspects. Based on the
online deployment of MAPD, we design a multi-agent dynamic parameter sharing
(MADPS) algorithm as an example of the MAPD's applications. Extensive
experiments demonstrate that our method is effective in measuring differences
in agent policies and specific behavioral tendencies. Moreover, in comparison
to other methods of parameter sharing, MADPS exhibits superior performance.Comment: 9 pages, 6 figure
Survey of Recent Multi-Agent Reinforcement Learning Algorithms Utilizing Centralized Training
Much work has been dedicated to the exploration of Multi-Agent Reinforcement
Learning (MARL) paradigms implementing a centralized learning with
decentralized execution (CLDE) approach to achieve human-like collaboration in
cooperative tasks. Here, we discuss variations of centralized training and
describe a recent survey of algorithmic approaches. The goal is to explore how
different implementations of information sharing mechanism in centralized
learning may give rise to distinct group coordinated behaviors in multi-agent
systems performing cooperative tasks.Comment: This article appeared in the news at:
https://www.army.mil/article/247261/army_researchers_develop_innovative_framework_for_training_a