Local Communication Protocols for Learning Complex Swarm Behaviors with Deep Reinforcement Learning

Swarm systems constitute a challenging problem for reinforcement learning (RL) as the algorithm needs to learn decentralized control policies that can cope with limited local sensing and communication abilities of the agents. While it is often difficult to directly define the behavior of the agents, simple communication protocols can be defined more easily using prior knowledge about the given task. In this paper, we propose a number of simple communication protocols that can be exploited by deep reinforcement learning to find decentralized control policies in a multi-robot swarm environment. The protocols are based on histograms that encode the local neighborhood relations of the agents and can also transmit task-specific information, such as the shortest distance and direction to a desired target. In our framework, we use an adaptation of Trust Region Policy Optimization to learn complex collaborative tasks, such as formation building and building a communication link. We evaluate our findings in a simulated 2D-physics environment, and compare the implications of different communication protocols.Comment: 13 pages, 4 figures, version 2, accepted at ANTS 201

Embodied Evolution in Collective Robotics: A Review

This paper provides an overview of evolutionary robotics techniques applied to on-line distributed evolution for robot collectives -- namely, embodied evolution. It provides a definition of embodied evolution as well as a thorough description of the underlying concepts and mechanisms. The paper also presents a comprehensive summary of research published in the field since its inception (1999-2017), providing various perspectives to identify the major trends. In particular, we identify a shift from considering embodied evolution as a parallel search method within small robot collectives (fewer than 10 robots) to embodied evolution as an on-line distributed learning method for designing collective behaviours in swarm-like collectives. The paper concludes with a discussion of applications and open questions, providing a milestone for past and an inspiration for future research.Comment: 23 pages, 1 figure, 1 tabl

Guided Deep Reinforcement Learning for Swarm Systems

In this paper, we investigate how to learn to control a group of cooperative agents with limited sensing capabilities such as robot swarms. The agents have only very basic sensor capabilities, yet in a group they can accomplish sophisticated tasks, such as distributed assembly or search and rescue tasks. Learning a policy for a group of agents is difficult due to distributed partial observability of the state. Here, we follow a guided approach where a critic has central access to the global state during learning, which simplifies the policy evaluation problem from a reinforcement learning point of view. For example, we can get the positions of all robots of the swarm using a camera image of a scene. This camera image is only available to the critic and not to the control policies of the robots. We follow an actor-critic approach, where the actors base their decisions only on locally sensed information. In contrast, the critic is learned based on the true global state. Our algorithm uses deep reinforcement learning to approximate both the Q-function and the policy. The performance of the algorithm is evaluated on two tasks with simple simulated 2D agents: 1) finding and maintaining a certain distance to each others and 2) locating a target.Comment: 15 pages, 8 figures, accepted at the AAMAS 2017 Autonomous Robots and Multirobot Systems (ARMS) Worksho

Quality-sensitive foraging by a robot swarm through virtual pheromone trails

Large swarms of simple autonomous robots can be employed to find objects clustered at random locations, and transport them to a central depot. This solution offers system parallelisation through concurrent environment exploration and object collection by several robots, but it also introduces the challenge of robot coordination. Inspired by ants’ foraging behaviour, we successfully tackle robot swarm coordination through indirect stigmergic communication in the form of virtual pheromone trails. We design and implement a robot swarm composed of up to 100 Kilobots using the recent technology Augmented Reality for Kilobots (ARK). Using pheromone trails, our memoryless robots rediscover object sources that have been located previously. The emerging collective dynamics show a throughput inversely proportional to the source distance. We assume environments with multiple sources, each providing objects of different qualities, and we investigate how the robot swarm balances the quality-distance trade-off by using quality-sensitive pheromone trails. To our knowledge this work represents the largest robotic experiment in stigmergic foraging, and is the first complete demonstration of ARK, showcasing the set of unique functionalities it provides

Sophisticated collective foraging with minimalist agents: a swarm robotics test

How groups of cooperative foragers can achieve efficient and robust collective foraging is of interest both to biologists studying social insects and engineers designing swarm robotics systems. Of particular interest are distance-quality trade-offs and swarm-size-dependent foraging strategies. Here we present a collective foraging system based on virtual pheromones, tested in simulation and in swarms of up to 200 physical robots. Our individual agent controllers are highly simplified, as they are based on binary pheromone sensors. Despite being simple, our individual controllers are able to reproduce classical foraging experiments conducted with more capable real ants that sense pheromone concentration and follow its gradient. One key feature of our controllers is a control parameter which balances the trade-off between distance selectivity and quality selectivity of individual foragers. We construct an optimal foraging theory model that accounts for distance and quality of resources, as well as overcrowding, and predicts a swarmsize-dependent strategy. We test swarms implementing our controllers against our optimality model and find that, for moderate swarm sizes, they can be parameterised to approximate the optimal foraging strategy. This study demonstrates the sufficiency of simple individual agent rules to generate sophisticated collective foraging behaviour

Path Planning of Mobile Agents using AI Technique

In this paper, we study coordinated motion in a swarm robotic system, called a swarm-bot. A swarm-bot is a self-assembling and self-organizing. Artifact composed of a swarm of s-bots, mobile robots with the ability to connect to and is connect from each other. The swarm-bot concept is particularly suited for tasks that require all-terrain navigation abilities, such as space exploration or rescue in collapsed buildings. As a first step toward the development of more complex control strategies, we investigate the case in which a swarm-bot has to explore an arena while avoiding falling into holes. In such a scenario, individual s-bots have sensory–motor limitations that prevent them navigating efficiently. These limitations can be overcome if the s-bots are made to cooperate. In particular, we exploit the s-bots’ ability to physically connect to each other. In order to synthesize the s-bots’ controller, we rely on artificial evolution, which we show to be a powerful tool for the production of simple and effective solutions to the hole avoidance task

Behavioral responses to colony-level properties affect disturbance resistance of red harvester ant colonies

Self-organizing biological systems, such as colonies of social insects, are characterized by their decentralized control and flexible responses to changing environments, often likened to swarm intelligence. Although decentralized control is well known to be a product of local interactions among agents, without the need for a bird’s-eye view, indirect knowledge of properties that indicate the current states of the entire system also helps each agent to respond to changes, thereby leading to a more adaptive system. In this study, we analyze the rules that govern workers’ behavioral responses to colony-level properties and assess whether they contribute to adaptive flexibility in social insect colonies. We focus on task allocation among red harvester ants (Pogonomyrmex barbatus) as a model system and develop an ordinary differential equation model to describe the system of task allocation among workers. We simulate 12 scenarios specifying how workers respond to changes in the colony-level properties of colony size and nutritional state. We found that when workers decrease their contact rates in response to increasing colony size, they enable achievement of a larger colony size, similar to that of P. barbatus colonies in nature, and when workers increase their foraging levels in response to decreasing colony-wide nutritional levels, they increase resilience to environmental disturbances. These negative feedback rules governing the response to colony-level properties are consistent with previous reports on ants and honeybees