Local Communication Protocols for Learning Complex Swarm Behaviors with Deep Reinforcement Learning

Swarm systems constitute a challenging problem for reinforcement learning (RL) as the algorithm needs to learn decentralized control policies that can cope with limited local sensing and communication abilities of the agents. While it is often difficult to directly define the behavior of the agents, simple communication protocols can be defined more easily using prior knowledge about the given task. In this paper, we propose a number of simple communication protocols that can be exploited by deep reinforcement learning to find decentralized control policies in a multi-robot swarm environment. The protocols are based on histograms that encode the local neighborhood relations of the agents and can also transmit task-specific information, such as the shortest distance and direction to a desired target. In our framework, we use an adaptation of Trust Region Policy Optimization to learn complex collaborative tasks, such as formation building and building a communication link. We evaluate our findings in a simulated 2D-physics environment, and compare the implications of different communication protocols.Comment: 13 pages, 4 figures, version 2, accepted at ANTS 201

Artificial Pheromone for Path Selection by a Foraging Swarm of Robots

Foraging robots involved in a search and retrieval task may create paths to navigate faster in their environment. In this context, a swarm of robots that has found several resources and created different paths may benefit strongly from path selection. Path selection enhances the foraging behavior by allowing the swarm to focus on the most profitable resource with the possibility for unused robots to stop participating in the path maintenance and to switch to another task. In order to achieve path selection, we implement virtual ants that lay artificial pheromone inside a network of robots. Virtual ants are local messages transmitted by robots; they travel along chains of robots and deposit artificial pheromone on the robots that are literally forming the chain and indicating the path. The concentration of artificial pheromone on the robots allows them to decide whether they are part of a selected path. We parameterize the mechanism with a mathematical model and provide an experimental validation using a swarm of 20 real robots. We show that our mechanism favors the selection of the closest resource is able to select a new path if a selected resource becomes unavailable and selects a newly detected and better resource when possible. As robots use very simple messages and behaviors, the system would be particularly well suited for swarms of microrobots with minimal abilitie

Subsumption architecture for enabling strategic coordination of robot swarms in a gaming scenario

The field of swarm robotics breaks away from traditional research by maximizing the performance of a group - swarm - of limited robots instead of optimizing the intelligence of a single robot. Similar to current-generation strategy video games, the player controls groups of units - squads - instead of the individual participants. These individuals are rather unintelligent robots, capable of little more than navigating and using their weapons. However, clever control of the squads of autonomous robots by the game players can make for intense, strategic matches. The gaming framework presented in this article provides players with strategic coordination of robot squads. The developed swarm intelligence techniques break up complex squad commands into several commands for each robot using robot formations and path finding while avoiding obstacles. These algorithms are validated through a 'Capture the Flag' gaming scenario where a complex squad command is split up into several robot commands in a matter of milliseconds

Negotiation of goal direction for cooperative transport

In this paper, we study the cooperative transport of a heavy object by a group of robots towards a goal. We investigate the case in which robots have partial and noisy knowledge of the goal direction and can not perceive the goal itself. The robots have to coordinate their motion to apply enough force on the object to move it. Furthermore, the robots should share knowledge in order to collectively improve their estimate of the goal direction and transport the object as fast and as accurately as possible towards the goal. We propose a bio-inspired mechanism of negotiation of direction that is fully distributed. Four different strategies are implemented and their performances are compared on a group of four real robots, varying the goal direction and the level of noise. We identify a strategy that enables effcient coordination of motion of the robots. Moreover, this strategy lets the robots improve their knowledge of the goal direction. Despite significant noise in the robots' communication, we achieve effective cooperative transport towards the goal and observe that the negotiation of direction entails interesting properties of robustness

Improving Social Odometry Robot Networks with Distributed Reputation Systems for Collaborative Purposes

The improvement of odometry systems in collaborative robotics remains an important challenge for several applications. Social odometry is a social technique which confers the robots the possibility to learn from the others. This paper analyzes social odometry and proposes and follows a methodology to improve its behavior based on cooperative reputation systems. We also provide a reference implementation that allows us to compare the performance of the proposed solution in highly dynamic environments with the performance of standard social odometry techniques. Simulation results quantitatively show the benefits of this collaborative approach that allows us to achieve better performances than social odometry

Embodied Evolution in Collective Robotics: A Review

This paper provides an overview of evolutionary robotics techniques applied to on-line distributed evolution for robot collectives -- namely, embodied evolution. It provides a definition of embodied evolution as well as a thorough description of the underlying concepts and mechanisms. The paper also presents a comprehensive summary of research published in the field since its inception (1999-2017), providing various perspectives to identify the major trends. In particular, we identify a shift from considering embodied evolution as a parallel search method within small robot collectives (fewer than 10 robots) to embodied evolution as an on-line distributed learning method for designing collective behaviours in swarm-like collectives. The paper concludes with a discussion of applications and open questions, providing a milestone for past and an inspiration for future research.Comment: 23 pages, 1 figure, 1 tabl

Physical interactions in swarm robotics: the hand-bot case study

This paper presents a case-study on the performance achieved by the mechanical interactions of self-assembling mobile robots. This study is based on the hand-bot robot, designed to operate within heterogeneous swarms of robots. The hand-bot is specialized in object manipulation and can improve its performance by exploiting physical collaborations by self-assembling with other hand-bots or with foot-bots (ground robots). The paper analyzes the achieved performance and demonstrates the highly super-linear properties of the accessible volume in respect to the number of robots. These extremely interesting performances are strongly linked to the self-assembling mechanisms and the physical nature of the interaction, and do not scale to a large number of robots. Finally, this study suggests that such interesting properties are more accessible for heterogeneous systems or devices achieving complex tasks

Quality-sensitive foraging by a robot swarm through virtual pheromone trails

Large swarms of simple autonomous robots can be employed to find objects clustered at random locations, and transport them to a central depot. This solution offers system parallelisation through concurrent environment exploration and object collection by several robots, but it also introduces the challenge of robot coordination. Inspired by ants’ foraging behaviour, we successfully tackle robot swarm coordination through indirect stigmergic communication in the form of virtual pheromone trails. We design and implement a robot swarm composed of up to 100 Kilobots using the recent technology Augmented Reality for Kilobots (ARK). Using pheromone trails, our memoryless robots rediscover object sources that have been located previously. The emerging collective dynamics show a throughput inversely proportional to the source distance. We assume environments with multiple sources, each providing objects of different qualities, and we investigate how the robot swarm balances the quality-distance trade-off by using quality-sensitive pheromone trails. To our knowledge this work represents the largest robotic experiment in stigmergic foraging, and is the first complete demonstration of ARK, showcasing the set of unique functionalities it provides

Modelling and Verification of Timed Robotic Controllers

Designing robotic systems can be very challenging, yet controllers are often specified using informal notations with development driven primarily by simulations and physical experiments, without relation to abstract models of requirements. The ability to perform formal analysis and replicate results across different robotic platforms is hindered by the lack of well-defined formal notations. In this paper we present a timed state-machine based formal notation for robotics that is informed by current practice. We motivate our work with an example from swarm robotics and define a compositional CSP-based discrete timed semantics suitable for refinement. Our results support verification and, importantly, enable rigorous connection with sound simulations and deployments.</p