Guided Deep Reinforcement Learning for Swarm Systems

In this paper, we investigate how to learn to control a group of cooperative agents with limited sensing capabilities such as robot swarms. The agents have only very basic sensor capabilities, yet in a group they can accomplish sophisticated tasks, such as distributed assembly or search and rescue tasks. Learning a policy for a group of agents is difficult due to distributed partial observability of the state. Here, we follow a guided approach where a critic has central access to the global state during learning, which simplifies the policy evaluation problem from a reinforcement learning point of view. For example, we can get the positions of all robots of the swarm using a camera image of a scene. This camera image is only available to the critic and not to the control policies of the robots. We follow an actor-critic approach, where the actors base their decisions only on locally sensed information. In contrast, the critic is learned based on the true global state. Our algorithm uses deep reinforcement learning to approximate both the Q-function and the policy. The performance of the algorithm is evaluated on two tasks with simple simulated 2D agents: 1) finding and maintaining a certain distance to each others and 2) locating a target.Comment: 15 pages, 8 figures, accepted at the AAMAS 2017 Autonomous Robots and Multirobot Systems (ARMS) Worksho

Human Swarm Interaction: An Experimental Study of Two Types of Interaction with Foraging Swarms

In this paper we present the first study of human-swarm interaction comparing two fundamental types of interaction, coined intermittent and environmental. These types are exemplified by two control methods, selection and beacon control, made available to a human operator to control a foraging swarm of robots. Selection and beacon control differ with respect to their temporal and spatial influence on the swarm and enable an operator to generate different strategies from the basic behaviors of the swarm. Selection control requires an active selection of groups of robots while beacon control exerts an influence on nearby robots within a set range. Both control methods are implemented in a testbed in which operators solve an information foraging problem by utilizing a set of swarm behaviors. The robotic swarm has only local communication and sensing capabilities. The number of robots in the swarm range from 50 to 200. Operator performance for each control method is compared in a series of missions in different environments with no obstacles up to cluttered and structured obstacles. In addition, performance is compared to simple and advanced autonomous swarms. Thirty-two participants were recruited for participation in the study. Autonomous swarm algorithms were tested in repeated simulations. Our results showed that selection control scales better to larger swarms and generally outperforms beacon control. Operators utilized different swarm behaviors with different frequency across control methods, suggesting an adaptation to different strategies induced by choice of control method. Simple autonomous swarms outperformed human operators in open environments, but operators adapted better to complex environments with obstacles. Human controlled swarms fell short of task-specific benchmarks under all conditions. Our results reinforce the importance of understanding and choosing appropriate types of human-swarm interaction when designing swarm systems, in addition to choosing appropriate swarm behaviors

Improving Robotic Decision-Making in Unmodeled Situations

Existing methods of autonomous robotic decision-making are often fragile when faced with inaccurate or incompletely modeled distributions of uncertainty, also known as ambiguity. While decision-making under ambiguity is a field of study that has been gaining interest, many existing methods tend to be computationally challenging, require many assumptions about the nature of the problem, and often require much prior knowledge. Therefore, they do not scale well to complex real-world problems where fulfilling all of these requirements is often impractical if not impossible. The research described in this dissertation investigates novel approaches to robotic decision-making strategies which are resilient to ambiguity that are not subject to as many of these requirements as most existing methods. The novel frameworks described in this research incorporate physical feedback, diversity, and swarm local interactions, three factors that are hypothesized to be key in creating resilience to ambiguity. These three factors are inspired by examples of robots which demonstrate resilience to ambiguity, ranging from simple vibrobots to decentralized robotic swarms. The proposed decision-making methods, based around a proposed framework known as Ambiguity Trial and Error (AT&E), are tested for both single robots and robotic swarms in several simulated robotic foraging case studies, and a real-world robotic foraging experiment. A novel method for transferring swarm resilience properties back to single agent decision-making is also explored. The results from the case studies show that the proposed methods demonstrate resilience to varying types of ambiguities, both stationary and non-stationary, while not requiring accurate modeling and assumptions, large amounts of prior training data, or computationally expensive decision-making policy solvers. Conclusions about these novel methods are then drawn from the simulation and experiment results and the future research directions leveraging the lessons learned from this research are discussed

Scalable target detection for large robot teams

In this paper, we present an asynchronous display method, coined image queue, which allows operators to search through a large amount of data gathered by autonomous robot teams. We discuss and investigate the advantages of an asynchronous display for foraging tasks with emphasis on Urban Search and Rescue. The image queue approach mines video data to present the operator with a relevant and comprehensive view of the environment in order to identify targets of interest such as injured victims. It fills the gap for comprehensive and scalable displays to obtain a network-centric perspective for UGVs. We compared the image queue to a traditional synchronous display with live video feeds and found that the image queue reduces errors and operator's workload. Furthermore, it disentangles target detection from concurrent system operations and enables a call center approach to target detection. With such an approach we can scale up to very large multi-robot systems gathering huge amounts of data that is then distributed to multiple operators. Copyright 2011 ACM