Analysis of multi-agent systems under varying degrees of trust, cooperation, and competition

Multi-agent systems rely heavily on coordination and cooperation to achieve a variety of tasks. It is often assumed that these agents will be fully cooperative, or have reliable and equal performance among group members. Instead, we consider cooperation as a spectrum of possible interactions, ranging from performance variations within the group to adversarial agents. This thesis examines several scenarios where cooperation and performance are not guaranteed. Potential applications include sensor coverage, emergency response, wildlife management, tracking, and surveillance. We use geometric methods, such as Voronoi tessellations, for design insight and Lyapunov-based stability theory to analyze our proposed controllers. Performance is verified through simulations and experiments on a variety of ground and aerial robotic platforms. First, we consider the problem of Voronoi-based coverage control, where a group of robots must spread out over an environment to provide coverage. Our approach adapts online to sensing and actuation performance variations with the group. The robots have no prior knowledge of their relative performance, and in a distributed fashion, compensate by assigning weaker robots a smaller portion of the environment. Next, we consider the problem of multi-agent herding, akin to shepherding. Here, a group of dog-like robots must drive a herd of non-cooperative sheep-like agents around the environment. Our key insight in designing the control laws for the herders is to enforce geometrical relationships that allow for the combined system dynamics to reduce to a single nonholonomic vehicle. We also investigate the cooperative pursuit of an evader by a group of quadrotors in an environment with no-fly zones. While the pursuers cannot enter the no-fly zones, the evader moves freely through the zones to avoid capture. Using tools for Voronoi-based coverage control, we provide an algorithm to distribute the pursuers around the zone's boundary and minimize capture time once the evader emerges. Finally, we present an algorithm for the guaranteed capture of multiple evaders by one or more pursuers in a bounded, convex environment. The pursuers utilize properties of the evader's Voronoi cell to choose a control strategy that minimizes the safe-reachable area of the evader, which in turn leads to the evader's capture

Deep Reinforcement Learning for Swarm Systems

Recently, deep reinforcement learning (RL) methods have been applied successfully to multi-agent scenarios. Typically, these methods rely on a concatenation of agent states to represent the information content required for decentralized decision making. However, concatenation scales poorly to swarm systems with a large number of homogeneous agents as it does not exploit the fundamental properties inherent to these systems: (i) the agents in the swarm are interchangeable and (ii) the exact number of agents in the swarm is irrelevant. Therefore, we propose a new state representation for deep multi-agent RL based on mean embeddings of distributions. We treat the agents as samples of a distribution and use the empirical mean embedding as input for a decentralized policy. We define different feature spaces of the mean embedding using histograms, radial basis functions and a neural network learned end-to-end. We evaluate the representation on two well known problems from the swarm literature (rendezvous and pursuit evasion), in a globally and locally observable setup. For the local setup we furthermore introduce simple communication protocols. Of all approaches, the mean embedding representation using neural network features enables the richest information exchange between neighboring agents facilitating the development of more complex collective strategies.Comment: 31 pages, 12 figures, version 3 (published in JMLR Volume 20

Virtual Target Selection for a Multiple-Pursuer Multiple-Evader Scenario

This paper considers an M-pursuer N-evader scenario involving virtual targets. The virtual targets serve as an intermediary target for the pursuers, allowing the pursuers to delay their final assignment to the evaders. However, upon reaching the virtual target, the pursuers must decide which evader to capture. It is assumed that there are more pursuers than evaders and that the pursuers are faster than the evaders. The objective is two-part: first, assign each pursuer to a virtual target and evader such that the pursuer team's energy is minimized, and second, choose the virtual targets' locations for this minimization problem. The approach taken is to consider the Apollonius geometry between each pursuer's virtual target location and each evader. Using the constructed Apollonius circles, the pursuer's travel distance and maneuver at a virtual target are obtained. These metrics serve as a gauge for the total energy required to capture a particular evader and are used to solve the joint virtual target selection and pursuer-evader assignment problem. This paper provides a mathematical definition of this problem, the solution approach taken, and an example.Comment: AIAA SciTech 2024 Preprin

Two-On-One Pursuit with a Non-zero Capture Radius

In this paper, we revisit the Two Cutters and Fugitive Ship differential game that was addressed by Isaacs, but move away from point capture. We consider a two-on-one pursuit-evasion differential game with simple motion and pursuers endowed with circular capture sets of radius l \u3e 0. The regions in the state space where only one pursuer effects the capture and the region in the state space where both pursuers cooperatively and isochronously capture the evader are characterized, thus solving the Game of Kind. Concerning the Game of Degree, the algorithm for the synthesis of the optimal state feedback strategies of the cooperating pursuers and of the evader is presented

Mobile robotic network deployment for intruder detection and tracking

This thesis investigates the problem of intruder detection and tracking using mobile robotic networks. In the first part of the thesis, we consider the problem of seeking an electromagnetic source using a team of robots that measure the local intensity of the emitted signal. We propose a planner for a team of robots based on Particle Swarm Optimization (PSO) which is a population based stochastic optimization technique. An equivalence is established between particles generated in the traditional PSO technique, and the mobile agents in the swarm. Since the positions of the robots are updated using the PSO algorithm, modifications are required to implement the PSO algorithm on real robots to incorporate collision avoidance strategies. The modifications necessary to implement PSO on mobile robots, and strategies to adapt to real environments are presented in this thesis. Our results are also validated on an experimental testbed. In the second part, we present a game theoretic framework for visibility-based target tracking in multi-robot teams. A team of observers (pursuers) and a team of targets (evaders) are present in an environment with obstacles. The objective of the team of observers is to track the team of targets for the maximum possible time. While the objective of the team of targets is to escape (break line-of-sight) in the minimum time. We decompose the problem into two layers. At the upper level, each pursuer is allocated to an evader through a minimum cost allocation strategy based on the risk of each evader, thereby, decomposing the agents into multiple single pursuer-single evader pairs. Two decentralized allocation strategies are proposed and implemented in this thesis. At the lower level, each pursuer computes its strategy based on the results of the single pursuer-single evader target-tracking problem. We initially address this problem in an environment containing a semi-infinite obstacle with one corner. The pursuer\u27s optimal tracking strategy is obtained regardless of the evader\u27s strategy using techniques from optimal control theory and differential games. Next, we extend the result to an environment containing multiple polygonal obstacles. We construct a pursuit field to provide a guiding vector for the pursuer which is a weighted sum of several component vectors. The performance of different combinations of component vectors is investigated. Finally, we extend our work to address the case when the obstacles are not polygonal, and the observers have constraints in motion

Contributions To Pursuit-Evasion Game Theory.

This dissertation studies adversarial conflicts among a group of agents moving in the plane, possibly among obstacles, where some agents are pursuers and others are evaders. The goal of the pursuers is to capture the evaders, where capture requires a pursuer to be either co-located with an evader, or in close proximity. The goal of the evaders is to avoid capture. These scenarios, where different groups compete to accomplish conflicting goals, are referred to as pursuit-evasion games, and the agents are called players. Games featuring one pursuer and one evader are analyzed using dominance, where a point in the plane is said to be dominated by a player if that player is able to reach the point before the opposing players, regardless of the opposing players' actions. Two generalizations of the Apollonius circle are provided. One solves games with environments containing obstacles, and the other provides an alternative solution method for the Homicidal Chauffeur game. Optimal pursuit and evasion strategies based on dominance are provided. One benefit of dominance analysis is that it extends to games with many players. Two foundational games are studied; one features multiple pursuers against a single evader, and the other features a single pursuer against multiple evaders. Both are solved using dominance through a reduction to single pursuer, single evader games. Another game featuring competing teams of pursuers is introduced, where an evader cooperates with friendly pursuers to rendezvous before being captured by adversaries. Next, the assumption of complete and perfect information is relaxed, and uncertainties in player speeds, player positions, obstacle locations, and cost functions are studied. The sensitivity of the dominance boundary to perturbations in parameters is provided, and probabilistic dominance is introduced. The effect of information is studied by comparing solutions of games with perfect information to games with uncertainty. Finally, a pursuit law is developed that requires minimal information and highlights a limitation of dominance regions. These contributions extend pursuit-evasion game theory to a number of games that have not previously been solved, and in some cases, the solutions presented are more amenable to implementation than previous methods.PhDAerospace EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/120650/1/dwoyler_1.pd

Reachable sets analysis in the cooperative control of pursuer vehicles

This thesis is concerned with the Pursuit-and-Evasion (PE) problem where the pursuer aims to minimize the time to capture the evader while the evader tries to prevent capture. In the problem, the evader has two advantages: a higher manoeuvrability and that the pursuer is uncertain about the evader's state. Cooperation among multiple pursuer vehicles can thus be used to overcome the evader’s advantages. The focus here is on the formulation and development of frameworks and algorithms for cooperation amongst pursuers, aiming at feasible implementation on real and autonomous vehicles. The thesis is split into Parts I and II. Part I considers the problem of capturing an evader of higher manoeuvrability in a deterministic PE game. The approach is the employment of Forward Reachable Set (FRS) analysis in the pursuers’ control. The analysis considers the coverage of the evader’s FRS, which is the set of reachable states at a future time, with the pursuer’s FRS and assumes that the chance of capturing the evader is dependent on the degree of the coverage. Using the union of multiple pursuers’ FRSs intuitively leads to more evader FRS coverage and this forms the mechanism of cooperation. A framework for cooperative control based on the FRS coverage, or FRS-based control, is proposed. Two control algorithms were developed within this framework. Part II additionally introduces the problem of evader state uncertainty due to noise and limited field-of-view of the pursuers’ sensors. A search-and-capture (SAC) problem is the result and a hybrid architecture, which includes multi-sensor estimation using the Particle Filter as well as FRS-based control, is proposed to accomplish the SAC task. The two control algorithms in Part I were tested in simulations against an optimal guidance algorithm. The results show that both algorithms yield a better performance in terms of time and miss distance. The results in Part II demonstrate the effectiveness of the hybrid architecture for the SAC task. The proposed frameworks and algorithms provide insights for the development of effective and more efficient control of pursuer vehicles and can be useful in the practical applications such as defence systems and civil law enforcement

Multi-Agent Control for Pursuer Coordination in Reach Avoid Games

This work addresses the fast-evader problem in pursuit-evasion games, where multi-pursuer coordination is leveraged to successfully trade-off kinematic superiority with numbers. The design of pursuer team strategies is developed under the framework of multi-agent control (also referred to as swarm control). The objective is the design of local level rules for a team of pursuers that results in the desired global behavior (evader capture). To that end, this work addresses three main issues regarding the design of scalable solutions for pursuer coordination against a fast evader: trading-off kinematic superiority with numbers through coordination, selecting the sufficient number of pursuers to guarantee capture, decentralized approach to satisfying a team objective while enforcing constraints. Through the construction of a surrogate objective for evader capture, the problem of pur- suer coordination is converted to a coverage control problem. The coverage problem treats the pursuer capture sets as resources to be distributed over a domain, which successfully enables the synthesis of swarm control solutions. Pursuer team size selection is achieved by decomposing the coverage problem into a static formation requirement and a tracking performance requirement for the individual agents. Lastly, a decentralized formulation of the coordinated capture problem and a framework for the enforcement of agent interaction constraints in aggressively maneuvering environments are introduced

Deep Reinforcement Learning for Swarm Systems

Recently, deep reinforcement learning (RL) methods have been applied successfully to multi-agent scenarios. Typically, the observation vector for decentralized decision making is represented by a concatenation of the (local) information an agent gathers about other agents. However, concatenation scales poorly to swarm systems with a large number of homogeneous agents as it does not exploit the fundamental properties inherent to these systems: (i) the agents in the swarm are interchangeable and (ii) the exact number of agents in the swarm is irrelevant. Therefore, we propose a new state representation for deep multi-agent RL based on mean embeddings of distributions, where we treat the agents as samples and use the empirical mean embedding as input for a decentralized policy. We define different feature spaces of the mean embedding using histograms, radial basis functions and neural networks trained end-to-end. We evaluate the representation on two well-known problems from the swarm literature in a globally and locally observable setup. For the local setup we furthermore introduce simple communication protocols. Of all approaches, the mean embedding representation using neural network features enables the richest information exchange between neighboring agents, facilitating the development of complex collective strategies