Coordinating decentralized learning and conflict resolution across agent boundaries

It is crucial for embedded systems to adapt to the dynamics of open environments. This adaptation process becomes especially challenging in the context of multiagent systems because of scalability, partial information accessibility and complex interaction of agents. It is a challenge for agents to learn good policies, when they need to plan and coordinate in uncertain, dynamic environments, especially when they have large state spaces. It is also critical for agents operating in a multiagent system (MAS) to resolve conflicts among the learned policies of different agents, since such conflicts may have detrimental influence on the overall performance. The focus of this research is to use a reinforcement learning based local optimization algorithm within each agent to learn multiagent policies in a decentralized fashion. These policies will allow each agent to adapt to changes in environmental conditions while reorganizing the underlying multiagent network when needed. The research takes an adaptive approach to resolving conflicts that can arise between locally optimal agent policies. First an algorithm that uses heuristic rules to locally resolve simple conflicts is presented. When the environment is more dynamic and uncertain, a mediator-based mechanism to resolve more complicated conflicts and selectively expand the agents' state space during the learning process is harnessed. For scenarios where mediator-based mechanisms with partially global views are ineffective, a more rigorous approach for global conflict resolution that synthesizes multiagent reinforcement learning (MARL) and distributed constraint optimization (DCOP) is developed. These mechanisms are evaluated in the context of a multiagent tornado tracking application called NetRads. Empirical results show that these mechanisms significantly improve the performance of the tornado tracking network for a variety of weather scenarios. The major contributions of this work are: a state of the art decentralized learning approach that supports agent interactions and reorganizes the underlying network when needed; the use of abstract classes of scenarios/states/actions that efficiently manages the exploration of the search space; novel conflict resolution algorithms of increasing complexity that use heuristic rules, sophisticated automated negotiation mechanisms and distributed constraint optimization methods respectively; and finally, a rigorous study of the interplay between two popular theories used to solve multiagent problems, namely decentralized Markov decision processes and distributed constraint optimization

Distributed Target Engagement in Large-scale Mobile Sensor Networks

Sensor networks comprise an emerging field of study that is expected to touch many aspects of our life. Research in this area was originally motivated by military applications. Afterward sensor networks have demonstrated tremendous promise in many other applications such as infrastructure security, environment and habitat monitoring, industrial sensing, traffic control, and surveillance applications. One key challenge in large-scale sensor networks is the efficient use of the network's resources to collect information about objects in a given Volume of Interest (VOI). Multi-sensor Multi-target tracking in surveillance applications is an example where the success of the network to track targets in a given volume of interest, efficiently and effectively, hinges significantly on the network's ability to allocate the right set of sensors to the right set of targets so as to achieve optimal performance. This task can be even more complicated if the surveillance application is such that the sensors and targets are expected to be mobile. To ensure timely tracking of targets in a given volume of interest, the surveillance sensor network needs to maintain engagement with all targets in this volume. Thus the network must be able to perform the following real-time tasks: 1) sensor-to-target allocation; 2) target tracking; 3) sensor mobility control and coordination. In this research I propose a combination of the Semi-Flocking algorithm, as a multi-target motion control and coordination approach, and a hierarchical Distributed Constraint Optimization Problem (DCOP) modelling algorithm, as an allocation approach, to tackle target engagement problem in large-scale mobile multi-target multi-sensor surveillance systems. Sensor-to-target allocation is an NP-hard problem. Thus, for sensor networks to succeed in such application, an efficient approach that can tackle this NP-hard problem in real-time is disparately needed. This research work proposes a novel approach to tackle this issue by modelling the problem as a Hierarchical DCOP. Although DCOPs has been proven to be both general and efficient they tend to be computationally expensive, and often intractable for large-scale problems. To address this challenge, this research proposes to divide the sensor-to-target allocation problem into smaller sub-DCOPs with shared constraints, eliminating significant computational and communication costs. Furthermore, a non-binary variable modelling is presented to reduce the number of inter-agent constraints. Target tracking and sensor mobility control and coordination are the other main challenges in these networks. Biologically inspired approaches have recently gained significant attention as a tool to address this issue. These approaches are exemplified by the two well-known algorithms, namely, the Flocking algorithm and the Anti-Flocking algorithm. Generally speaking, although these two biologically inspired algorithms have demonstrated promising performance, they expose deficiencies when it comes to their ability to maintain simultaneous reliable dynamic area coverage and target coverage. To address this challenge, Semi-Flocking, a biologically inspired algorithm that benefits from key characteristics of both the Flocking and Anti-Flocking algorithms, is proposed. The Semi-Flocking algorithm approaches the problem by assigning a small flock of sensors to each target, while at the same time leaving some sensors free to explore the environment. Also, this thesis presents an extension of the Semi-Flocking in which it is combined with a constrained clustering approach to provide better coverage over maneuverable targets. To have a reliable target tracking, another extension of Semi-Flocking algorithm is presented which is a coupled distributed estimation and motion control algorithm. In this extension the Semi-Flocking algorithm is employed for the purpose of a multi-target motion control, and Kalman-Consensus Filter (KCF) for the purpose of motion estimation. Finally, this research will show that the proposed Hierarchical DCOP algorithm can be elegantly combined with the Semi-Flocking algorithm and its extensions to create a coupled control and allocation approach. Several experimental analysis conducted in this research illustrate how the operation of the proposed algorithms outperforms other approaches in terms of incurred computational and communication costs, area coverage, target coverage for both linear and maneuverable targets, target detection time, number of undetected targets and target coverage in noise conditions sensor network. Also it is illustrated that this algorithmic combination can successfully engage multiple sensors to multiple mobile targets such that the number of uncovered targets is minimized and the sensors' mean utilization factor sensor surveillance systems.is maximized