CONTROL AND ESTIMATION ALGORITHMS FOR MULTIPLE-AGENT SYSTEMS

Tese arquivada ao abrigo da Portaria nº 227/2017 de 25 de julhoIn this thesis we study crucial problems within complex, large scale, networked control systems and mobile sensor networks. The ¯rst one is the problem of decomposition of a large-scale system into several interconnected subsystems, based on the imposed information structure constraints. After associating an intelligent agent with each subsystem, we face with a problem of formulating their local estimation and control laws and designing inter-agent communication strategies which ensure stability, desired performance, scalability and robustness of the overall system. Another problem addressed in this thesis, which is critical in mobile sensor networks paradigm, is the problem of searching positions for mobile nodes in order to achieve optimal overall sensing capabilities. Novel, overlapping decentralized state and parameter estimation schemes based on the consensus strategy have been proposed, in both continuous-time and discrete-time. The algorithms are proposed in the form of a multi-agent network based on a combination of local estimators and a dynamic consensus strategy, assuming possible intermittent observations and communication faults. Under general conditions concerning the agent resources and the network topology, conditions are derived for the stability and convergence of the algorithms. For the state estimation schemes, a strategy based on minimization of the steady-state mean-square estimation error is proposed for selection of the consensus gains; these gains can also be adjusted by local adaptation schemes. It is also demonstrated that there exists a connection between the network complexity and e±ciency of denoising, i.e., of suppression of the measurement noise in°uence. Several numerical examples serve to illustrate characteristic properties of the proposed algorithm and to demonstrate its applicability to real problems. Furthermore, several structures and algorithms for multi-agent control based on a dynamic consensus strategy have been proposed. Two novel classes of structured, overlapping decentralized control algorithms are presented. For the ¯rst class, an agreement between the agents is implemented at the level of control inputs, while the second class is based on the agreement at the state estimation level. The proposed control algorithms have been illustrated by several examples. Also, the second class of the proposed consensus based control scheme has been applied to decentralized overlapping tracking control of planar formations of UAVs. A comparison is given with the proposed novel design methodology based on the expansion/contraction paradigm and the inclusion principle. Motivated by the applications to the optimal mobile sensor positioning within mobile sensor networks, the perturbation-based extremum seeking algorithm has been modifed and extended. It has been assumed that the integrator gain and the perturbation amplitude are time varying (decreasing in time with a proper rate) and that the output is corrupted with measurement noise. The proposed basic, one dimensional, algorithm has been extended to two dimensional, hybrid schemes and directly applied to the planar optimal mobile sensor positioning, where the vehicles can be modeled as velocity actuated point masses, force actuated point masses, or nonholonomic unicycles. The convergence of all the proposed algorithms, with probability one and in the mean square sense, has been proved. Also, the problem of target assignment in multi-agent systems using multi-variable extremum seeking algorithm has been addressed. An algorithm which e®ectively solves the problem has been proposed, based on the local extremum seeking of the specially designed global utility functions which capture the dependance among di®erent, possibly con°icting objectives of the agents. It has been demonstrated how the utility function parameters and agents' initial conditions impact the trajectories and destinations of the agents. All the proposed extremum seeking based algorithms have been illustrated with several simulations

Information Theory and Cooperative Control in Networked Multi-Agent Systems with Applications to Smart Grid

This dissertation focuses on information theoretic aspects of and cooperative control techniques in networked multi-agent systems (NMAS) with communication constraints. In the first part of the dissertation, information theoretic limitations of tracking problems in networked control systems, especially leader-follower systems with communication constraints, are studied. Necessary conditions on the data rate of each communication link for tracking of the leader-follower systems are provided. By considering the forward and feedback channels as one cascade channel, we also provide a lower bound for the data rate of the cascade channel for the system to track a reference signal such that the tracking error has finite second moment. Finally, the aforementioned results are extended to the case in which the leader system and follower system have different system models. In the second part, we propose an easily scalable hierarchical decision-making and control architecture for smart grid with communication constraints in which distributed customers equipped with renewable distributed generation (RDG) interact and trade energy in the grid. We introduce the key components and their interactions in the proposed control architecture and discuss the design of distributed controllers which deal with short-term and long-term grid stability, power load balancing and energy routing. At microgrid level, under the assumption of user cooperation and inter-user communications, we propose a distributed networked control strategy to solve the demand-side management problem in microgrids. Moreover, by considering communication delays between users and microgrid central controller, we propose a distributed networked control strategy with prediction to solve the demand-side management problem with communication delays. In the third part, we consider the disturbance attenuation and stabilization problem in networked control systems. To be specific, we consider the string stability in a large group of interconnected systems over a communication network. Its potential applications could be found in formation tracking control in groups of robots, as well as uncertainty reduction and disturbance attenuation in smart grid. We propose a leader-following consensus protocol for such interconnected systems and derive the sufficient conditions, in terms of communication topology and control parameters, for string stability. Simulation results and performance in terms of disturbance propagation are also given. In the fourth part, we consider distributed tracking and consensus in networked multi-agent systems with noisy time-varying graphs and incomplete data. In particular, a distributed tracking with consensus algorithm is developed for the space-object tracking with a satellite surveillance network. We also intend to investigate the possible application of such methods in smart grid networks. Later, conditions for achieving distributed consensus are discussed and the rate of convergence is quantified for noisy time-varying graphs with incomplete data. We also provide detailed simulation results and performance comparison of the proposed distributed tracking with consensus algorithm in the case of space-object tracking problem and that of distributed local Kalman filtering with centralized fusion and centralized Kalman filter. The information theoretic limitations developed in the first part of this dissertation provide guildlines for design and analysis of tracking problems in networked control systems. The results reveal the mutual interaction and joint application of information theory and control theory in networked control systems. Second, the proposed architectures and approaches enable scalability in smart grid design and allow resource pooling among distributed energy resources (DER) so that the grid stability and optimality is maintained. The proposed distributed networked control strategy with prediction provides an approach for cooperative control at RDG-equipped customers within a self-contained microgrid with different feedback delays. Our string stability analysis in the third part of this dissertation allows a single networked control system to be extended to a large group of interconnected subsystems while system stability is still maintained. It also reveals the disturbance propagation through the network and the effect of disturbance in one subsystem on other subsystems. The proposed leader-following consensus protocol in the constrained communication among users reveals the effect of communication in stabilization of networked control systems and the interaction between communication and control over a network. Finally, the distributed tracking and consensus in networked multi-agent systems problem shows that information sharing among users improves the quality of local estimates and helps avoid conflicting and inefficient distributed decisions. It also reveals the effect of the graph topologies and incomplete node measurements on the speed of achieving distributed decision and final consensus accuracy

Deep Reinforcement Learning for Swarm Systems

Recently, deep reinforcement learning (RL) methods have been applied successfully to multi-agent scenarios. Typically, these methods rely on a concatenation of agent states to represent the information content required for decentralized decision making. However, concatenation scales poorly to swarm systems with a large number of homogeneous agents as it does not exploit the fundamental properties inherent to these systems: (i) the agents in the swarm are interchangeable and (ii) the exact number of agents in the swarm is irrelevant. Therefore, we propose a new state representation for deep multi-agent RL based on mean embeddings of distributions. We treat the agents as samples of a distribution and use the empirical mean embedding as input for a decentralized policy. We define different feature spaces of the mean embedding using histograms, radial basis functions and a neural network learned end-to-end. We evaluate the representation on two well known problems from the swarm literature (rendezvous and pursuit evasion), in a globally and locally observable setup. For the local setup we furthermore introduce simple communication protocols. Of all approaches, the mean embedding representation using neural network features enables the richest information exchange between neighboring agents facilitating the development of more complex collective strategies.Comment: 31 pages, 12 figures, version 3 (published in JMLR Volume 20

Differential Inequalities in Multi-Agent Coordination and Opinion Dynamics Modeling

Distributed algorithms of multi-agent coordination have attracted substantial attention from the research community; the simplest and most thoroughly studied of them are consensus protocols in the form of differential or difference equations over general time-varying weighted graphs. These graphs are usually characterized algebraically by their associated Laplacian matrices. Network algorithms with similar algebraic graph theoretic structures, called being of Laplacian-type in this paper, also arise in other related multi-agent control problems, such as aggregation and containment control, target surrounding, distributed optimization and modeling of opinion evolution in social groups. In spite of their similarities, each of such algorithms has often been studied using separate mathematical techniques. In this paper, a novel approach is offered, allowing a unified and elegant way to examine many Laplacian-type algorithms for multi-agent coordination. This approach is based on the analysis of some differential or difference inequalities that have to be satisfied by the some "outputs" of the agents (e.g. the distances to the desired set in aggregation problems). Although such inequalities may have many unbounded solutions, under natural graphic connectivity conditions all their bounded solutions converge (and even reach consensus), entailing the convergence of the corresponding distributed algorithms. In the theory of differential equations the absence of bounded non-convergent solutions is referred to as the equation's dichotomy. In this paper, we establish the dichotomy criteria of Laplacian-type differential and difference inequalities and show that these criteria enable one to extend a number of recent results, concerned with Laplacian-type algorithms for multi-agent coordination and modeling opinion formation in social groups.Comment: accepted to Automatic