Building collaboration in multi-agent systems using reinforcement learning

© Springer Nature Switzerland AG 2018. This paper presents a proof-of concept study for demonstrating the viability of building collaboration among multiple agents through standard Q learning algorithm embedded in particle swarm optimisation. Collaboration is formulated to be achieved among the agents via competition, where the agents are expected to balance their action in such a way that none of them drifts away of the team and none intervene any fellow neighbours territory, either. Particles are devised with Q learning for self training to learn how to act as members of a swarm and how to produce collaborative/collective behaviours. The produced experimental results are supportive to the proposed idea suggesting that a substantive collaboration can be build via proposed learning algorithm

Consensus disturbance rejection for Lipschitz nonlinear multi-agent systems with input delay: a DOBC approach

In this paper, a new predictor-based consensus disturbance rejection method is proposed for high-order multi agent systems with Lipschitz nonlinearity and input delay. First, a distributed disturbance observer for consensus control is developed for each agent to estimate the disturbance under the delay constraint. Based on the conventional predictor feedback approach, a non-ideal predictor based control scheme is constructed for each agent by utilizing the estimate of the disturbance and the prediction of the relative state information. Then, rigorous analysis is carried out to ensure that the extra terms associated with disturbances and nonlinear functions are properly considered. Sufficient conditions for the consensus of the multi-agent systems with disturbance rejection are derived based on the analysis in the framework of Lyapunov-Krasovskii functionals. A simulation example is included to demonstrate the performance of the proposed control scheme. (C) 2016 The Franklin Institute. Published by Elsevier Ltd. All rights reserved.National Natural Science Foundation of China [61673034]SCI(E)ARTICLE1,SI298-31535

Formation Control Algorithms for Multi-UAV Systems with Unstable Topologies and Hybrid Delays

Multi-UAV systems rely on the communication network to exchange mission-critical data for their coordination and deployment, while communication delays could cause significant challenges to both tasks. The impact of the delays becomes even more severe if the delay, network structure and formation are all time-varying, a common challenge faced by real-world multiUAV systems. To address this challenge, we consider time-varying delays that exist in multiple channels caused by transmitting information and internal delays that exist in UAVs themselves caused by obtaining and processing their own data. We design an effective distributed formation control protocol for a multiUAV system to achieve time-varying formation; this protocol is particularly useful for dealing with time-varying multi-UAV network topologies as well. We provide rigorous convergence analysis for different scenarios with or without hybrid delays and obtain sufficient conditions for achieving the time-varying formation. Furthermore, we propose an algorithm for quantifying the maximum delay allowed by the system. Based on the designed formation algorithm, a deployment strategy is proposed to coordinate multi-UAV systems in a practical environment. Numerical analysis and UAV hardware experiments are conducted to evaluate the performance of the theoretical results and investigate the feasibility of generated flight trajectories

Design and implementation of predictive control for networked multi-process systems

This thesis is concerned with the design and application of the prediction method in the NMAS (networked multi-agent system) external consensus problem. The prediction method has been popular in networked single agent systems due to its capability of actively compensating for network-related constraints. This characteristic has motivated researchers to apply the prediction method to closed-loop multi-process controls over network systems. This thesis conducts an in-depth analysis of the suitability of the prediction method for the control of NMAS. In the external consensus problem, NMAS agents must achieve a common output (e.g. water level) that corresponds to the designed consensus protocol. The output is determined by the external reference input, which is provided to only one agent in the NMAS. This agreement is achieved through data exchanges between agents over network communications. In the presence of a network, the existence of network delay and data loss is inevitable. The main challenge in this thesis is thus to design an external consensus protocol with an efficient capability for network constraints compensation. The main contribution of this thesis is the enhancement of the prediction algorithm’s capability in NMAS applications. The external consensus protocol is presented for heterogeneous NMAS with four types of network constraints by utilising the developed prediction algorithm. The considered network constraints are constant network delay, asymmetric constant network delay, bounded random network delay, and large consecutive data losses. In the first case, this thesis presents the designed algorithm, which is able to compensate for uniform constant network delay in linear heterogeneous NMAS. The result is accompanied by stability criteria of the whole NMAS, an optimal coupling gains selection analysis, and empirical data from the experimental results. ‘Uniform network delay’ in this context refers to a situation in which the agent experiences a delay in accessing its own information, which is identical to the delay in data transfer from its neighbouring agent(s) in the network In the second case, this thesis presents an extension of the designed algorithm in the previous chapter, with the enhanced capability of compensating for asymmetric constant network delay in the NMAS. In contrast with the first case—which required the same prediction length as each neighbouring agent, subject to the same values of constant network delay—this case imposed varied constant network delays between agents, which required multi-prediction lengths for each agent. Thus, to simplify the computation, we selected a single prediction length for all agents and determined the possible maximum value of the constant network delay that existed in the NMAS. We tested the designed control algorithm on three heterogeneous pilotscale test rig setups. In the third case, we present a further enhancement of the designed control algorithm, which includes the capability of compensating for bounded random network delay in the NMAS. We achieve this by adding delay measurement signal generator within each agent control system. In this work, the network delay is considered to be half of the measured total delay in the network loop, which can be measured using a ramp signal. This method assumes that the duration for each agent to receive data from its neighbouring agent is equal to the time for the agent’s own transmitted data to be received by its neighbouring agent(s). In the final case, we propose a novel strategy for combining the predictive control with a new gain error ratio (GER) formula. This strategy is not only capable of compensating for a large number of consecutive data losses (CDLs) in the external consensus problem; it can also compensate for network constraints without affecting the consensus convergence time of the whole system. Thus, this strategy is not only able to solve the external consensus problem but is also robust to the number of CDL occurrences in NMAS. In each case, the designed control algorithm is compared with a Proportional-Integral (PI) controller. The evaluation of the NMAS output performance is conducted for each by simulations, analytical calculations, and practical experiments. In this thesis, the research work is accomplished through the integration of basic blocks and a bespoke Networked Control toolbox in MATLAB Simulink, together with NetController hardware

Event-Triggered Consensus and Formation Control in Multi-Agent Coordination

The focus of this thesis is to study distributed event-triggered control for multi-agent systems (MASs) facing constraints in practical applications. We consider several problems in the field, ranging from event-triggered consensus with information quantization, event-triggered edge agreement under synchronized/unsynchronized clocks, event-triggered leader-follower consensus with Euler-Lagrange agent dynamics and cooperative event-triggered rigid formation control. The first topic is named as event-triggered consensus with quantized relative state measurements. In this topic, we develop two event-triggered controllers with quantized relative state measurements to achieve consensus for an undirected network where each agent is modelled by single integrator dynamics. Both uniform and logarithmic quantizers are considered, which, together with two different controllers, yield four cases of study in this topic. The quantized information is used to update the control input as well as to determine the next trigger event. We show that approximate consensus can be achieved by the proposed algorithms and Zeno behaviour can be completely excluded if constant offsets with some computable lower bounds are added to the trigger conditions. The second topic considers event-triggered edge agreement problems. Two cases, namely the synchronized clock case and the unsynchronized clock case, are studied. In the synchronized clock case, all agents are activated simultaneously to measure the relative state information over edge links under a global clock. Edge events are defined and their occurrences trigger the update of control inputs for the two agents sharing the link. We show that average consensus can be achieved with our proposed algorithm. In the unsynchronized clock case, each agent executes control algorithms under its own clock which is not synchronized with other agents' clocks. An edge event only triggers control input update for an individual agent. It is shown that all agents will reach consensus in a totally asynchronous manner. In the third topic, we propose three different distributed event-triggered control algorithms to achieve leader-follower consensus for a network of Euler-Lagrange agents. We firstly propose two model-independent algorithms for a subclass of Euler-Lagrange agents without the vector of gravitational potential forces. A variable-gain algorithm is employed when the sensing graph is undirected; algorithm parameters are selected in a fully distributed manner with much greater flexibility compared to all previous work concerning event-triggered consensus problems. When the sensing graph is directed, a constant-gain algorithm is employed. The control gains must be centrally designed to exceed several lower bounding inequalities which require limited knowledge of bounds on the matrices describing the agent dynamics, bounds on network topology information and bounds on the initial conditions. When the Euler-Lagrange agents have dynamics which include the vector of gravitational potential forces, an adaptive algorithm is proposed. This requires more information about the agent dynamics but allows for the estimation of uncertain agent parameters. The last topic discusses cooperative stabilization control of rigid formations via an event-triggered approach. We first design a centralized event-triggered formation control system, in which a central event controller determines the next triggering time and broadcasts the event signal to all the agents for control input update. We then build on this approach to propose a distributed event control strategy, in which each agent can use its local event trigger and local information to update the control input at its own event time. For both cases, the trigger condition, event function and trigger behaviour are discussed in detail, and the exponential convergence of the formation system is guaranteed