Stochastic optimal adaptive controller and communication protocol design for networked control systems

Networked Control System (NCS) is a recent topic of research wherein the feedback control loops are closed through a real-time communication network. Many design challenges surface in such systems due to network imperfections such as random delays, packet losses, quantization effects and so on. Since existing control techniques are unsuitable for such systems, in this dissertation, a suite of novel stochastic optimal adaptive design methodologies is undertaken for both linear and nonlinear NCS in presence of uncertain system dynamics and unknown network imperfections such as network-induced delays and packet losses. The design is introduced in five papers. In Paper 1, a stochastic optimal adaptive control design is developed for unknown linear NCS with uncertain system dynamics and unknown network imperfections. A value function is adjusted forward-in-time and online, and a novel update law is proposed for tuning value function estimator parameters. Additionally, by using estimated value function, optimal adaptive control law is derived based on adaptive dynamic programming technique. Subsequently, this design methodology is extended to solve stochastic optimal strategies of linear NCS zero-sum games in Paper 2. Since most systems are inherently nonlinear, a novel stochastic optimal adaptive control scheme is then developed in Paper 3 for nonlinear NCS with unknown network imperfections. On the other hand, in Paper 4, the network protocol behavior (e.g. TCP and UDP) are considered and optimal adaptive control design is revisited using output feedback for linear NCS. Finally, Paper 5 explores a co-design framework where both the controller and network scheduling protocol designs are addressed jointly so that proposed scheme can be implemented into next generation Cyber Physical Systems --Abstract, page iv

Event-triggered robust control for multi-player nonzero-sum games with input constraints and mismatched uncertainties

In this article, an event-triggered robust control (ETRC) method is investigated for multi-player nonzero-sum games of continuous-time input constrained nonlinear systems with mismatched uncertainties. By constructing an auxiliary system and designing an appropriate value function, the robust control problem of input constrained nonlinear systems is transformed into an optimal regulation problem. Then, a critic neural network (NN) is adopted to approximate the value function of each player for solving the event-triggered coupled Hamilton-Jacobi equation and obtaining control laws. Based on a designed event-triggering condition, control laws are updated when events occur only. Thus, both computational burden and communication bandwidth are reduced. We prove that the weight approximation errors of critic NNs and the closed-loop uncertain multi-player system states are all uniformly ultimately bounded thanks to the Lyapunov's direct method. Finally, two examples are provided to demonstrate the effectiveness of the developed ETRC method

Event sampled optimal adaptive regulation of linear and a class of nonlinear systems

In networked control systems (NCS), wherein a communication network is used to close the feedback loop, the transmission of feedback signals and execution of the controller is currently carried out at periodic sampling instants. Thus, this scheme requires a significant computational power and network bandwidth. In contrast, the event-based aperiodic sampling and control, which is introduced recently, appears to relieve the computational burden and high network resource utilization. Therefore, in this dissertation, a suite of novel event sampled adaptive regulation schemes in both discrete and continuous time domain for uncertain linear and nonlinear systems are designed. Event sampled Q-learning and adaptive/neuro dynamic programming (ADP) schemes without value and policy iterations are utilized for the linear and nonlinear systems, respectively, in both the time domains. Neural networks (NN) are employed as approximators for nonlinear systems and, hence, the universal approximation property of NN in the event-sampled framework is introduced. The tuning of the parameters and the NN weights are carried out in an aperiodic manner at the event sampled instants leading to a further saving in computation when compared to traditional NN based control. The adaptive regulator when applied on a linear NCS with time-varying network delays and packet losses shows a 30% and 56% reduction in computation and network bandwidth usage, respectively. In case of nonlinear NCS with event sampled ADP based regulator, a reduction of 27% and 66% is observed when compared to periodic sampled schemes. The sampling and transmission instants are determined through adaptive event sampling conditions derived using Lyapunov technique by viewing the closed-loop event sampled linear and nonlinear systems as switched and/or impulsive dynamical systems. --Abstract, page iii

Control and game-theoretic methods for secure cyber-physical-human systems

This work focuses on systems comprising tightly interconnected physical and digital components. Those, aptly named, cyber-physical systems will be the core of the Fourth Industrial Revolution. Thus, cyber-physical systems will be called upon to interact with humans, either in a cooperative fashion, or as adversaries to malicious human agents that will seek to corrupt their operation. In this work, we will present methods that enable an autonomous system to operate safely among human agents and to gain an advantage in cyber-physical security scenarios by employing tools from control, game and learning theories. Our work revolves around three main axes: unpredictability-based defense, operation among agents with bounded rationality and verification of safety properties for autonomous systems. In taking advantage of the complex nature of cyber-physical systems, our unpredictability-based defense work will focus both on attacks on actuating and sensing components, which will be addressed via a novel switching-based Moving Target Defense framework, and on Denial-of-Service attacks on the underlying network via a zero-sum game exploiting redundant communication channels. Subsequently, we will take a more abstract view of complex system security by exploring the principles of bounded rationality. We will show how attackers of bounded rationality can coordinate in inducing erroneous decisions to a system while they remain stealthy. Methods of cognitive hierarchy will be employed for decision prediction, while closed form solutions of the optimization problem and the conditions of convergence to the Nash equilibrium will be investigated. The principles of bounded rationality will be brought to control systems via the use of policy iteration algorithms, enabling data-driven attack prediction in a more realistic fashion than what can be offered by game equilibrium solutions. The issue of intelligence in security scenarios will be further considered via concepts of learning manipulation through a proposed framework where bounded rationality is understood as a hierarchy in learning, rather than optimizing, capability. This viewpoint will allow us to propose methods of exploiting the learning process of an imperfect opponent in order to affect their cognitive state via the use of tools from optimal control theory. Finally, in the context of safety, we will explore verification and compositionality properties of linear systems that are designed to be added to a cascade network of similar systems. To obfuscate the need for knowledge of the system's dynamics, we will state decentralized conditions that guarantee a specific dissipativity properties for the system, which are shown to be solved by reinforcement learning techniques. Subsequently, we will propose a framework that employs a hierarchical solution of temporal logic specifications and reinforcement learning problems for optimal tracking.Ph.D

Advanced Controls Of Cyber Physical Energy Systems

Cyber system is a fairly important component of the energy systems. The network imperfections can significantly reduce the control performance if not be properly treated together with the physical system during the control designs. In the proposed research, the advanced controls of cyber-physical energy systems are explored in depth. The focus of our research is on two typical energy systems including the large-scale smart grid (e.g. wide-area power system) and the smart microgrid (e.g. shipboard power system and inverter-interfaced AC/DC microgrid). In order to proactively reduce the computation and communication burden of the wide-area power systems (WAPSs), an event/self-triggered control method is developed. Besides, a reinforcement learning method is designed to counteract the unavoidable network imperfections of WAPSs such as communication delay and packet dropout with unknown system dynamics. For smart microgrids, various advanced control techniques, e.g., output constrained control, consensus-based control, neuro network and game theory etc., have been successfully applied to improve their physical performance. The proposed control algorithms have been tested through extensive simulations including the real-time simulation, the power-hardware-in-the-loop simulation and on the hardware testbed. Based on the existing work, further research of microgrids will be conducted to develop the improved control algorithms with cyber uncertainties

Many-agent Reinforcement Learning

Multi-agent reinforcement learning (RL) solves the problem of how each agent should behave optimally in a stochastic environment in which multiple agents are learning simultaneously. It is an interdisciplinary domain with a long history that lies in the joint area of psychology, control theory, game theory, reinforcement learning, and deep learning. Following the remarkable success of the AlphaGO series in single-agent RL, 2019 was a booming year that witnessed significant advances in multi-agent RL techniques; impressive breakthroughs have been made on developing AIs that outperform humans on many challenging tasks, especially multi-player video games. Nonetheless, one of the key challenges of multi-agent RL techniques is the scalability; it is still non-trivial to design efficient learning algorithms that can solve tasks including far more than two agents ($N \gg 2$), which I name by \emph{many-agent reinforcement learning} (MARL\footnote{I use the world of ``MARL" to denote multi-agent reinforcement learning with a particular focus on the cases of many agents; otherwise, it is denoted as ``Multi-Agent RL" by default.}) problems. In this thesis, I contribute to tackling MARL problems from four aspects. Firstly, I offer a self-contained overview of multi-agent RL techniques from a game-theoretical perspective. This overview fills the research gap that most of the existing work either fails to cover the recent advances since 2010 or does not pay adequate attention to game theory, which I believe is the cornerstone to solving many-agent learning problems. Secondly, I develop a tractable policy evaluation algorithm -- $\alpha^\alpha$-Rank -- in many-agent systems. The critical advantage of $\alpha^\alpha$-Rank is that it can compute the solution concept of $\alpha$-Rank tractably in multi-player general-sum games with no need to store the entire pay-off matrix. This is in contrast to classic solution concepts such as Nash equilibrium which is known to be $PPAD$-hard in even two-player cases. $\alpha^\alpha$-Rank allows us, for the first time, to practically conduct large-scale multi-agent evaluations. Thirdly, I introduce a scalable policy learning algorithm -- mean-field MARL -- in many-agent systems. The mean-field MARL method takes advantage of the mean-field approximation from physics, and it is the first provably convergent algorithm that tries to break the curse of dimensionality for MARL tasks. With the proposed algorithm, I report the first result of solving the Ising model and multi-agent battle games through a MARL approach. Fourthly, I investigate the many-agent learning problem in open-ended meta-games (i.e., the game of a game in the policy space). Specifically, I focus on modelling the behavioural diversity in meta-games, and developing algorithms that guarantee to enlarge diversity during training. The proposed metric based on determinantal point processes serves as the first mathematically rigorous definition for diversity. Importantly, the diversity-aware learning algorithms beat the existing state-of-the-art game solvers in terms of exploitability by a large margin. On top of the algorithmic developments, I also contribute two real-world applications of MARL techniques. Specifically, I demonstrate the great potential of applying MARL to study the emergent population dynamics in nature, and model diverse and realistic interactions in autonomous driving. Both applications embody the prospect that MARL techniques could achieve huge impacts in the real physical world, outside of purely video games

Towards Autonomous Computer Networks in Support of Critical Systems

L'abstract è presente nell'allegato / the abstract is in the attachmen

On the role and opportunities in teamwork design for advanced multi-robot search systems

Intelligent robotic systems are becoming ever more present in our lives across a multitude of domains such as industry, transportation, agriculture, security, healthcare and even education. Such systems enable humans to focus on the interesting and sophisticated tasks while robots accomplish tasks that are either too tedious, routine or potentially dangerous for humans to do. Recent advances in perception technologies and accompanying hardware, mainly attributed to rapid advancements in the deep-learning ecosystem, enable the deployment of robotic systems equipped with onboard sensors as well as the computational power to perform autonomous reasoning and decision making online. While there has been significant progress in expanding the capabilities of single and multi-robot systems during the last decades across a multitude of domains and applications, there are still many promising areas for research that can advance the state of cooperative searching systems that employ multiple robots. In this article, several prospective avenues of research in teamwork cooperation with considerable potential for advancement of multi-robot search systems will be visited and discussed. In previous works we have shown that multi-agent search tasks can greatly benefit from intelligent cooperation between team members and can achieve performance close to the theoretical optimum. The techniques applied can be used in a variety of domains including planning against adversarial opponents, control of forest fires and coordinating search-and-rescue missions. The state-of-the-art on methods of multi-robot search across several selected domains of application is explained, highlighting the pros and cons of each method, providing an up-to-date view on the current state of the domains and their future challenges