Event-triggered robust control for multi-player nonzero-sum games with input constraints and mismatched uncertainties

In this article, an event-triggered robust control (ETRC) method is investigated for multi-player nonzero-sum games of continuous-time input constrained nonlinear systems with mismatched uncertainties. By constructing an auxiliary system and designing an appropriate value function, the robust control problem of input constrained nonlinear systems is transformed into an optimal regulation problem. Then, a critic neural network (NN) is adopted to approximate the value function of each player for solving the event-triggered coupled Hamilton-Jacobi equation and obtaining control laws. Based on a designed event-triggering condition, control laws are updated when events occur only. Thus, both computational burden and communication bandwidth are reduced. We prove that the weight approximation errors of critic NNs and the closed-loop uncertain multi-player system states are all uniformly ultimately bounded thanks to the Lyapunov's direct method. Finally, two examples are provided to demonstrate the effectiveness of the developed ETRC method

Multi-H∞ controls for unknown input-interference nonlinear system with reinforcement learning

This article studies the multi-H∞ controls for the input-interference nonlinear systems via adaptive dynamic programming (ADP) method, which allows for multiple inputs to have the individual selfish component of the strategy to resist weighted interference. In this line, the ADP scheme is used to learn the Nash-optimization solutions of the input-interference nonlinear system such that multiple H∞ performance indices can reach the defined Nash equilibrium. First, the input-interference nonlinear system is given and the Nash equilibrium is defined. An adaptive neural network (NN) observer is introduced to identify the input-interference nonlinear dynamics. Then, the critic NNs are used to learn the multiple H∞ performance indices. A novel adaptive law is designed to update the critic NN weights by minimizing the Hamiltonian-Jacobi-Isaacs (HJI) equation, which can be used to directly calculate the multi-H∞ controls effectively by using input-output data such that the actor structure is avoided. Moreover, the control system stability and updated parameter convergence are proved. Finally, two numerical examples are simulated to verify the proposed ADP scheme for the input-interference nonlinear system

Advanced Controls Of Cyber Physical Energy Systems

Cyber system is a fairly important component of the energy systems. The network imperfections can significantly reduce the control performance if not be properly treated together with the physical system during the control designs. In the proposed research, the advanced controls of cyber-physical energy systems are explored in depth. The focus of our research is on two typical energy systems including the large-scale smart grid (e.g. wide-area power system) and the smart microgrid (e.g. shipboard power system and inverter-interfaced AC/DC microgrid). In order to proactively reduce the computation and communication burden of the wide-area power systems (WAPSs), an event/self-triggered control method is developed. Besides, a reinforcement learning method is designed to counteract the unavoidable network imperfections of WAPSs such as communication delay and packet dropout with unknown system dynamics. For smart microgrids, various advanced control techniques, e.g., output constrained control, consensus-based control, neuro network and game theory etc., have been successfully applied to improve their physical performance. The proposed control algorithms have been tested through extensive simulations including the real-time simulation, the power-hardware-in-the-loop simulation and on the hardware testbed. Based on the existing work, further research of microgrids will be conducted to develop the improved control algorithms with cyber uncertainties

Reinforcement Learning and Game Theory for Smart Grid Security

This dissertation focuses on one of the most critical and complicated challenges facing electric power transmission and distribution systems which is their vulnerability against failure and attacks. Large scale power outages in Australia (2016), Ukraine (2015), India (2013), Nigeria (2018), and the United States (2011, 2003) have demonstrated the vulnerability of power grids to cyber and physical attacks and failures. These incidents clearly indicate the necessity of extensive research efforts to protect the power system from external intrusion and to reduce the damages from post-attack effects. We analyze the vulnerability of smart power grids to cyber and physical attacks and failures, design different gametheoretic approaches to identify the critical components vulnerable to attack and propose their associated defense strategy, and utilizes machine learning techniques to solve the game-theoretic problems in adversarial and collaborative adversarial power grid environment. Our contributions can be divided into three major parts:Vulnerability identification: Power grid outages have disastrous impacts on almost every aspect of modern life. Despite their inevitability, the effects of failures on power grids’ performance can be limited if the system operator can predict and identify the vulnerable elements of power grids. To enable these capabilities we study machine learning algorithms to identify critical power system elements adopting a cascaded failure simulator as a threat and attack model. We use generation loss, time to reach a certain percentage of line outage/generation loss, number of line outages, etc. as evaluation metrics to evaluate the consequences of threat and attacks on the smart power grid.Adversarial gaming in power system: With the advancement of the technologies, the smart attackers are deploying different techniques to supersede the existing protection scheme. In order to defend the power grid from these smart attackers, we introduce an adversarial gaming environment using machine learning techniques which is capable of replicating the complex interaction between the attacker and the power system operators. The numerical results show that a learned defender successfully narrows down the attackers’ attack window and reduce damages. The results also show that considering some crucial factors, the players can independently execute actions without detailed information about each other.Deep learning for adversarial gaming: The learning and gaming techniques to identify vulnerable components in the power grid become computationally expensive for large scale power systems. The power system operator needs to have the advanced skills to deal with the large dimensionality of the problem. In order to aid the power system operator in finding and analyzing vulnerability for large scale power systems, we study a deep learning technique for adversary game which is capable of dealing with high dimensional power system state space with less computational time and increased computational efficiency. Overall, the results provided in this dissertation advance power grids’ resilience and security by providing a better understanding of the systems’ vulnerability and by developing efficient algorithms to identify vulnerable components and appropriate defensive strategies to reduce the damages of the attack

Game-Theoretic Frameworks and Strategies for Defense Against Network Jamming and Collocation Attacks

Modern networks are becoming increasingly more complex, heterogeneous, and densely connected. While more diverse services are enabled to an ever-increasing number of users through ubiquitous networking and pervasive computing, several important challenges have emerged. For example, densely connected networks are prone to higher levels of interference, which makes them more vulnerable to jamming attacks. Also, the utilization of software-based protocols to perform routing, load balancing and power management functions in Software-Defined Networks gives rise to more vulnerabilities that could be exploited by malicious users and adversaries. Moreover, the increased reliance on cloud computing services due to a growing demand for communication and computation resources poses formidable security challenges due to the shared nature and virtualization of cloud computing. In this thesis, we study two types of attacks: jamming attacks on wireless networks and side-channel attacks on cloud computing servers. The former attacks disrupt the natural network operation by exploiting the static topology and dynamic channel assignment in wireless networks, while the latter attacks seek to gain access to unauthorized data by co-residing with target virtual machines (VMs) on the same physical node in a cloud server. In both attacks, the adversary faces a static attack surface and achieves her illegitimate goal by exploiting a stationary aspect of the network functionality. Hence, this dissertation proposes and develops counter approaches to both attacks using moving target defense strategies. We study the strategic interactions between the adversary and the network administrator within a game-theoretic framework. First, in the context of jamming attacks, we present and analyze a game-theoretic formulation between the adversary and the network defender. In this problem, the attack surface is the network connectivity (the static topology) as the adversary jams a subset of nodes to increase the level of interference in the network. On the other side, the defender makes judicious adjustments of the transmission footprint of the various nodes, thereby continuously adapting the underlying network topology to reduce the impact of the attack. The defender\u27s strategy is based on playing Nash equilibrium strategies securing a worst-case network utility. Moreover, scalable decomposition-based approaches are developed yielding a scalable defense strategy whose performance closely approaches that of the non-decomposed game for large-scale and dense networks. We study a class of games considering discrete as well as continuous power levels. In the second problem, we consider multi-tenant clouds, where a number of VMs are typically collocated on the same physical machine to optimize performance and power consumption and maximize profit. This increases the risk of a malicious virtual machine performing side-channel attacks and leaking sensitive information from neighboring VMs. The attack surface, in this case, is the static residency of VMs on a set of physical nodes, hence we develop a timed migration defense approach. Specifically, we analyze a timing game in which the cloud provider decides when to migrate a VM to a different physical machine to mitigate the risk of being compromised by a collocated malicious VM. The adversary decides the rate at which she launches new VMs to collocate with the victim VMs. Our formulation captures a data leakage model in which the cost incurred by the cloud provider depends on the duration of collocation with malicious VMs. It also captures costs incurred by the adversary in launching new VMs and by the defender in migrating VMs. We establish sufficient conditions for the existence of Nash equilibria for general cost functions, as well as for specific instantiations, and characterize the best response for both players. Furthermore, we extend our model to characterize its impact on the attacker\u27s payoff when the cloud utilizes intrusion detection systems that detect side-channel attacks. Our theoretical findings are corroborated with extensive numerical results in various settings as well as a proof-of-concept implementation in a realistic cloud setting

CONTROLLER SYNTHESIS UNDER INFORMATION AND FINITE-TIME LOGICAL CONSTRAINTS

In robotics, networks, and many related fields, a typical controller design problem needs to address both logical and informational constraints. The logical constraints may arise due to the complex task description or decision making process, while the information-related constraints emerge naturally as a consequence of the limitations on communication and computation capabilities. In the first part of the thesis, we consider the problem of synthesizing an event-based controller to address the information-related constraints in the controller design. We consider dynamical systems that are operating under continuous state feedback. This assumes that the measurements are continuously transmitted to the controller in order to generate the input and thus, increases the cost of communication by requiring huge communication resources. In many situations, it so happens that the measurement does not change fast enough that continuous transmission is required. Taking motivation from this, we consider the case where instead continuous feedback we seek an intermittent-feedback. As a result, the system trajectory will deviate from its ideal behavior. However, the question is how much would it deviate? Given the allowed bound on this deviation, can we design some controller that requires fewer measurements than the original controller and still manages to keep the deviation within this prescribed bound? Two important questions remain: 1) What will be the structure of the (optimal) controller? 2) How will the system know the (optimal) instances to transmit the measurement? When the system sends out measurement to controller, it is called as an ``event". Thus, we are looking for an event-generator and a controller to perform event-based control under the constraints on the availability of the state information. The next part focuses on controller synthesis problems that have logical, spatio-temporal constraints on the trajectory of the system; a robot motion planning problem fits as a good example of these kind of finite-time logically constrained problems. We adopt an automata-based approach to abstract the motion of the robot into an automata, and verify the satisfaction of the logical constraints on this automata. The abstraction of the dynamics of the robot into an automata is based on certain reachability guarantee of the robot's dynamics. The controller synthesis problem over the abstracted automata can be represented as a shortest-path-problem. In part III, we consider the problem of jointly addressing the logical and information constraints. The problem is approached with the notion of robustness of logical constraints. We propose two different frameworks for this problem with two different notions of robustness and two different approaches for the controller synthesis. One framework relies on the abstraction of the dynamical systems into a finite transition system, whereas the other relies on tools and results from prescribed performance control to design continuous feedback control to satisfy the robust logical constraints. We adopt an hierarchical controller synthesis method where a continuous feedback controller is designed to satisfy the (robust) logical constraints, and later, that controller is replaced by a suitable event-triggered intermittent feedback controller to cope with informational constraints

ISAACS: Iterative Soft Adversarial Actor-Critic for Safety

The deployment of robots in uncontrolled environments requires them to operate robustly under previously unseen scenarios, like irregular terrain and wind conditions. Unfortunately, while rigorous safety frameworks from robust optimal control theory scale poorly to high-dimensional nonlinear dynamics, control policies computed by more tractable "deep" methods lack guarantees and tend to exhibit little robustness to uncertain operating conditions. This work introduces a novel approach enabling scalable synthesis of robust safety-preserving controllers for robotic systems with general nonlinear dynamics subject to bounded modeling error by combining game-theoretic safety analysis with adversarial reinforcement learning in simulation. Following a soft actor-critic scheme, a safety-seeking fallback policy is co-trained with an adversarial "disturbance" agent that aims to invoke the worst-case realization of model error and training-to-deployment discrepancy allowed by the designer's uncertainty. While the learned control policy does not intrinsically guarantee safety, it is used to construct a real-time safety filter (or shield) with robust safety guarantees based on forward reachability rollouts. This shield can be used in conjunction with a safety-agnostic control policy, precluding any task-driven actions that could result in loss of safety. We evaluate our learning-based safety approach in a 5D race car simulator, compare the learned safety policy to the numerically obtained optimal solution, and empirically validate the robust safety guarantee of our proposed safety shield against worst-case model discrepancy.Comment: Accepted in 5th Annual Learning for Dynamics & Control Conference (L4DC), University of Pennsylvani