Reinforcement Learning and Game Theory for Smart Grid Security

This dissertation focuses on one of the most critical and complicated challenges facing electric power transmission and distribution systems which is their vulnerability against failure and attacks. Large scale power outages in Australia (2016), Ukraine (2015), India (2013), Nigeria (2018), and the United States (2011, 2003) have demonstrated the vulnerability of power grids to cyber and physical attacks and failures. These incidents clearly indicate the necessity of extensive research efforts to protect the power system from external intrusion and to reduce the damages from post-attack effects. We analyze the vulnerability of smart power grids to cyber and physical attacks and failures, design different gametheoretic approaches to identify the critical components vulnerable to attack and propose their associated defense strategy, and utilizes machine learning techniques to solve the game-theoretic problems in adversarial and collaborative adversarial power grid environment. Our contributions can be divided into three major parts:Vulnerability identification: Power grid outages have disastrous impacts on almost every aspect of modern life. Despite their inevitability, the effects of failures on power grids’ performance can be limited if the system operator can predict and identify the vulnerable elements of power grids. To enable these capabilities we study machine learning algorithms to identify critical power system elements adopting a cascaded failure simulator as a threat and attack model. We use generation loss, time to reach a certain percentage of line outage/generation loss, number of line outages, etc. as evaluation metrics to evaluate the consequences of threat and attacks on the smart power grid.Adversarial gaming in power system: With the advancement of the technologies, the smart attackers are deploying different techniques to supersede the existing protection scheme. In order to defend the power grid from these smart attackers, we introduce an adversarial gaming environment using machine learning techniques which is capable of replicating the complex interaction between the attacker and the power system operators. The numerical results show that a learned defender successfully narrows down the attackers’ attack window and reduce damages. The results also show that considering some crucial factors, the players can independently execute actions without detailed information about each other.Deep learning for adversarial gaming: The learning and gaming techniques to identify vulnerable components in the power grid become computationally expensive for large scale power systems. The power system operator needs to have the advanced skills to deal with the large dimensionality of the problem. In order to aid the power system operator in finding and analyzing vulnerability for large scale power systems, we study a deep learning technique for adversary game which is capable of dealing with high dimensional power system state space with less computational time and increased computational efficiency. Overall, the results provided in this dissertation advance power grids’ resilience and security by providing a better understanding of the systems’ vulnerability and by developing efficient algorithms to identify vulnerable components and appropriate defensive strategies to reduce the damages of the attack

Thirty Years of Machine Learning: The Road to Pareto-Optimal Wireless Networks

Future wireless networks have a substantial potential in terms of supporting a broad range of complex compelling applications both in military and civilian fields, where the users are able to enjoy high-rate, low-latency, low-cost and reliable information services. Achieving this ambitious goal requires new radio techniques for adaptive learning and intelligent decision making because of the complex heterogeneous nature of the network structures and wireless services. Machine learning (ML) algorithms have great success in supporting big data analytics, efficient parameter estimation and interactive decision making. Hence, in this article, we review the thirty-year history of ML by elaborating on supervised learning, unsupervised learning, reinforcement learning and deep learning. Furthermore, we investigate their employment in the compelling applications of wireless networks, including heterogeneous networks (HetNets), cognitive radios (CR), Internet of things (IoT), machine to machine networks (M2M), and so on. This article aims for assisting the readers in clarifying the motivation and methodology of the various ML algorithms, so as to invoke them for hitherto unexplored services as well as scenarios of future wireless networks.Comment: 46 pages, 22 fig

Deep Learning Techniques for Power System Operation: Modeling and Implementation

The fast development of the deep learning (DL) techniques in the most recent years has drawn attention from both academia and industry. And there have been increasing applications of the DL techniques in many complex real-world situations, including computer vision, medical diagnosis, and natural language processing. The great power and flexibility of DL can be attributed to its hierarchical learning structure that automatically extract features from mass amounts of data. In addition, DL applies an end-to-end solving mechanism, and directly generates the output from the input, where the traditional machine learning methods usually break down the problem and combine the results. The end-to-end mechanism considerably improve the computational efficiency of the DL.The power system is one of the most complex artificial infrastructures, and many power system control and operation problems share the same features as the above mentioned real-world applications, such as time variability and uncertainty, partial observability, which impedes the performance of the conventional model-based methods. On the other hand, with the wide spread implementation of Advanced Metering Infrastructures (AMI), the SCADA, the Wide Area Monitoring Systems (WAMS), and many other measuring system providing massive data from the field, the data-driven deep learning technique is becoming an intriguing alternative method to enable the future development and success of the smart grid. This dissertation aims to explore the potential of utilizing the deep-learning-based approaches to solve a broad range of power system modeling and operation problems. First, a comprehensive literature review is conducted to summarize the existing applications of deep learning techniques in power system area. Second, the prospective application of deep learning techniques in several scenarios in power systems, including contingency screening, cascading outage search, multi-microgrid energy management, residential HVAC system control, and electricity market bidding are discussed in detail in the following 2-6 chapters. The problem formulation, the specific deep learning approaches in use, and the simulation results are all presented, and also compared with the currently used model-based method as a verification of the advantage of deep learning. Finally, the conclusions are provided in the last chapter, as well as the directions for future researches. It’s hoped that this dissertation can work as a single spark of fire to enlighten more innovative ideas and original studies, widening and deepening the application of deep learning technique in the field of power system, and eventually bring some positive impacts to the real-world bulk grid resilient and economic control and operation

On the Use of Reinforcement Learning for Attacking and Defending Load Frequency Control

The electric grid is an attractive target for cyberattackers given its critical nature in society. With the increasing sophistication of cyberattacks, effective grid defense will benefit from proactively identifying vulnerabilities and attack strategies. We develop a deep reinforcement learning-based method that recognizes vulnerabilities in load frequency control, an essential process that maintains grid security and reliability. We demonstrate how our method can synthesize a variety of attacks involving false data injection and load switching, while specifying the attack and threat models - providing insight into potential attack strategies and impact. We discuss how our approach can be employed for testing electric grid vulnerabilities. Moreover our method can be employed to generate data to inform the design of defense strategies and develop attack detection methods. For this, we design and compare a (deep learning-based) supervised attack detector with an unsupervised anomaly detector to highlight the benefits of developing defense strategies based on identified attack strategies

Agent-based simulation of electricity markets: a literature review

Liberalisation, climate policy and promotion of renewable energy are challenges to players of the electricity sector in many countries. Policy makers have to consider issues like market power, bounded rationality of players and the appearance of fluctuating energy sources in order to provide adequate legislation. Furthermore the interactions between markets and environmental policy instruments become an issue of increasing importance. A promising approach for the scientific analysis of these developments is the field of agent-based simulation. The goal of this article is to provide an overview of the current work applying this methodology to the analysis of electricity markets. --