Deep Reinforcement Learning for Wireless Sensor Scheduling in Cyber-Physical Systems

In many Cyber-Physical Systems, we encounter the problem of remote state estimation of geographically distributed and remote physical processes. This paper studies the scheduling of sensor transmissions to estimate the states of multiple remote, dynamic processes. Information from the different sensors have to be transmitted to a central gateway over a wireless network for monitoring purposes, where typically fewer wireless channels are available than there are processes to be monitored. For effective estimation at the gateway, the sensors need to be scheduled appropriately, i.e., at each time instant one needs to decide which sensors have network access and which ones do not. To address this scheduling problem, we formulate an associated Markov decision process (MDP). This MDP is then solved using a Deep Q-Network, a recent deep reinforcement learning algorithm that is at once scalable and model-free. We compare our scheduling algorithm to popular scheduling algorithms such as round-robin and reduced-waiting-time, among others. Our algorithm is shown to significantly outperform these algorithms for many example scenarios

Learning Scheduling Algorithms for Data Processing Clusters

Efficiently scheduling data processing jobs on distributed compute clusters requires complex algorithms. Current systems, however, use simple generalized heuristics and ignore workload characteristics, since developing and tuning a scheduling policy for each workload is infeasible. In this paper, we show that modern machine learning techniques can generate highly-efficient policies automatically. Decima uses reinforcement learning (RL) and neural networks to learn workload-specific scheduling algorithms without any human instruction beyond a high-level objective such as minimizing average job completion time. Off-the-shelf RL techniques, however, cannot handle the complexity and scale of the scheduling problem. To build Decima, we had to develop new representations for jobs' dependency graphs, design scalable RL models, and invent RL training methods for dealing with continuous stochastic job arrivals. Our prototype integration with Spark on a 25-node cluster shows that Decima improves the average job completion time over hand-tuned scheduling heuristics by at least 21%, achieving up to 2x improvement during periods of high cluster load

Resilience-driven planning and operation of networked microgrids featuring decentralisation and flexibility

High-impact and low-probability extreme events including both man-made events and natural weather events can cause severe damage to power systems. These events are typically rare but featured in long duration and large scale. Many research efforts have been conducted on the resilience enhancement of modern power systems. In recent years, microgrids (MGs) with distributed energy resources (DERs) including both conventional generation resources and renewable energy sources provide a viable solution for the resilience enhancement of such multi-energy systems during extreme events. More specifically, several islanded MGs after extreme events can be connected with each other as a cluster, which has the advantage of significantly reducing load shedding through energy sharing among them. On the other hand, mobile power sources (MPSs) such as mobile energy storage systems (MESSs), electric vehicles (EVs), and mobile emergency generators (MEGs) have been gradually deployed in current energy systems for resilience enhancement due to their significant advantages on mobility and flexibility. Given such a context, a literature review on resilience-driven planning and operation problems featuring MGs is presented in detail, while research limitations are summarised briefly. Then, this thesis investigates how to develop appropriate planning and operation models for the resilience enhancement of networked MGs via different types of DERs (e.g., MGs, ESSs, EVs, MESSs, etc.). This research is conducted in the following application scenarios: 1. This thesis proposes novel operation strategies for hybrid AC/DC MGs and networked MGs towards resilience enhancement. Three modelling approaches including centralised control, hierarchical control, and distributed control have been applied to formulate the proposed operation problems. A detailed non-linear AC OPF algorithm is employed to model each MG capturing all the network and technical constraints relating to stability properties (e.g., voltage limits, active and reactive power flow limits, and power losses), while uncertainties associated with renewable energy sources and load profiles are incorporated into the proposed models via stochastic programming. Impacts of limited generation resources, load distinction intro critical and non-critical, and severe contingencies (e.g., multiple line outages) are appropriately captured to mimic a realistic scenario. 2. This thesis introduces MPSs (e.g., EVs and MESSs) into the suggested networked MGs against the severe contingencies caused by extreme events. Specifically, time-coupled routing and scheduling characteristics of MPSs inside each MG are modelled to reduce load shedding when large damage is caused to each MG during extreme events. Both transportation networks and power networks are considered in the proposed models, while transporting time of MPSs between different transportation nodes is also appropriately captured. 3. This thesis focuses on developing realistic planning models for the optimal sizing problem of networked MGs capturing a trade-off between resilience and cost, while both internal uncertainties and external contingencies are considered in the suggested three-level planning model. Additionally, a resilience-driven planning model is developed to solve the coupled optimal sizing and pre-positioning problem of MESSs in the context of decentralised networked MGs. Internal uncertainties are captured in the model via stochastic programming, while external contingencies are included through the three-level structure. 4. This thesis investigates the application of artificial intelligence techniques to power system operations. Specifically, a model-free multi-agent reinforcement learning (MARL) approach is proposed for the coordinated routing and scheduling problem of multiple MESSs towards resilience enhancement. The parameterized double deep Q-network method (P-DDQN) is employed to capture a hybrid policy including both discrete and continuous actions. A coupled power-transportation network featuring a linearised AC OPF algorithm is realised as the environment, while uncertainties associated with renewable energy sources, load profiles, line outages, and traffic volumes are incorporated into the proposed data-driven approach through the learning procedure.Open Acces