Definition and evaluation of model-free coordination of electrical vehicle charging with reinforcement learning

Demand response (DR) becomes critical to manage the charging load of a growing electric vehicle (EV) deployment. Initial DR studies mainly adopt model predictive control, but models are largely uncertain for the EV scenario (e.g., customer behavior). Model-free approaches, based on reinforcement learning (RL), are an attractive alternative. We propose a new Markov decision process (MDP) formulation in the RL framework, to jointly coordinate a set of charging stations. State-of-the-art algorithms either focus on a single EV, or control an aggregate of EVs in multiple steps (e.g., 1) make aggregate load decisions and 2) translate the aggregate decision to individual EVs). In contrast, our RL approach jointly controls the whole set of EVs at once. We contribute a new MDP formulation with a scalable state representation independent of the number of charging stations. Using a batch RL algorithm, fitted $Q$ -iteration, we learn an optimal charging policy. With simulations using real-world data, we: 1) differentiate settings in training the RL policy (e.g., the time span covered by training data); 2) compare its performance to an oracle all-knowing benchmark (providing an upper performance bound); 3) analyze performance fluctuations throughout a full year; and 4) demonstrate generalization capacity to larger sets of charging stations

Online Battery Protective Energy Management for Energy-Transportation Nexus

Grid-connected electric vehicles (GEVs) and energy-transportation nexus bring a bright prospect to improve the penetration of renewable energy and the economy of microgrids (MGs). However, it is challenging to determine optimal vehicle-to-grid (V2G) strategies due to the complex battery aging mechanism and volatile MG states. This article develops a novel online battery anti-aging energy management method for energy-transportation nexus by using a novel deep reinforcement learning (DRL) framework. Based on battery aging characteristic analysis and rain-flow cycle counting technology, the quantification of aging cost in V2G strategies is realized by modeling the impact of number of cycles, depth of discharge, and charge and discharge rate. The established life loss model is used to evaluate battery anti-aging effectiveness of agent actions. The coordination of GEVs charging is modeled as multiobjective learning by using a DRL algorithm. The training objective is to maximize renewable penetration while reducing MG power fluctuations and vehicle battery aging costs. The developed energy-transportation nexus energy management method is verified to be effective in optimal power balancing and battery anti-aging control on a MG in the U.K. This article provides an efficient and economical tool for MG power balancing by optimally coordinating GEVs charging and renewable energy, thus helping promote a low-cost decarbonization transition.</p

A Robust and Constrained Multi-Agent Reinforcement Learning Framework for Electric Vehicle AMoD Systems

Electric vehicles (EVs) play critical roles in autonomous mobility-on-demand (AMoD) systems, but their unique charging patterns increase the model uncertainties in AMoD systems (e.g. state transition probability). Since there usually exists a mismatch between the training and test (true) environments, incorporating model uncertainty into system design is of critical importance in real-world applications. However, model uncertainties have not been considered explicitly in EV AMoD system rebalancing by existing literature yet and remain an urgent and challenging task. In this work, we design a robust and constrained multi-agent reinforcement learning (MARL) framework with transition kernel uncertainty for the EV rebalancing and charging problem. We then propose a robust and constrained MARL algorithm (ROCOMA) that trains a robust EV rebalancing policy to balance the supply-demand ratio and the charging utilization rate across the whole city under state transition uncertainty. Experiments show that the ROCOMA can learn an effective and robust rebalancing policy. It outperforms non-robust MARL methods when there are model uncertainties. It increases the system fairness by 19.6% and decreases the rebalancing costs by 75.8%.Comment: 8 page

Reduced state space and cost function in reinforcement learning for demand response control of multiple EV charging stations

Electric vehicle (EV) charging stations represent a substantial load with significant flexibility. Balancing such load with model-free demand response (DR) based on reinforcement learning (RL) is an attractive approach. We build on previous RL research using a Markov decision process (MDP) to simultaneously coordinate multiple charging stations. The previously proposed approach is computationally expensive in terms of large training times, limiting its feasibility and practicality. We propose to a priori force the control policy to always fulfill any charging demand that does not offer any flexibility at a given point, and thus use an updated cost function. We compare the policy of the newly proposed approach with the original (costly) one, for the case of load flattening, in terms of (i) processing time to learn the RL-based charging policy, and (ii) overall performance of the policy decisions in terms of meeting the target load for unseen test data

Grid flexibility by electrifying energy systems for sustainable aviation

This thesis was submitted for the award of Doctor of Philosophy and was awarded by Brunel University LondonDecarbonisation of aviation goals set by Flightpath 2050 Europe’s Vision for Aviation requires that the airports become emission-free by 2050. This thesis original contribution to knowledge is to explore the incorporation of aviation electrification technologies, including electric aircraft (EA), electrified ground support equipment (GSE), and airport parking electric vehicles (EVs), into power systems, evaluating their influence on grid infrastructure and operations, as well as their potential to support the grid operation. A comprehensive review of aviation electrification technologies revealed a research gap in the integration of these technologies into the power systems. The thesis contributes to electricity network infrastructure planning for electrification of aviation and airport-based distributed energy resources (DER) that provide ancillary services to the power grid. A multi-objective airport microgrid planning framework is developed, comparing EA charging strategies and revealing that battery swap performs better. Vehicle-to-grid (V2G) strategy with parking EVs improves the microgrid's performance. A techno-economic assessment of wireless charging systems for electric airport shuttle buses shows better economic performance than conventional buses and other charging options. A novel Aviation-to-Grid (A2G) flexibility concept provides frequency response services to the GB power system using EA battery charging systems, with typical A2G service capacity showing significant variation across eight UK airports. A deep reinforcement learning (DRL)-based A2G dispatch approach evaluates the impact of EA charger capacity on energy dispatch results, with higher capacities leading to higher revenue and lower operation costs. To summarise, this thesis addresses the research gaps in integrating aviation electrification technologies into power systems, offering valuable insights for airport operators aiming to decarbonise air transport activities through the adoption of these technologies. The study also provides an understanding of the impacts on grid operators in terms of infrastructure planning and operations. This comprehensive approach ensures a cohesive understanding of the challenges and opportunities presented by aviation electrification and its integration into power systems

A systematic review of machine learning techniques related to local energy communities

In recent years, digitalisation has rendered machine learning a key tool for improving processes in several sectors, as in the case of electrical power systems. Machine learning algorithms are data-driven models based on statistical learning theory and employed as a tool to exploit the data generated by the power system and its users. Energy communities are emerging as novel organisations for consumers and prosumers in the distribution grid. These communities may operate differently depending on their objectives and the potential service the community wants to offer to the distribution system operator. This paper presents the conceptualisation of a local energy community on the basis of a review of 25 energy community projects. Furthermore, an extensive literature review of machine learning algorithms for local energy community applications was conducted, and these algorithms were categorised according to forecasting, storage optimisation, energy management systems, power stability and quality, security, and energy transactions. The main algorithms reported in the literature were analysed and classified as supervised, unsupervised, and reinforcement learning algorithms. The findings demonstrate the manner in which supervised learning can provide accurate models for forecasting tasks. Similarly, reinforcement learning presents interesting capabilities in terms of control-related applications.publishedVersio

Resilient Load Restoration in Microgrids Considering Mobile Energy Storage Fleets: A Deep Reinforcement Learning Approach

Mobile energy storage systems (MESSs) provide mobility and flexibility to enhance distribution system resilience. The paper proposes a Markov decision process (MDP) formulation for an integrated service restoration strategy that coordinates the scheduling of MESSs and resource dispatching of microgrids. The uncertainties in load consumption are taken into account. The deep reinforcement learning (DRL) algorithm is utilized to solve the MDP for optimal scheduling. Specifically, the twin delayed deep deterministic policy gradient (TD3) is applied to train the deep Q-network and policy network, then the well trained policy can be deployed in on-line manner to perform multiple actions simultaneously. The proposed model is demonstrated on an integrated test system with three microgrids connected by Sioux Falls transportation network. The simulation results indicate that mobile and stationary energy resources can be well coordinated to improve system resilience.Comment: Submitted to 2020 IEEE Power and Energy Society General Meetin

Multi-agent reinforcement learning for the coordination of residential energy flexibility

This thesis investigates whether residential energy flexibility can be coordinated without sharing personal data at scale to achieve a positive impact on energy users and the grid. To tackle climate change, energy uses are being electrified at pace, just as electricity is increasingly provided by non-dispatchable renewable energy sources. These shifts increase the requirements for demand-side flexibility. Despite the potential of residential energy to provide such flexibility, it has remained largely untapped due to cost, social acceptance, and technical barriers. This thesis investigates the use of multi-agent reinforcement learning to overcome these challenges. This thesis presents a novel testing environment, which models electric vehicles, space heating, and flexible household loads in a distribution network. Additionally, a generative adversarial network-based data generator is developed to obtain realistic training and testing data. Experiments conducted in this environment showed that standard independent learners fail to coordinate in the partially observable stochastic environment. To address this, additional coordination mechanisms are proposed for agents to practise coordination in a centralised simulated rehearsal, ahead of fully decentralised implementation. Two such coordination mechanisms are proposed: optimisation-informed independent learning, and a centralised but factored critic network. In the former, agents lean from omniscient convex optimisation results ahead of fully decentralised coordination. This enables cooperation at scale where standard independent learners under partial observability could not be coordinated. In the latter, agents employ a deep neural factorisation network to learn to assess their impact on global rewards. This approach delivers comparable performance for four agents and more, with a 34-fold speed improvement for 30 agents and only first-order computational time growth. Finally, the impacts of implementing implicit coordination using these multi- agent reinforcement learning methodologies are modelled. It is observed that even without explicit grid constraint management, cooperating energy users reduce the likelihood of voltage deviations. The cooperative management of voltage constraints could be further promoted by the MARL policies, whereby their likelihood could be reduced by 43.08% relative to an uncoordinated baseline, albeit with trade-offs in other costs. However, while this thesis demonstrates the technical feasibility of MARL-based cooperation, further market mechanisms are required to reward all participants for their cooperation