A Deep Reinforcement Learning-Based Charging Scheduling Approach with Augmented Lagrangian for Electric Vehicle

This paper addresses the problem of optimizing charging/discharging schedules of electric vehicles (EVs) when participate in demand response (DR). As there exist uncertainties in EVs' remaining energy, arrival and departure time, and future electricity prices, it is quite difficult to make charging decisions to minimize charging cost while guarantee that the EV's battery state-of-the-charge (SOC) is within certain range. To handle with this dilemma, this paper formulates the EV charging scheduling problem as a constrained Markov decision process (CMDP). By synergistically combining the augmented Lagrangian method and soft actor critic algorithm, a novel safe off-policy reinforcement learning (RL) approach is proposed in this paper to solve the CMDP. The actor network is updated in a policy gradient manner with the Lagrangian value function. A double-critics network is adopted to synchronously estimate the action-value function to avoid overestimation bias. The proposed algorithm does not require strong convexity guarantee of examined problems and is sample efficient. Comprehensive numerical experiments with real-world electricity price demonstrate that our proposed algorithm can achieve high solution optimality and constraints compliance

Learning-Augmented Scheduling for Solar-Powered Electric Vehicle Charging

We tackle the complex challenge of scheduling the charging of electric vehicles (EVs) equipped with solar panels and batteries, particularly under out-of-distribution (OOD) conditions. Traditional scheduling approaches, such as reinforcement learning (RL) and model predictive control (MPC), often fail to provide satisfactory results when faced with OOD data, struggling to balance robustness (worst-case performance) and consistency (near-optimal average performance). To address this gap, we introduce a novel learning-augmented policy. This policy employs a dynamic robustness budget, which is adapted in real-time based on the reinforcement learning policy's performance. Specifically, it leverages the temporal difference (TD) error, a measure of the learning policy's prediction accuracy, to assess the trustworthiness of the machine-learned policy. This method allows for a more effective balance between consistency and robustness in EV charging schedules, significantly enhancing adaptability and efficiency in real-world, unpredictable environments. Our results demonstrate that this approach markedly improves scheduling effectiveness and reliability, particularly in OOD contexts, paving the way for more resilient and adaptive EV charging systems

Optimal Scheduling of Electric Vehicle Charging with Deep Reinforcement Learning considering End Users Flexibility

The rapid growth of decentralized energy resources and especially Electric Vehicles (EV), that are expected to increase sharply over the next decade, will put further stress on existing power distribution networks, increasing the need for higher system reliability and flexibility. In an attempt to avoid unnecessary network investments and to increase the controllability over distribution networks, network operators develop demand response (DR) programs that incentivize end users to shift their consumption in return for financial or other benefits. Artificial intelligence (AI) methods are in the research forefront for residential load scheduling applications, mainly due to their high accuracy, high computational speed and lower dependence on the physical characteristics of the models under development. The aim of this work is to identify households' EV cost-reducing charging policy under a Time-of-Use tariff scheme, with the use of Deep Reinforcement Learning, and more specifically Deep Q-Networks (DQN). A novel end users flexibility potential reward is inferred from historical data analysis, where households with solar power generation have been used to train and test the designed algorithm. The suggested DQN EV charging policy can lead to more than 20% of savings in end users electricity bills

A Deep Q-Learning based Smart Scheduling of EVs for Demand Response in Smart Grids

Economic and policy factors are driving the continuous increase in the adoption and usage of electrical vehicles (EVs). However, despite being a cleaner alternative to combustion engine vehicles, EVs have negative impacts on the lifespan of microgrid equipment and energy balance due to increased power demand and the timing of their usage. In our view grid management should leverage on EVs scheduling flexibility to support local network balancing through active participation in demand response programs. In this paper, we propose a model-free solution, leveraging Deep Q-Learning to schedule the charging and discharging activities of EVs within a microgrid to align with a target energy profile provided by the distribution system operator. We adapted the Bellman Equation to assess the value of a state based on specific rewards for EV scheduling actions and used a neural network to estimate Q-values for available actions and the epsilon-greedy algorithm to balance exploitation and exploration to meet the target energy profile. The results are promising showing that the proposed solution can effectively schedule the EVs charging and discharging actions to align with the target profile with a Person coefficient of 0.99, handling effective EVs scheduling situations that involve dynamicity given by the e-mobility features, relying only on data with no knowledge of EVs and microgrid dynamics.Comment: Submitted to journa

DClEVerNet: Deep Combinatorial Learning for Efficient EV Charging Scheduling in Large-scale Networked Facilities

With the electrification of transportation, the rising uptake of electric vehicles (EVs) might stress distribution networks significantly, leaving their performance degraded and stability jeopardized. To accommodate these new loads cost-effectively, modern power grids require coordinated or ``smart'' charging strategies capable of optimizing EV charging scheduling in a scalable and efficient fashion. With this in view, the present work focuses on reservation management programs for large-scale, networked EV charging stations. We formulate a time-coupled binary optimization problem that maximizes EV users' total welfare gain while accounting for the network's available power capacity and stations' occupancy limits. To tackle the problem at scale while retaining high solution quality, a data-driven optimization framework combining techniques from the fields of Deep Learning and Approximation Algorithms is introduced. The framework's key ingredient is a novel input-output processing scheme for neural networks that allows direct extrapolation to problem sizes substantially larger than those included in the training set. Extensive numerical simulations based on synthetic and real-world data traces verify the effectiveness and superiority of the presented approach over two representative scheduling algorithms. Lastly, we round up the contributions by listing several immediate extensions to the proposed framework and outlining the prospects for further exploration

Scheduling Electric Vehicle Charging for Grid Load Balancing

In recent years, electric vehicles (EVs) have been widely adopted because of their environmental benefits. However, the increasing volume of EVs poses capacity issues for grid operators as simultaneously charging many EVs may result in grid instabilities. Scheduling EV charging for grid load balancing has a potential to prevent load peaks caused by simultaneous EV charging and contribute to balance of supply and demand. This paper proposes a user-preference-based scheduling approach to minimize costs for the user while balancing grid loads. The EV owners benefit by charging when the electricity cost is lower, but still within the user-defined preferred charging periods. On the other hand, the approach reduces the pressure on the grid by balancing the grid load. Two methods, the greedy algorithm and nonlinear programming, are considered along with users’ charging preferences and durations. For scheduling small numbers of charging activities, the nonlinear programming model achieves better load balancing than the greedy algorithm; however, for scheduling medium to large numbers of charging activities, the greedy algorithm has a clear advantage in terms of time complexity

Federated Reinforcement Learning for Electric Vehicles Charging Control on Distribution Networks

With the growing popularity of electric vehicles (EVs), maintaining power grid stability has become a significant challenge. To address this issue, EV charging control strategies have been developed to manage the switch between vehicle-to-grid (V2G) and grid-to-vehicle (G2V) modes for EVs. In this context, multi-agent deep reinforcement learning (MADRL) has proven its effectiveness in EV charging control. However, existing MADRL-based approaches fail to consider the natural power flow of EV charging/discharging in the distribution network and ignore driver privacy. To deal with these problems, this paper proposes a novel approach that combines multi-EV charging/discharging with a radial distribution network (RDN) operating under optimal power flow (OPF) to distribute power flow in real time. A mathematical model is developed to describe the RDN load. The EV charging control problem is formulated as a Markov Decision Process (MDP) to find an optimal charging control strategy that balances V2G profits, RDN load, and driver anxiety. To effectively learn the optimal EV charging control strategy, a federated deep reinforcement learning algorithm named FedSAC is further proposed. Comprehensive simulation results demonstrate the effectiveness and superiority of our proposed algorithm in terms of the diversity of the charging control strategy, the power fluctuations on RDN, the convergence efficiency, and the generalization ability

Grid flexibility by electrifying energy systems for sustainable aviation

This thesis was submitted for the award of Doctor of Philosophy and was awarded by Brunel University LondonDecarbonisation of aviation goals set by Flightpath 2050 Europe’s Vision for Aviation requires that the airports become emission-free by 2050. This thesis original contribution to knowledge is to explore the incorporation of aviation electrification technologies, including electric aircraft (EA), electrified ground support equipment (GSE), and airport parking electric vehicles (EVs), into power systems, evaluating their influence on grid infrastructure and operations, as well as their potential to support the grid operation. A comprehensive review of aviation electrification technologies revealed a research gap in the integration of these technologies into the power systems. The thesis contributes to electricity network infrastructure planning for electrification of aviation and airport-based distributed energy resources (DER) that provide ancillary services to the power grid. A multi-objective airport microgrid planning framework is developed, comparing EA charging strategies and revealing that battery swap performs better. Vehicle-to-grid (V2G) strategy with parking EVs improves the microgrid's performance. A techno-economic assessment of wireless charging systems for electric airport shuttle buses shows better economic performance than conventional buses and other charging options. A novel Aviation-to-Grid (A2G) flexibility concept provides frequency response services to the GB power system using EA battery charging systems, with typical A2G service capacity showing significant variation across eight UK airports. A deep reinforcement learning (DRL)-based A2G dispatch approach evaluates the impact of EA charger capacity on energy dispatch results, with higher capacities leading to higher revenue and lower operation costs. To summarise, this thesis addresses the research gaps in integrating aviation electrification technologies into power systems, offering valuable insights for airport operators aiming to decarbonise air transport activities through the adoption of these technologies. The study also provides an understanding of the impacts on grid operators in terms of infrastructure planning and operations. This comprehensive approach ensures a cohesive understanding of the challenges and opportunities presented by aviation electrification and its integration into power systems