Reduced state space and cost function in reinforcement learning for demand response control of multiple EV charging stations

Electric vehicle (EV) charging stations represent a substantial load with significant flexibility. Balancing such load with model-free demand response (DR) based on reinforcement learning (RL) is an attractive approach. We build on previous RL research using a Markov decision process (MDP) to simultaneously coordinate multiple charging stations. The previously proposed approach is computationally expensive in terms of large training times, limiting its feasibility and practicality. We propose to a priori force the control policy to always fulfill any charging demand that does not offer any flexibility at a given point, and thus use an updated cost function. We compare the policy of the newly proposed approach with the original (costly) one, for the case of load flattening, in terms of (i) processing time to learn the RL-based charging policy, and (ii) overall performance of the policy decisions in terms of meeting the target load for unseen test data

A Deep Q-Learning based Smart Scheduling of EVs for Demand Response in Smart Grids

Economic and policy factors are driving the continuous increase in the adoption and usage of electrical vehicles (EVs). However, despite being a cleaner alternative to combustion engine vehicles, EVs have negative impacts on the lifespan of microgrid equipment and energy balance due to increased power demand and the timing of their usage. In our view grid management should leverage on EVs scheduling flexibility to support local network balancing through active participation in demand response programs. In this paper, we propose a model-free solution, leveraging Deep Q-Learning to schedule the charging and discharging activities of EVs within a microgrid to align with a target energy profile provided by the distribution system operator. We adapted the Bellman Equation to assess the value of a state based on specific rewards for EV scheduling actions and used a neural network to estimate Q-values for available actions and the epsilon-greedy algorithm to balance exploitation and exploration to meet the target energy profile. The results are promising showing that the proposed solution can effectively schedule the EVs charging and discharging actions to align with the target profile with a Person coefficient of 0.99, handling effective EVs scheduling situations that involve dynamicity given by the e-mobility features, relying only on data with no knowledge of EVs and microgrid dynamics.Comment: Submitted to journa

Grid flexibility by electrifying energy systems for sustainable aviation

This thesis was submitted for the award of Doctor of Philosophy and was awarded by Brunel University LondonDecarbonisation of aviation goals set by Flightpath 2050 Europe’s Vision for Aviation requires that the airports become emission-free by 2050. This thesis original contribution to knowledge is to explore the incorporation of aviation electrification technologies, including electric aircraft (EA), electrified ground support equipment (GSE), and airport parking electric vehicles (EVs), into power systems, evaluating their influence on grid infrastructure and operations, as well as their potential to support the grid operation. A comprehensive review of aviation electrification technologies revealed a research gap in the integration of these technologies into the power systems. The thesis contributes to electricity network infrastructure planning for electrification of aviation and airport-based distributed energy resources (DER) that provide ancillary services to the power grid. A multi-objective airport microgrid planning framework is developed, comparing EA charging strategies and revealing that battery swap performs better. Vehicle-to-grid (V2G) strategy with parking EVs improves the microgrid's performance. A techno-economic assessment of wireless charging systems for electric airport shuttle buses shows better economic performance than conventional buses and other charging options. A novel Aviation-to-Grid (A2G) flexibility concept provides frequency response services to the GB power system using EA battery charging systems, with typical A2G service capacity showing significant variation across eight UK airports. A deep reinforcement learning (DRL)-based A2G dispatch approach evaluates the impact of EA charger capacity on energy dispatch results, with higher capacities leading to higher revenue and lower operation costs. To summarise, this thesis addresses the research gaps in integrating aviation electrification technologies into power systems, offering valuable insights for airport operators aiming to decarbonise air transport activities through the adoption of these technologies. The study also provides an understanding of the impacts on grid operators in terms of infrastructure planning and operations. This comprehensive approach ensures a cohesive understanding of the challenges and opportunities presented by aviation electrification and its integration into power systems

Routing Optimization of Electric Vehicles for Charging With Event-Driven Pricing Strategy

With the increasing market penetration of electric vehicles (EVs), the charging behavior and driving characteristics of EVs have an increasing impact on the operation of power grids and traffic networks. Existing research on EV routing planning and charging navigation strategies mainly focuses on vehicle-road-network interactions, but the vehicle-to-vehicle interaction has rarely been considered, particularly in studying simultaneous charging requests. To investigate the interaction of multiple vehicles in routing planning and charging, a routing optimization of EVs for charging with an event-driven pricing strategy is proposed. The urban area of a city is taken as a case for numerical simulation, which demonstrates that the proposed strategy can not only alleviate the long-time queuing for EV fast charging but also improve the utilization rate of charging infrastructures. Note to Practitioners - This article was inspired by the concerns of difficulties for electric vehicle (EV)'s fast charging and the imbalance of the utilization rate of charging facilities. Existing route optimization and charging navigation research are mainly applicable to static traffic networks, which cannot dynamically adjust driving routes and charging strategies with real-time traffic information. Besides, the mutual impact between vehicles is rarely considered in these works in routing planning. To resolve the shortcomings of existing models, a receding-horizon-based strategy that can be applied to dynamic traffic networks is proposed. In this article, various factors that the user is concerned about within the course of driving are converted into driving costs, through which each road section of traffic networks is assigned the corresponding values. Combined with the graph theory analysis method, the mathematical form of the dynamic traffic network is presented. Then, the article carefully plans and adjusts EV driving routes and charging strategies. Numerical results demonstrate that the proposed method can significantly increase the adoption of EV fast charging while alleviating unreasonable distributions of regional charging demand.</p

Scenarios for the development of smart grids in the UK: synthesis report

‘Smart grid’ is a catch-all term for the smart options that could transform the ways society produces, delivers and consumes energy, and potentially the way we conceive of these services. Delivering energy more intelligently will be fundamental to decarbonising the UK electricity system at least possible cost, while maintaining security and reliability of supply. Smarter energy delivery is expected to allow the integration of more low carbon technologies and to be much more cost effective than traditional methods, as well as contributing to economic growth by opening up new business and innovation opportunities. Innovating new options for energy system management could lead to cost savings of up to £10bn, even if low carbon technologies do not emerge. This saving will be much higher if UK renewable energy targets are achieved. Building on extensive expert feedback and input, this report describes four smart grid scenarios which consider how the UK’s electricity system might develop to 2050. The scenarios outline how political decisions, as well as those made in regulation, finance, technology, consumer and social behaviour, market design or response, might affect the decisions of other actors and limit or allow the availability of future options. The project aims to explore the degree of uncertainty around the current direction of the electricity system and the complex interactions of a whole host of factors that may lead to any one of a wide range of outcomes. Our addition to this discussion will help decision makers to understand the implications of possible actions and better plan for the future, whilst recognising that it may take any one of a number of forms

Development Schemes of Electric Vehicle Charging Protocols and Implementation of Algorithms for Fast Charging under Dynamic Environments Leading towards Grid-to-Vehicle Integration

This thesis focuses on the development of electric vehicle (EV) charging protocols under a dynamic environment using artificial intelligence (AI), to achieve Vehicle-to-Grid (V2G) integration and promote automobile electrification. The proposed framework comprises three major complementary steps. Firstly, the DC fast charging scheme is developed under different ambient conditions such as temperature and relative humidity. Subsequently, the transient performance of the controller is improved while implementing the proposed DC fast charging scheme. Finally, various novel techno-economic scenarios and case studies are proposed to integrate EVs with the utility grid. The proposed novel scheme is composed of hierarchical stages; In the first stage, an investigation of the temperature or/and relative humidity impact on the charging process is implemented using the constant current-constant voltage (CC-CV) protocol. Where the relative humidity impact on the charging process was not investigated or mentioned in the literature survey. This was followed by the feedforward backpropagation neural network (FFBP-NN) classification algorithm supported by the statistical analysis of an instant charging current sample of only 10 seconds at any ambient condition. Then the FFBP-NN perfectly estimated the EV’s battery terminal voltage, charging current, and charging interval time with an error of 1% at the corresponding temperature and relative humidity. Then, a nonlinear identification model of the lithium-polymer ion battery dynamic behaviour is introduced based on the Hammerstein-Wiener (HW) model with an experimental error of 1.1876%. Compared with the CC-CV fast charging protocol, intelligent novel techniques based on the multistage charging current protocol (MSCC) are proposed using the Cuckoo optimization algorithm (COA). COA is applied to the Hierarchical technique (HT) and the Conditional random technique (CRT). Compared with the CC-CV charging protocol, an improvement in the charging efficiency of 8% and 14.1% was obtained by the HT and the CRT, respectively, in addition to a reduction in energy losses of 7.783% and 10.408% and a reduction in charging interval time of 18.1% and 22.45%, respectively. The stated charging protocols have been implemented throughout a smart charger. The charger comprises a DC-DC buck converter controlled by an artificial neural network predictive controller (NNPC), trained and supported by the long short-term memory neural network (LSTM). The LSTM network model was utilized in the offline forecasting of the PV output power, which was fed to the NNPC as the training data. The NNPC–LSTM controller was compared with the fuzzy logic (FL) and the conventional PID controllers and perfectly ensured that the optimum transient performance with a minimum battery terminal voltage ripple reached 1 mV with a very high-speed response of 1 ms in reaching the predetermined charging current stages. Finally, to alleviate the power demand pressure of the proposed EV charging framework on the utility grid, a novel smart techno-economic operation of an electric vehicle charging station (EVCS) in Egypt controlled by the aggregator is suggested based on a hierarchical model of multiple scenarios. The deterministic charging scheduling of the EVs is the upper stage of the model to balance the generated and consumed power of the station. Mixed-integer linear programming (MILP) is used to solve the first stage, where the EV charging peak demand value is reduced by 3.31% (4.5 kW). The second challenging stage is to maximize the EVCS profit whilst minimizing the EV charging tariff. In this stage, MILP and Markov Decision Process Reinforcement Learning (MDP-RL) resulted in an increase in EVCS revenue by 28.88% and 20.10%, respectively. Furthermore, the grid-to-vehicle (G2V) and vehicle-to-grid (V2G) technologies are applied to the stochastic EV parking across the day, controlled by the aggregator to alleviate the utility grid load demand. The aggregator determined the number of EVs that would participate in the electric power trade and sets the charging/discharging capacity level for each EV. The proposed model minimized the battery degradation cost while maximizing the revenue of the EV owner and minimizing the utility grid load demand based on the genetic algorithm (GA). The implemented procedure reduced the degradation cost by an average of 40.9256%, increased the EV SOC by 27%, and ensured an effective grid stabilization service by shaving the load demand to reach a predetermined grid average power across the day where the grid load demand decreased by 26.5% (371 kW)

Reinforcement learning for power scheduling in a grid-tied pv-battery electric vehicles charging station

Grid-tied renewable energy sources (RES) based electric vehicle (EV) charging stations are an example of a distributed generator behind the meter system (DGBMS) which characterizes most modern power infrastructure. To perform power scheduling in such a DGBMS, stochastic variables such as load profile of the charging station, output profile of the RES and tariff profile of the utility must be considered at every decision step. The stochasticity in this kind of optimization environment makes power scheduling a challenging task that deserves substantial research attention. This dissertation investigates the application of reinforcement learning (RL) techniques in solving the power scheduling problem in a grid-tied PV-powered EV charging station with the incorporation of a battery energy storage system. RL is a reward-motivated optimization technique that was derived from the way animals learn to optimize their behavior in a new environment. Unlike other optimization methods such as numerical and soft computing techniques, RL does not require an accurate model of the optimization environment in order to arrive at an optimal solution. This study developed and evaluated the feasibility of two RL algorithms, namely, an asynchronous Q-learning algorithm and an advantage actor-critic (A2C) algorithm, in performing power scheduling in the EV charging station under static conditions. To assess the performances of the proposed algorithms, the conventional Q-learning and actor-critic algorithm were implemented to compare their global cost convergence and their learning characteristics. First, the power scheduling problem was expressed as a sequential decision-making process. Then an asynchronous Q-learning algorithm was developed to solve it. Further, an advantage actor-critic (A2C) algorithm was developed and was used to solve the power scheduling problem. The two algorithms were tested using a 24-hour load, generation and utility grid tariff profiles under static optimization conditions. The performance of the asynchronous Q-learning algorithm was compared with that of the conventional Q-learning method in terms of the global cost, stability and scalability. Likewise, the A2C was compared with the conventional actor-critic method in terms of stability, scalability and convergence. Simulation results showed that both the developed algorithms (asynchronous Q-learning algorithm and A2C) converged to lower global costs and displayed more stable learning characteristics than their conventional counterparts. This research established that proper restriction of the action-space of a Q-learning algorithm improves its stability and convergence. It was also observed that such a restriction may come with compromise of computational speed and scalability. Of the four algorithms analyzed, the A2C was found to produce a power schedule with the lowest global cost and the best usage of the battery energy storage system