1,575 research outputs found
Recommended from our members
Reinforcement Learning for Hybrid and Plug-In Hybrid Electric Vehicle Energy Management: Recent Advances and Prospects
A Deep Q-Learning based Smart Scheduling of EVs for Demand Response in Smart Grids
Economic and policy factors are driving the continuous increase in the
adoption and usage of electrical vehicles (EVs). However, despite being a
cleaner alternative to combustion engine vehicles, EVs have negative impacts on
the lifespan of microgrid equipment and energy balance due to increased power
demand and the timing of their usage. In our view grid management should
leverage on EVs scheduling flexibility to support local network balancing
through active participation in demand response programs. In this paper, we
propose a model-free solution, leveraging Deep Q-Learning to schedule the
charging and discharging activities of EVs within a microgrid to align with a
target energy profile provided by the distribution system operator. We adapted
the Bellman Equation to assess the value of a state based on specific rewards
for EV scheduling actions and used a neural network to estimate Q-values for
available actions and the epsilon-greedy algorithm to balance exploitation and
exploration to meet the target energy profile. The results are promising
showing that the proposed solution can effectively schedule the EVs charging
and discharging actions to align with the target profile with a Person
coefficient of 0.99, handling effective EVs scheduling situations that involve
dynamicity given by the e-mobility features, relying only on data with no
knowledge of EVs and microgrid dynamics.Comment: Submitted to journa
A Deep Reinforcement Learning-Based Charging Scheduling Approach with Augmented Lagrangian for Electric Vehicle
This paper addresses the problem of optimizing charging/discharging schedules
of electric vehicles (EVs) when participate in demand response (DR). As there
exist uncertainties in EVs' remaining energy, arrival and departure time, and
future electricity prices, it is quite difficult to make charging decisions to
minimize charging cost while guarantee that the EV's battery
state-of-the-charge (SOC) is within certain range. To handle with this dilemma,
this paper formulates the EV charging scheduling problem as a constrained
Markov decision process (CMDP). By synergistically combining the augmented
Lagrangian method and soft actor critic algorithm, a novel safe off-policy
reinforcement learning (RL) approach is proposed in this paper to solve the
CMDP. The actor network is updated in a policy gradient manner with the
Lagrangian value function. A double-critics network is adopted to synchronously
estimate the action-value function to avoid overestimation bias. The proposed
algorithm does not require strong convexity guarantee of examined problems and
is sample efficient. Comprehensive numerical experiments with real-world
electricity price demonstrate that our proposed algorithm can achieve high
solution optimality and constraints compliance
Learning-Augmented Scheduling for Solar-Powered Electric Vehicle Charging
We tackle the complex challenge of scheduling the charging of electric
vehicles (EVs) equipped with solar panels and batteries, particularly under
out-of-distribution (OOD) conditions. Traditional scheduling approaches, such
as reinforcement learning (RL) and model predictive control (MPC), often fail
to provide satisfactory results when faced with OOD data, struggling to balance
robustness (worst-case performance) and consistency (near-optimal average
performance). To address this gap, we introduce a novel learning-augmented
policy. This policy employs a dynamic robustness budget, which is adapted in
real-time based on the reinforcement learning policy's performance.
Specifically, it leverages the temporal difference (TD) error, a measure of the
learning policy's prediction accuracy, to assess the trustworthiness of the
machine-learned policy. This method allows for a more effective balance between
consistency and robustness in EV charging schedules, significantly enhancing
adaptability and efficiency in real-world, unpredictable environments. Our
results demonstrate that this approach markedly improves scheduling
effectiveness and reliability, particularly in OOD contexts, paving the way for
more resilient and adaptive EV charging systems
An Efficient Distributed Multi-Agent Reinforcement Learning for EV Charging Network Control
The increasing trend in adopting electric vehicles (EVs) will significantly
impact the residential electricity demand, which results in an increased risk
of transformer overload in the distribution grid. To mitigate such risks, there
are urgent needs to develop effective EV charging controllers. Currently, the
majority of the EV charge controllers are based on a centralized approach for
managing individual EVs or a group of EVs. In this paper, we introduce a
decentralized Multi-agent Reinforcement Learning (MARL) charging framework that
prioritizes the preservation of privacy for EV owners. We employ the
Centralized Training Decentralized Execution-Deep Deterministic Policy Gradient
(CTDE-DDPG) scheme, which provides valuable information to users during
training while maintaining privacy during execution. Our results demonstrate
that the CTDE framework improves the performance of the charging network by
reducing the network costs. Moreover, we show that the Peak-to-Average Ratio
(PAR) of the total demand is reduced, which, in turn, reduces the risk of
transformer overload during the peak hours.Comment: 8 pages, 4 figures, accepted at Allerton 202
Optimal Scheduling of Electric Vehicle Charging with Deep Reinforcement Learning considering End Users Flexibility
The rapid growth of decentralized energy resources and especially Electric
Vehicles (EV), that are expected to increase sharply over the next decade, will
put further stress on existing power distribution networks, increasing the need
for higher system reliability and flexibility. In an attempt to avoid
unnecessary network investments and to increase the controllability over
distribution networks, network operators develop demand response (DR) programs
that incentivize end users to shift their consumption in return for financial
or other benefits. Artificial intelligence (AI) methods are in the research
forefront for residential load scheduling applications, mainly due to their
high accuracy, high computational speed and lower dependence on the physical
characteristics of the models under development. The aim of this work is to
identify households' EV cost-reducing charging policy under a Time-of-Use
tariff scheme, with the use of Deep Reinforcement Learning, and more
specifically Deep Q-Networks (DQN). A novel end users flexibility potential
reward is inferred from historical data analysis, where households with solar
power generation have been used to train and test the designed algorithm. The
suggested DQN EV charging policy can lead to more than 20% of savings in end
users electricity bills
Definition and evaluation of model-free coordination of electrical vehicle charging with reinforcement learning
Demand response (DR) becomes critical to manage the charging load of a growing electric vehicle (EV) deployment. Initial DR studies mainly adopt model predictive control, but models are largely uncertain for the EV scenario (e.g., customer behavior). Model-free approaches, based on reinforcement learning (RL), are an attractive alternative. We propose a new Markov decision process (MDP) formulation in the RL framework, to jointly coordinate a set of charging stations. State-of-the-art algorithms either focus on a single EV, or control an aggregate of EVs in multiple steps (e.g., 1) make aggregate load decisions and 2) translate the aggregate decision to individual EVs). In contrast, our RL approach jointly controls the whole set of EVs at once. We contribute a new MDP formulation with a scalable state representation independent of the number of charging stations. Using a batch RL algorithm, fitted -iteration, we learn an optimal charging policy. With simulations using real-world data, we: 1) differentiate settings in training the RL policy (e.g., the time span covered by training data); 2) compare its performance to an oracle all-knowing benchmark (providing an upper performance bound); 3) analyze performance fluctuations throughout a full year; and 4) demonstrate generalization capacity to larger sets of charging stations
Federated Reinforcement Learning for Electric Vehicles Charging Control on Distribution Networks
With the growing popularity of electric vehicles (EVs), maintaining power
grid stability has become a significant challenge. To address this issue, EV
charging control strategies have been developed to manage the switch between
vehicle-to-grid (V2G) and grid-to-vehicle (G2V) modes for EVs. In this context,
multi-agent deep reinforcement learning (MADRL) has proven its effectiveness in
EV charging control. However, existing MADRL-based approaches fail to consider
the natural power flow of EV charging/discharging in the distribution network
and ignore driver privacy. To deal with these problems, this paper proposes a
novel approach that combines multi-EV charging/discharging with a radial
distribution network (RDN) operating under optimal power flow (OPF) to
distribute power flow in real time. A mathematical model is developed to
describe the RDN load. The EV charging control problem is formulated as a
Markov Decision Process (MDP) to find an optimal charging control strategy that
balances V2G profits, RDN load, and driver anxiety. To effectively learn the
optimal EV charging control strategy, a federated deep reinforcement learning
algorithm named FedSAC is further proposed. Comprehensive simulation results
demonstrate the effectiveness and superiority of our proposed algorithm in
terms of the diversity of the charging control strategy, the power fluctuations
on RDN, the convergence efficiency, and the generalization ability
Recommended from our members
Grid flexibility by electrifying energy systems for sustainable aviation
This thesis was submitted for the award of Doctor of Philosophy and was awarded by Brunel University LondonDecarbonisation of aviation goals set by Flightpath 2050 Europe’s Vision for Aviation
requires that the airports become emission-free by 2050. This thesis original contribution to
knowledge is to explore the incorporation of aviation electrification technologies, including
electric aircraft (EA), electrified ground support equipment (GSE), and airport parking electric
vehicles (EVs), into power systems, evaluating their influence on grid infrastructure and
operations, as well as their potential to support the grid operation.
A comprehensive review of aviation electrification technologies revealed a research gap in the
integration of these technologies into the power systems. The thesis contributes to electricity
network infrastructure planning for electrification of aviation and airport-based distributed
energy resources (DER) that provide ancillary services to the power grid.
A multi-objective airport microgrid planning framework is developed, comparing EA charging
strategies and revealing that battery swap performs better. Vehicle-to-grid (V2G) strategy with
parking EVs improves the microgrid's performance. A techno-economic assessment of wireless charging
systems for electric airport shuttle buses shows better economic performance than conventional
buses and other charging options.
A novel Aviation-to-Grid (A2G) flexibility concept provides frequency response services to the GB
power system using EA battery charging systems, with typical A2G service capacity showing
significant variation across eight UK airports. A deep reinforcement learning (DRL)-based A2G
dispatch approach evaluates the impact of EA charger capacity on energy dispatch results, with
higher capacities leading to higher revenue and lower operation costs.
To summarise, this thesis addresses the research gaps in integrating aviation
electrification technologies into power systems, offering valuable insights for airport operators
aiming to decarbonise air transport activities through the adoption of these technologies. The
study also provides an understanding of the impacts on grid operators in terms of infrastructure
planning and operations. This comprehensive approach ensures a cohesive understanding of the
challenges and opportunities presented by aviation
electrification and its integration into power systems
Learning-based Predictive Control via Real-time Aggregate Flexibility
Aggregators have emerged as crucial tools for the coordination of
distributed, controllable loads. To be used effectively, an aggregator must be
able to communicate the available flexibility of the loads they control, as
known as the aggregate flexibility to a system operator. However, most of
existing aggregate flexibility measures often are slow-timescale estimations
and much less attention has been paid to real-time coordination between an
aggregator and an operator. In this paper, we consider solving an online
optimization in a closed-loop system and present a design of real-time
aggregate flexibility feedback, termed the maximum entropy feedback (MEF). In
addition to deriving analytic properties of the MEF, combining learning and
control, we show that it can be approximated using reinforcement learning and
used as a penalty term in a novel control algorithm -- the penalized predictive
control (PPC), which modifies vanilla model predictive control (MPC). The
benefits of our scheme are (1). Efficient Communication. An operator running
PPC does not need to know the exact states and constraints of the loads, but
only the MEF. (2). Fast Computation. The PPC often has much less number of
variables than an MPC formulation. (3). Lower Costs. We show that under certain
regularity assumptions, the PPC is optimal. We illustrate the efficacy of the
PPC using a dataset from an adaptive electric vehicle charging network and show
that PPC outperforms classical MPC.Comment: 13 pages, 5 figures, extension of arXiv:2006.1381
- …