Definition and evaluation of model-free coordination of electrical vehicle charging with reinforcement learning

Demand response (DR) becomes critical to manage the charging load of a growing electric vehicle (EV) deployment. Initial DR studies mainly adopt model predictive control, but models are largely uncertain for the EV scenario (e.g., customer behavior). Model-free approaches, based on reinforcement learning (RL), are an attractive alternative. We propose a new Markov decision process (MDP) formulation in the RL framework, to jointly coordinate a set of charging stations. State-of-the-art algorithms either focus on a single EV, or control an aggregate of EVs in multiple steps (e.g., 1) make aggregate load decisions and 2) translate the aggregate decision to individual EVs). In contrast, our RL approach jointly controls the whole set of EVs at once. We contribute a new MDP formulation with a scalable state representation independent of the number of charging stations. Using a batch RL algorithm, fitted $Q$ -iteration, we learn an optimal charging policy. With simulations using real-world data, we: 1) differentiate settings in training the RL policy (e.g., the time span covered by training data); 2) compare its performance to an oracle all-knowing benchmark (providing an upper performance bound); 3) analyze performance fluctuations throughout a full year; and 4) demonstrate generalization capacity to larger sets of charging stations

Reinforcement learning for EV charging optimization : A holistic perspective for commercial vehicle fleets

Recent years have seen an unprecedented uptake in electric vehicles, driven by the global push to reduce carbon emissions. At the same time, intermittent renewables are being deployed increasingly. These developments are putting flexibility measures such as dynamic load management in the spotlight of the energy transition. Flexibility measures must consider EV charging, as it has the ability to introduce grid constraints: In Germany, the cumulative power of all EV onboard chargers amounts to ca. 120 GW, while the German peak load only amounts to 80 GW. Commercial operations have strong incentives to optimize charging and flatten peak loads in real-time, given that the highest quarter-hour can determine the power-related energy bill, and that a blown fuse due to overloading can halt operations. Increasing research efforts have therefore gone into real-time-capable optimization methods. Reinforcement Learning (RL) has particularly gained attention due to its versatility, performance and real- time capabilities. This thesis implements such an approach and introduces FleetRL as a realistic RL environment for EV charging, with a focus on commercial vehicle fleets. Through its implementation, it was found that RL saved up to 83% compared to static benchmarks, and that grid overloading was entirely avoided in some scenarios by sacrificing small portions of SOC, or by delaying the charging process. Linear optimization with one year of perfect knowledge outperformed RL, but reached its practical limits in one use-case, where a feasible solution could not be found by the solver. Overall, this thesis makes a strong case for RL-based EV charging. It further provides a foundation which can be built upon: a modular, open-source software framework that integrates an MDP model, schedule generation, and non-linear battery degradationElektrifieringen av transportsektorn är en nödvändig men utmanande uppgift. I kombination med ökande solcellsproduktion och förnybara energikällor skapar det ett dilemma för elnätet som kräver omfattande flexibilitetsåtgärder. Dessa åtgärder måste inkludera laddning av elbilar, ett fenomen som har lett till aldrig tidigare skådade belastningstoppar. Ur ett kommersiellt perspektiv är incitamentet att optimera laddningsprocessen och säkerställa drifttid. Forskningen har fokuserat på realtidsoptimeringsmetoder som Deep Reinforcement Learning (DRL). Denna avhandling introducerar FleetRL som en ny RL-miljö för EV-laddning av kommersiella flottor. Genom att tillämpa ramverket visade det sig att RL sparade upp till 83% jämfört med statiska riktmärken, och att överbelastning av nätet helt kunde undvikas i de flesta scenarier. Linjär optimering överträffade RL men nådde sina gränser i snävt begränsade användningsfall. Efter att ha funnit ett positivt business case för varje kommersiellt användningsområde, ger denna avhandling ett starkt argument för RL-baserad laddning och en grund för framtida arbete via praktiska insikter och ett modulärt mjukvaruramverk med öppen källko

A Novel Reinforcement Learning-Optimization Approach for Integrating Wind Energy to Power System with Vehicle-to-Grid Technology

High integration of intermittent renewable energy sources (RES), specifically wind power, has created complexities in power system operations due to their limited controllability and predictability. In addition, large fleets of Electric Vehicles (EVs) are expected to have a large impact on electricity consumption, contributing to the volatility. In this dissertation, a well-coordinated smart charging approach is developed that utilizes the flexibility of EV owners in a way where EVs are used as distributed energy storage units and flexible loads to absorb the fluctuations in the wind power output in a vehicle-to-grid (V2G) setup. Challenges for people participation in V2G, such as battery degradation and insecurity about unexpected trips, are also addressed by using an interactive mechanism in smart grid. First, a static deterministic model is formulated using multi-objective mixed-integer quadratic programming (MIQP) assuming known parameters day ahead of time. Subsequently, a formulation for real-time dynamic schedule is provided using a rolling-horizon with expected value approximation. Simulation experiments demonstrate a significant increase in wind utilization and reduction in charging cost and battery degradation compared to an uncontrolled charging scenario. Formulating the scheduling problem of the EV-wind integrated power system using conventional stochastic programming (SP) approaches is challenging due to the presence of many uncertain parameters with unknown underlying distributions, such as wind, price, and different commuting patterns of EV owners. To alleviate the problem, a model-free Reinforcement Learning (RL) algorithm integrated with deterministic optimization is proposed that can be applied on many multi-stage stochastic problems while mitigating some of the challenges of conventional SP methods (e.g., large scenario tree, computational complexity) as well as the challenges in model-free RL (e.g., slow convergence, unstable learning in dynamic environment). The simulation results of applying the combined approach on the EV scheduling problem demonstrate the effectiveness of the RL-Optimization method in solving the multi-stage EV charge/discharge scheduling problem. The proposed methods perform better than standard RL approaches (e.g., DDQN) in terms of convergence speed and finding the global optima. Moreover, to address the curse of dimensionality issue in RL with large action-state space, a heuristic EV fleet charging/discharging scheme is used combined with RL-optimization approach to solve the EV scheduling problem for a large number of EVs

Power trip : smart management of shared autonomous electric vehicle fleets

The climate crisis requires substantial shifts in the transportation and energy sectors. Greater use of intermittent renewable energy sources requires demand- and supply-side flexibility in electricity markets. Deployment of on-demand, shared, fully automated, and electric vehicle (SAEV) fleets offers natural synergies in meeting such challenges. Smart charging (and discharging) of electric vehicles (EVs) can shift loads away from peak demand to reduce, or at least delay, expensive infrastructure upgrades, while fleet managers lower emissions and power costs in real time. This dissertation explores (1) optimization-based idle-vehicle dispatch strategies to improve SAEV fleet operations in the Austin metro, (2) integration of power and transportation system (EV-use) modeling across the Chicago metro area, and (3) a case study of demand response participation and charging station siting in a region with multiple energy suppliers. Optimizing SAEV repositioning and charging dispatch strategies jointly lowered rider wait times by 39%, on average, and increased daily trips served per SAEV by 28% (up to 6.4 additional riders), compared to separate range-agnostic repositioning and heuristic charging strategies. Joint strategies may also decrease the SAEV fleet’s empty travel by 5.7 to 12.8 percentage points (depending on geofencing and charging station density). If fleets pay dynamic electricity prices and wish to internalize their upstream charging emissions damages, a new multi-stage charging problem is required. A day-ahead energy transaction problem provides targets for a within-day idle-vehicle dispatch strategy that balances charging, discharging, repositioning, and maintenance decisions. This strategy allowed the Austin SAEV fleet to lower daily power costs (by 15.5% or $0.79/day/SAEV, on average) while reducing health damages from generation-related pollution (2.8$0.43/day/SAEV, on average). Fleet managers obtained higher profits ($8 per SAEV per day) by serving more passengers per day than with simpler (price-agnostic) dispatch strategies. This dissertation also coupled an agent-based travel demand simulator (POLARIS) with an electricity grid model (A-LEAF) to evaluate charging impacts on the power grid across seasons, household-EV adoption levels, SAEV mode shares, and dynamic ride-sharing assumptions in 2035 for the Chicago, Illinois metro. At relatively low EV penetration levels (8$100/MWh, or $0.10/kWh) and peak summer hours (+$300/MWh) due to higher energy fees and steep congestion fees on Illinois’ 2015-era transmission system. Although a smart-charging SAEV fleet uses wholesale prices to reduce electricity demand during peak hours, spreading charging demand in hours before and after the baseline peak creates new "ridges" in energy demand, which raise prices for all. These simulation results underscore the importance of investing in transmission system expansion and reducing barriers to upgrading or building new transmission infrastructure. If vehicles and chargers support bidirectional charging, SAEVs can improve grid reliability and resilience at critical times through demand response (DR) programs that allow load curtailment and vehicle-to-grid (V2G) power. Scenario testing of DR requests in Austin ranging from 1 MW to 12 MW between 4 and 5 PM reveals break-even compensation costs (to SAEV owners) that range from $86/kW to$4,160/kW (if the city imposes unoccupied travel fees), depending on vehicle locations and battery levels at the time of the DR request. Smaller requests can be met without V2G by reducing charging speeds, usually from 120 kW speed to Level 2 charging. Finally, an incremental charging station heuristic was designed to capture differences in land costs and electricity rate structures from different energy suppliers in the same region. The daily amortized costs over 10 years of hardware, installation, and land costs were estimated to be nearly $0.30/SAEV/day, compared to$0.38/SAEV/day with a baseline heuristic strategy ignoring land costs and marginal costs of expanding existing sites. SAEV charging costs showed no substantial difference between heuristic strategies, although combined daily energy fees were more expensive at $0.43/SAEV/day. Including land costs in charging station investment heuristics is necessary, and modelers should include spatially varying energy prices since the average daily per-vehicle energy costs are higher than the physical station costs. Taken together, this dissertation’s contributions offer hope for a decarbonizing world that provides affordable, clean, and convenient on-demand mobility.Civil, Architectural, and Environmental Engineerin