Definition and evaluation of model-free coordination of electrical vehicle charging with reinforcement learning

Demand response (DR) becomes critical to manage the charging load of a growing electric vehicle (EV) deployment. Initial DR studies mainly adopt model predictive control, but models are largely uncertain for the EV scenario (e.g., customer behavior). Model-free approaches, based on reinforcement learning (RL), are an attractive alternative. We propose a new Markov decision process (MDP) formulation in the RL framework, to jointly coordinate a set of charging stations. State-of-the-art algorithms either focus on a single EV, or control an aggregate of EVs in multiple steps (e.g., 1) make aggregate load decisions and 2) translate the aggregate decision to individual EVs). In contrast, our RL approach jointly controls the whole set of EVs at once. We contribute a new MDP formulation with a scalable state representation independent of the number of charging stations. Using a batch RL algorithm, fitted $Q$ -iteration, we learn an optimal charging policy. With simulations using real-world data, we: 1) differentiate settings in training the RL policy (e.g., the time span covered by training data); 2) compare its performance to an oracle all-knowing benchmark (providing an upper performance bound); 3) analyze performance fluctuations throughout a full year; and 4) demonstrate generalization capacity to larger sets of charging stations

Quantitive analysis of electric vehicle flexibility : a data-driven approach

The electric vehicle (EV) flexibility, indicates to what extent the charging load can be coordinated (i.e., to flatten the load curve or to utilize renewable energy resources). However, such flexibility is neither well analyzed nor effectively quantified in literature. In this paper we fill this gap and offer an extensive analysis of the flexibility characteristics of 390k EV charging sessions and propose measures to quantize their flexibility exploitation. Our contributions include: (1) characterization of the EV charging behavior by clustering the arrival and departure time combinations that leads to the identification of type of EV charging behavior, (2) in-depth analysis of the characteristics of the charging sessions in each behavioral cluster and investigation of the influence of weekdays and seasonal changes on those characteristics including arrival, sojourn and idle times, and (3) proposing measures and an algorithm to quantitatively analyze how much flexibility (in terms of duration and amount) is used at various times of a day, for two representative scenarios. Understanding the characteristics of that flexibility (e.g., amount, time and duration of availability) and when it is used (in terms of both duration and amount) helps to develop more realistic price and incentive schemes in DR algorithms to efficiently exploit the offered flexibility or to estimate when to stimulate additional flexibility. (C) 2017 Elsevier Ltd. All rights reserved

Reduced state space and cost function in reinforcement learning for demand response control of multiple EV charging stations

Electric vehicle (EV) charging stations represent a substantial load with significant flexibility. Balancing such load with model-free demand response (DR) based on reinforcement learning (RL) is an attractive approach. We build on previous RL research using a Markov decision process (MDP) to simultaneously coordinate multiple charging stations. The previously proposed approach is computationally expensive in terms of large training times, limiting its feasibility and practicality. We propose to a priori force the control policy to always fulfill any charging demand that does not offer any flexibility at a given point, and thus use an updated cost function. We compare the policy of the newly proposed approach with the original (costly) one, for the case of load flattening, in terms of (i) processing time to learn the RL-based charging policy, and (ii) overall performance of the policy decisions in terms of meeting the target load for unseen test data

Quantifying flexibility in EV charging as DR potential : analysis of two real-world data sets

The increasing adoption of electric vehicles (EVs) presents both challenges and opportunities for the power grid, especially for distribution system operators (DSOs). The demand represented by EVs can be significant, but on the other hand, sojourn times of EVs could be longer than the time required to charge their batteries to the desired level (e.g., to cover the next trip). The latter observation means that the electrical load from EVs is characterized by a certain level of flexibility, which could be exploited for example in demand response (DR) approaches (e.g., to balance generation from renewable energy sources). This paper analyzes two data sets, one from a charging-at-home field trial in Flanders (about 8.5k charging sessions) and another from a large-scale EV public charging pole deployment in The Netherlands (more than 1M sessions). We rigorously analyze the collected data and quantify aforementioned flexibility: (1) we characterize the EV charging behavior by clustering the arrival and departure time combinations, identifying three behaviors (charging near home, charging near work, and park to charge), (2) we fit statistical models for the sojourn time, and flexibility (i.e., non-charging idle time) for each type of observed behavior, and (3) quantify the the potential of DR exploitation as the maximal load that could be achieved by coordinating EV charging for a given time of day t, continuously until t vertical bar Delt

Bayesian cylindrical data modeling using Abe-Ley mixtures

This paper proposes a Metropolis-Hastings algorithm based on Markov chain Monte Carlo sampling, to estimate the parameters of the Abe-Ley distribution, which is a recently proposed Weibull-Sine-Skewed-von Mises mixture model, for bivariate circular-linear data. Current literature estimates the parameters of these mixture models using the expectation-maximization method, but we will show that this exhibits a few shortcomings for the considered mixture model. First, standard expectation-maximization does not guarantee convergence to a global optimum, because the likelihood is multi-modal, which results from the high dimensionality of the mixture's likelihood. Second, given that expectation-maximization provides point estimates of the parameters only, the uncertainties of the estimates (e.g., confidence intervals) are not directly available in these methods. Hence, extra calculations are needed to quantify such uncertainty. We propose a Metropolis-Hastings based algorithm that avoids both shortcomings of expectation-maximization. Indeed, Metropolis-Hastings provides an approximation to the complete (posterior) distribution, given that it samples from the joint posterior of the mixture parameters. This facilitates direct inference (e.g., about uncertainty, multi-modality) from the estimation. In developing the algorithm, we tackle various challenges including convergence speed, label switching and selecting the optimum number of mixture components. We then (i) verify the effectiveness of the proposed algorithm on sample datasets with known true parameters, and further (ii) validate our methodology on an environmental dataset (a traditional application domain of Abe-Ley mixtures where measurements are function of direction). Finally, we (iii) demonstrate the usefulness of our approach in an application domain where the circular measurement is periodic in time. (C) 2018 Elsevier Inc. All rights reserved