Optimal Energy Management of a Grid-Tied Solar PV-Battery Microgrid: A Reinforcement Learning Approach

In the near future, microgrids will become more prevalent as they play a critical role in integrating distributed renewable energy resources into the main grid. Nevertheless, renewable energy sources, such as solar and wind energy can be extremely volatile as they are weather dependent. These resources coupled with demand can lead to random variations on both the generation and load sides, thus complicating optimal energy management. In this article, a reinforcement learning approach has been proposed to deal with this non-stationary scenario, in which the energy management system (EMS) is modelled as a Markov decision process (MDP). A novel modification of the control problem has been presented that improves the use of energy stored in the battery such that the dynamic demand is not subjected to future high grid tariffs. A comprehensive reward function has also been developed which decreases infeasible action explorations thus improving the performance of the data-driven technique. A Q-learning algorithm is then proposed to minimize the operational cost of the microgrid under unknown future information. To assess the performance of the proposed EMS, a comparison study between a trading EMS model and a non-trading case is performed using a typical commercial load curve and PV profile over a 24-h horizon. Numerical simulation results indicate that the agent learns to select an optimized energy schedule that minimizes energy cost (cost of power purchased from the utility and battery wear cost) in all the studied cases. However, comparing the non-trading EMS to the trading EMS model operational costs, the latter one was found to decrease costs by 4.033% in summer season and 2.199% in winter season

Optimal energy management for a grid-tied solar PV-battery microgrid: A reinforcement learning approach

There has been a shift towards energy sustainability in recent years, and this shift should continue. The steady growth of energy demand because of population growth, as well as heightened worries about the number of anthropogenic gases released into the atmosphere and deployment of advanced grid technologies, has spurred the penetration of renewable energy resources (RERs) at different locations and scales in the power grid. As a result, the energy system is moving away from the centralized paradigm of large, controllable power plants and toward a decentralized network based on renewables. Microgrids, either grid-connected or islanded, provide a key solution for integrating RERs, load demand flexibility, and energy storage systems within this framework. Nonetheless, renewable energy resources, such as solar and wind energy, can be extremely stochastic as they are weather dependent. These resources coupled with load demand uncertainties lead to random variations on both the generation and load sides, thus challenging optimal energy management. This thesis develops an optimal energy management system (EMS) for a grid-tied solar PV-battery microgrid. The goal of the EMS is to obtain the minimum operational costs (cost of power exchange with the utility and battery wear cost) while still considering network constraints, which ensure grid violations are avoided. A reinforcement learning (RL) approach is proposed to minimize the operational cost of the microgrid under this stochastic setting. RL is a reward-motivated optimization technique derived from how animals learn to optimize their behaviour in new environments. Unlike other conventional model-based optimization approaches, RL doesn't need an explicit model of the optimization system to get optimal solutions. The EMS is modelled as a Markov Decision Process (MDP) to achieve optimality considering the state, action, and reward function. The feasibility of two RL algorithms, namely, conventional Q-learning algorithm and deep Q network algorithm, are developed, and their efficacy in performing optimal energy management for the designed system is evaluated in this thesis. First, the energy management problem is expressed as a sequential decision-making process, after which two algorithms, trading, and non-trading algorithm, are developed. In the trading algorithm case, excess microgrid's energy can be sold back to the utility to increase revenue, while in the latter case constraining rules are embedded in the designed EMS to ensure that no excess energy is sold back to the utility. Then a Q-learning algorithm is developed to minimize the operational cost of the microgrid under unknown future information. Finally, to evaluate the performance of the proposed EMS, a comparison study between a trading case EMS model and a non-trading case is performed using a typical commercial load curve and PV generation profile over a 24- hour horizon. Numerical simulation results indicated that the algorithm learned to select an optimized energy schedule that minimizes energy cost (cost of power purchased from the utility based on the time-varying tariff and battery wear cost) in both summer and winter case studies. However, comparing the non-trading EMS to the trading EMS model operational costs, the latter one decreased cost by 4.033% in the summer season and 2.199% in the winter season. Secondly, a deep Q network (DQN) method that uses recent learning algorithm enhancements, including experience replay and target network, is developed to learn the system uncertainties, including load demand, grid prices and volatile power supply from the renewables solve the optimal energy management problem. Unlike the Q-learning method, which updates the Q-function using a lookup table (which limits its scalability and overall performance in stochastic optimization), the DQN method uses a deep neural network that approximates the Q- function via statistical regression. The performance of the proposed method is evaluated with differently fluctuating load profiles, i.e., slow, medium, and fast. Simulation results substantiated the efficacy of the proposed method as the algorithm was established to learn from experience to raise the battery state of charge and optimally shift loads from a one-time instance, thus supporting the utility grid in reducing aggregate peak load. Furthermore, the performance of the proposed DQN approach was compared to the conventional Q-learning algorithm in terms of achieving a minimum global cost. Simulation results showed that the DQN algorithm outperformed the conventional Q-learning approach, reducing system operational costs by 15%, 24%, and 26% for the slow, medium, and fast fluctuating load profiles in the studied cases

A systematic review of machine learning techniques related to local energy communities

In recent years, digitalisation has rendered machine learning a key tool for improving processes in several sectors, as in the case of electrical power systems. Machine learning algorithms are data-driven models based on statistical learning theory and employed as a tool to exploit the data generated by the power system and its users. Energy communities are emerging as novel organisations for consumers and prosumers in the distribution grid. These communities may operate differently depending on their objectives and the potential service the community wants to offer to the distribution system operator. This paper presents the conceptualisation of a local energy community on the basis of a review of 25 energy community projects. Furthermore, an extensive literature review of machine learning algorithms for local energy community applications was conducted, and these algorithms were categorised according to forecasting, storage optimisation, energy management systems, power stability and quality, security, and energy transactions. The main algorithms reported in the literature were analysed and classified as supervised, unsupervised, and reinforcement learning algorithms. The findings demonstrate the manner in which supervised learning can provide accurate models for forecasting tasks. Similarly, reinforcement learning presents interesting capabilities in terms of control-related applications.publishedVersio

Reinforcement learning for power scheduling in a grid-tied pv-battery electric vehicles charging station

Grid-tied renewable energy sources (RES) based electric vehicle (EV) charging stations are an example of a distributed generator behind the meter system (DGBMS) which characterizes most modern power infrastructure. To perform power scheduling in such a DGBMS, stochastic variables such as load profile of the charging station, output profile of the RES and tariff profile of the utility must be considered at every decision step. The stochasticity in this kind of optimization environment makes power scheduling a challenging task that deserves substantial research attention. This dissertation investigates the application of reinforcement learning (RL) techniques in solving the power scheduling problem in a grid-tied PV-powered EV charging station with the incorporation of a battery energy storage system. RL is a reward-motivated optimization technique that was derived from the way animals learn to optimize their behavior in a new environment. Unlike other optimization methods such as numerical and soft computing techniques, RL does not require an accurate model of the optimization environment in order to arrive at an optimal solution. This study developed and evaluated the feasibility of two RL algorithms, namely, an asynchronous Q-learning algorithm and an advantage actor-critic (A2C) algorithm, in performing power scheduling in the EV charging station under static conditions. To assess the performances of the proposed algorithms, the conventional Q-learning and actor-critic algorithm were implemented to compare their global cost convergence and their learning characteristics. First, the power scheduling problem was expressed as a sequential decision-making process. Then an asynchronous Q-learning algorithm was developed to solve it. Further, an advantage actor-critic (A2C) algorithm was developed and was used to solve the power scheduling problem. The two algorithms were tested using a 24-hour load, generation and utility grid tariff profiles under static optimization conditions. The performance of the asynchronous Q-learning algorithm was compared with that of the conventional Q-learning method in terms of the global cost, stability and scalability. Likewise, the A2C was compared with the conventional actor-critic method in terms of stability, scalability and convergence. Simulation results showed that both the developed algorithms (asynchronous Q-learning algorithm and A2C) converged to lower global costs and displayed more stable learning characteristics than their conventional counterparts. This research established that proper restriction of the action-space of a Q-learning algorithm improves its stability and convergence. It was also observed that such a restriction may come with compromise of computational speed and scalability. Of the four algorithms analyzed, the A2C was found to produce a power schedule with the lowest global cost and the best usage of the battery energy storage system

Smart Grid Enabling Low Carbon Future Power Systems Towards Prosumers Era

In efforts to meet the targets of carbon emissions reduction in power systems, policy makers formulate measures for facilitating the integration of renewable energy sources and demand side carbon mitigation. Smart grid provides an opportunity for bidirectional communication among policy makers, generators and consumers. With the help of smart meters, increasing number of consumers is able to produce, store, and consume energy, giving them the new role of prosumers. This thesis aims to address how smart grid enables prosumers to be appropriately integrated into energy markets for decarbonising power systems. This thesis firstly proposes a Stackelberg game-theoretic model for dynamic negotiation of policy measures and determining optimal power profiles of generators and consumers in day-ahead market. Simulation results show that the proposed model is capable of saving electricity bills, reducing carbon emissions, and increasing the penetration of renewable energy sources. Secondly, a data-driven prosumer-centric energy scheduling tool is developed by using learning approaches to reduce computational complexity from model-based optimisation. This scheduling tool exploits convolutional neural networks to extract prosumption patterns, and uses scenarios to analyse possible variations of uncertainties caused by the intermittency of renewable energy sources and flexible demand. Case studies confirm that the proposed scheduling tool can accurately predict optimal scheduling decisions under various system scales and uncertain scenarios. Thirdly, a blockchain-based peer-to-peer trading framework is designed to trade energy and carbon allowance. The bidding/selling prices of individual prosumers can directly incentivise the reshaping of prosumption behaviours. Case studies demonstrate the execution of smart contract on the Ethereum blockchain and testify that the proposed trading framework outperforms the centralised trading and aggregator-based trading in terms of regional energy balance and reducing carbon emissions caused by long-distance transmissions

Advanced Warehouse Energy Storage System Control Using Deep Supervised and Reinforcement Learning

The world is undergoing a shift from fossil fuels to renewable energy sources due to the threat of global warming, which has led to a substantial increase in complex buildingintegrated energy systems. These systems increasingly feature local renewable energy production and energy storage systems that require intelligent control algorithms. Traditional approaches, such as rule-based algorithms, are dependent upon timeconsuming human expert design and maintenance to control the energy systems efficiently. Although machine learning has gained increasing amounts of research attention in recent years, its application to energy cost optimization in warehouses still remains in a relatively early stage. Suggested newer approaches are often too complex to implement efficiently, very computationally expensive, or lacking in performance. This Ph.D. thesis explores, designs, and verifies the use of deep learning and reinforcement learning approaches to solve the bottleneck of human expert resource dependency with respect to efficient control of complex building-integrated energy systems. A technologically advanced smart warehouse for food storage and distribution is utilized as acase study. The warehouse has a commercially available Intelligent Energy ManagementSystem (IEMS).publishedVersio