Optimal energy management for a grid-tied solar PV-battery microgrid: A reinforcement learning approach

There has been a shift towards energy sustainability in recent years, and this shift should continue. The steady growth of energy demand because of population growth, as well as heightened worries about the number of anthropogenic gases released into the atmosphere and deployment of advanced grid technologies, has spurred the penetration of renewable energy resources (RERs) at different locations and scales in the power grid. As a result, the energy system is moving away from the centralized paradigm of large, controllable power plants and toward a decentralized network based on renewables. Microgrids, either grid-connected or islanded, provide a key solution for integrating RERs, load demand flexibility, and energy storage systems within this framework. Nonetheless, renewable energy resources, such as solar and wind energy, can be extremely stochastic as they are weather dependent. These resources coupled with load demand uncertainties lead to random variations on both the generation and load sides, thus challenging optimal energy management. This thesis develops an optimal energy management system (EMS) for a grid-tied solar PV-battery microgrid. The goal of the EMS is to obtain the minimum operational costs (cost of power exchange with the utility and battery wear cost) while still considering network constraints, which ensure grid violations are avoided. A reinforcement learning (RL) approach is proposed to minimize the operational cost of the microgrid under this stochastic setting. RL is a reward-motivated optimization technique derived from how animals learn to optimize their behaviour in new environments. Unlike other conventional model-based optimization approaches, RL doesn't need an explicit model of the optimization system to get optimal solutions. The EMS is modelled as a Markov Decision Process (MDP) to achieve optimality considering the state, action, and reward function. The feasibility of two RL algorithms, namely, conventional Q-learning algorithm and deep Q network algorithm, are developed, and their efficacy in performing optimal energy management for the designed system is evaluated in this thesis. First, the energy management problem is expressed as a sequential decision-making process, after which two algorithms, trading, and non-trading algorithm, are developed. In the trading algorithm case, excess microgrid's energy can be sold back to the utility to increase revenue, while in the latter case constraining rules are embedded in the designed EMS to ensure that no excess energy is sold back to the utility. Then a Q-learning algorithm is developed to minimize the operational cost of the microgrid under unknown future information. Finally, to evaluate the performance of the proposed EMS, a comparison study between a trading case EMS model and a non-trading case is performed using a typical commercial load curve and PV generation profile over a 24- hour horizon. Numerical simulation results indicated that the algorithm learned to select an optimized energy schedule that minimizes energy cost (cost of power purchased from the utility based on the time-varying tariff and battery wear cost) in both summer and winter case studies. However, comparing the non-trading EMS to the trading EMS model operational costs, the latter one decreased cost by 4.033% in the summer season and 2.199% in the winter season. Secondly, a deep Q network (DQN) method that uses recent learning algorithm enhancements, including experience replay and target network, is developed to learn the system uncertainties, including load demand, grid prices and volatile power supply from the renewables solve the optimal energy management problem. Unlike the Q-learning method, which updates the Q-function using a lookup table (which limits its scalability and overall performance in stochastic optimization), the DQN method uses a deep neural network that approximates the Q- function via statistical regression. The performance of the proposed method is evaluated with differently fluctuating load profiles, i.e., slow, medium, and fast. Simulation results substantiated the efficacy of the proposed method as the algorithm was established to learn from experience to raise the battery state of charge and optimally shift loads from a one-time instance, thus supporting the utility grid in reducing aggregate peak load. Furthermore, the performance of the proposed DQN approach was compared to the conventional Q-learning algorithm in terms of achieving a minimum global cost. Simulation results showed that the DQN algorithm outperformed the conventional Q-learning approach, reducing system operational costs by 15%, 24%, and 26% for the slow, medium, and fast fluctuating load profiles in the studied cases

Model-free non-invasive health assessment for battery energy storage assets

Increasing penetration of renewable energy generation in the modern power network introduces uncertainty about the energy available to maintain a balance between generation and demand due to its time-fluctuating output that is strongly dependent on the weather. With the development of energy storage technology, there is the potential for this technology to become a key element to help overcome this intermittency in a generation. However, the increasing penetration of battery energy storage within the power network introduces an additional challenge to asset owners on how to monitor and manage battery health. The accurate estimation of the health of this device is crucial in determining its reliability, power-delivering capability and ability to contribute to the operation of the whole power system. Generally, doing this requires invasive measurements or computationally expensive physics-based models, which do not scale up cost-effectively to a fleet of assets. As storage aggregation becomes more commonplace, there is a need for a health metric that will be able to predict battery health based only on the limited information available, eliminating the necessity of installation of extensive telemetry in the system. This work develops a solution to battery health prognostics by providing an alternative, a non-invasive approach to the estimation of battery health that estimates the extent to which a battery asset has been maloperated based only on the battery-operating regime imposed on the device. The model introduced in this work is based on the Hidden Markov Model, which stochastically models the battery limitations imposed by its chemistry as a combination of present and previous sequential charging actions, and articulates the preferred operating regime as a measure of health consequence. The resulting methodology is demonstrated on distribution network level electrical demand and generation data, accurately predicting maloperation under a number of battery technology scenarios. The effectiveness of the proposed battery maloperation model as a proxy for actual battery degradation for lithium-ion technology was also tested against lab tested battery degradation data, showing that the proposed health measure in terms of maloperation level reflected that measured in terms of capacity fade. The developed model can support condition monitoring and remaining useful life estimates, but in the wider context could also be used as the policy function in an automated scheduler to utilise assets while optimising their health.Increasing penetration of renewable energy generation in the modern power network introduces uncertainty about the energy available to maintain a balance between generation and demand due to its time-fluctuating output that is strongly dependent on the weather. With the development of energy storage technology, there is the potential for this technology to become a key element to help overcome this intermittency in a generation. However, the increasing penetration of battery energy storage within the power network introduces an additional challenge to asset owners on how to monitor and manage battery health. The accurate estimation of the health of this device is crucial in determining its reliability, power-delivering capability and ability to contribute to the operation of the whole power system. Generally, doing this requires invasive measurements or computationally expensive physics-based models, which do not scale up cost-effectively to a fleet of assets. As storage aggregation becomes more commonplace, there is a need for a health metric that will be able to predict battery health based only on the limited information available, eliminating the necessity of installation of extensive telemetry in the system. This work develops a solution to battery health prognostics by providing an alternative, a non-invasive approach to the estimation of battery health that estimates the extent to which a battery asset has been maloperated based only on the battery-operating regime imposed on the device. The model introduced in this work is based on the Hidden Markov Model, which stochastically models the battery limitations imposed by its chemistry as a combination of present and previous sequential charging actions, and articulates the preferred operating regime as a measure of health consequence. The resulting methodology is demonstrated on distribution network level electrical demand and generation data, accurately predicting maloperation under a number of battery technology scenarios. The effectiveness of the proposed battery maloperation model as a proxy for actual battery degradation for lithium-ion technology was also tested against lab tested battery degradation data, showing that the proposed health measure in terms of maloperation level reflected that measured in terms of capacity fade. The developed model can support condition monitoring and remaining useful life estimates, but in the wider context could also be used as the policy function in an automated scheduler to utilise assets while optimising their health

Energy storage systems and grid code requirements for large-scale renewables integration in insular grids

This thesis addresses the topic of energy storage systems supporting increased penetration of renewables in insular systems. An overview of energy storage management, forecasting tools and demand side solutions is carried out, comparing the strategic utilization of storage and other competing strategies. Particular emphasis is given to energy storage systems on islands, as a new contribution to earlier studies, addressing their particular requirements, the most appropriate technologies and existing operating projects throughout the world. Several real-world case studies are presented and discussed in detail. Lead-acid battery design parameters are assessed for energy storage applications on insular grids, comparing different battery models. The wind curtailment mitigation effect by means of energy storage resources is also explored. Grid code requirements for large-scale integration of renewables are discussed in an island context, as another new contribution to earlier studies. The current trends on grid code formulation, towards an improved integration of distributed renewable resources in island systems, are addressed. Finally, modeling and control strategies with energy storage systems are addressed. An innovative energy management technique to be used in the day-ahead scheduling of insular systems with Vanadium Redox Flow battery is presented.Esta tese aborda a temática dos sistemas de armazenamento de energia visando o aumento da penetração de energias renováveis em sistemas insulares. Uma visão geral é apresentada acerca da gestão do armazenamento de energia, ferramentas de previsão e soluções do lado da procura de energia, comparando a utilização estratégica do armazenamento e outras estratégias concorrentes. É dada ênfase aos sistemas de armazenamento de energia em ilhas, como uma nova contribuição no estado da arte, abordando as suas necessidades específicas, as tecnologias mais adequadas e os projetos existentes e em funcionamento a nível mundial. Vários casos de estudos reais são apresentados e discutidos em detalhe. Parâmetros de projeto de baterias de chumbo-ácido são avaliados para aplicações de armazenamento de energia em redes insulares, comparando diferentes modelos de baterias. O efeito de redução do potencial de desperdício de energia do vento, recorrendo ao armazenamento de energia, também é perscrutado. As especificidades subjacentes aos códigos de rede para a integração em larga escala de energias renováveis são discutidas em contexto insular, sendo outra nova contribuição no estado da arte. As tendências atuais na elaboração de códigos de rede, no sentido de uma melhor integração da geração distribuída renovável em sistemas insulares, são abordadas. Finalmente, é estudada a modelação e as estratégias de controlo com sistemas de armazenamento de energia. Uma metodologia de gestão de energia inovadora é apresentada para a exploração de curto prazo de sistemas insulares com baterias de fluxo Vanádio Redox

Power System Simulation, Control and Optimization

This Special Issue “Power System Simulation, Control and Optimization” offers valuable insights into the most recent research developments in these topics. The analysis, operation, and control of power systems are increasingly complex tasks that require advanced simulation models to analyze and control the effects of transformations concerning electricity grids today: Massive integration of renewable energies, progressive implementation of electric vehicles, development of intelligent networks, and progressive evolution of the applications of artificial intelligence

Integrating project planning objectives as part of sizing battery energy storage systems

Development of Battery Energy Storage System projects and their subsequent installation and connection to electrical grids throughout the world is expected to increase over the coming years. Two key concerns are at the forefront of entities undertaking these installations. The first is determining the optimal BESS size for a given application. The second is whether this optimal BESS size reflects the goals (referred to as planning objectives in this dissertation) set out in the BESS project planning phase. Recognising these two concerns, it is determined that BESS sizing approaches must be fit for purpose, can be used adequality as a planning tool and capable of modelling important planning objectives. The Front-End Planning framework was utilised in this dissertation as a means to assess if existing BESS sizing approaches are suitable for modelling planning objectives as part of BESS project planning. In total, 32 of the most-cited articles from the BESS sizing literature were reviewed for their inclusiveness of scoping elements set out by the Front-End Planning framework. The results of this review showed that existing BESS sizing approaches are lacking in three key planning objectives called Investment Scale, Investment Timing and Dispatch Adaptability. This research sought to answer the following questions: 1) Is it possible to form the planning objectives Investment Scale, Investment Timing and Dispatch Adaptability as part of optimising energy capacity size for new BESS installations seeking maximum profit? 2) Are there any circumstances where the inclusion of the three planning objectives as part of BESS sizing helps overcome shortcomings of existing sizing approaches? To incorporate the planning objective Investment Scale as part of BESS sizing, maximisation of opposing financial objective functions using two different multi-objective optimisation methods called Rating Method and Paired Comparison was used. These approaches were tested on a simple microgrid under various electricity price scenarios. The results show that the Rating Method performed best when selecting BESS size in significant knee regions near maximum daily worth. The Rating Method can also select optimal BESS size at maximum daily worth when less-significant knee regions are present. This approach gives an appropriate balance between forming the planning objective Investment Scale and maximising profit. To incorporate the planning objective Investment Timing as part of BESS sizing, two different models were used, referred to as the operational model (controlling operational decisions i.e. BESS dispatch) and the planning model (controlling BESS size at different yearly intervals). Reinforcement learning was used as the operational model solution method, while global optimisation was used as the solution method for planning model. This approach was tested on data from the Integrated Single Electricity Market Day-Ahead Market. It was found that splitting BESS operational decisions and BESS planning decisions into two different models is an effective technique. To incorporate the planning objective Dispatch Adaptability as part of BESS sizing, model-based and model-free stochastic optimisation methods are used. This was done for model-free optimisation by utilising deep reinforcement learning methods, while stochastic programming was used as the solution method for model-based approach. Both approaches were tested on historical Day-Ahead and Intraday Markets electricity clearing prices from the Integrated Single Electricity Market. It was found that the model-based approach outperformed the model-free approach. However, it is not clear that such a broad statement can be made about model-free and model-based approaches in general based on the results gained through this thesis. The significance of this study’s results is that BESS sizing is now more functional and adaptable for project planning purposes. It is now possible to size BESS without suffering scale issues resulting from ever-diminishing returns of larger BESS sizes, where the timing of the investment can be chosen optimally rather than assuming “here and now” investment, and where the operational strategy employed to simulate BESS dispatch is more reflective of actual BESS use and adaptability

Energy Management

Forecasts point to a huge increase in energy demand over the next 25 years, with a direct and immediate impact on the exhaustion of fossil fuels, the increase in pollution levels and the global warming that will have significant consequences for all sectors of society. Irrespective of the likelihood of these predictions or what researchers in different scientific disciplines may believe or publicly say about how critical the energy situation may be on a world level, it is without doubt one of the great debates that has stirred up public interest in modern times. We should probably already be thinking about the design of a worldwide strategic plan for energy management across the planet. It would include measures to raise awareness, educate the different actors involved, develop policies, provide resources, prioritise actions and establish contingency plans. This process is complex and depends on political, social, economic and technological factors that are hard to take into account simultaneously. Then, before such a plan is formulated, studies such as those described in this book can serve to illustrate what Information and Communication Technologies have to offer in this sphere and, with luck, to create a reference to encourage investigators in the pursuit of new and better solutions

IEA ECES Annex 31 Final Report - Energy Storage with Energy Efficient Buildings and Districts: Optimization and Automation

At present, the energy requirements in buildings are majorly met from non-renewable sources where the contribution of renewable sources is still in its initial stage. Meeting the peak energy demand by non-renewable energy sources is highly expensive for the utility companies and it critically influences the environment through GHG emissions. In addition, renewable energy sources are inherently intermittent in nature. Therefore, to make both renewable and nonrenewable energy sources more efficient in building/district applications, they should be integrated with energy storage systems. Nevertheless, determination of the optimal operation and integration of energy storage with buildings/districts are not straightforward. The real strength of integrating energy storage technologies with buildings/districts is stalled by the high computational demand (or even lack of) tools and optimization techniques. Annex 31 aims to resolve this gap by critically addressing the challenges in integrating energy storage systems in buildings/districts from the perspective of design, development of simplified modeling tools and optimization techniques

A comprehensive study of key Electric Vehicle (EV) components, technologies, challenges, impacts, and future direction of development

Abstract: Electric vehicles (EV), including Battery Electric Vehicle (BEV), Hybrid Electric Vehicle (HEV), Plug-in Hybrid Electric Vehicle (PHEV), Fuel Cell Electric Vehicle (FCEV), are becoming more commonplace in the transportation sector in recent times. As the present trend suggests, this mode of transport is likely to replace internal combustion engine (ICE) vehicles in the near future. Each of the main EV components has a number of technologies that are currently in use or can become prominent in the future. EVs can cause significant impacts on the environment, power system, and other related sectors. The present power system could face huge instabilities with enough EV penetration, but with proper management and coordination, EVs can be turned into a major contributor to the successful implementation of the smart grid concept. There are possibilities of immense environmental benefits as well, as the EVs can extensively reduce the greenhouse gas emissions produced by the transportation sector. However, there are some major obstacles for EVs to overcome before totally replacing ICE vehicles. This paper is focused on reviewing all the useful data available on EV configurations, battery energy sources, electrical machines, charging techniques, optimization techniques, impacts, trends, and possible directions of future developments. Its objective is to provide an overall picture of the current EV technology and ways of future development to assist in future researches in this sector