338 research outputs found
Multi-Microgrid Collaborative Optimization Scheduling Using an Improved Multi-Agent Soft Actor-Critic Algorithm
The implementation of a multi-microgrid (MMG) system with multiple renewable
energy sources enables the facilitation of electricity trading. To tackle the
energy management problem of a MMG system, which consists of multiple renewable
energy microgrids belonging to different operating entities, this paper
proposes a MMG collaborative optimization scheduling model based on a
multi-agent centralized training distributed execution framework. To enhance
the generalization ability of dealing with various uncertainties, we also
propose an improved multi-agent soft actor-critic (MASAC) algorithm, which
facilitates en-ergy transactions between multi-agents in MMG, and employs
automated machine learning (AutoML) to optimize the MASAC hyperparameters to
further improve the generalization of deep reinforcement learning (DRL). The
test results demonstrate that the proposed method successfully achieves power
complementarity between different entities, and reduces the MMG system
operating cost. Additionally, the proposal significantly outperforms other
state-of-the-art reinforcement learning algorithms with better economy and
higher calculation efficiency.Comment: Accepted by Energie
Optimal energy management for a grid-tied solar PV-battery microgrid: A reinforcement learning approach
There has been a shift towards energy sustainability in recent years, and this shift should continue. The steady growth of energy demand because of population growth, as well as heightened worries about the number of anthropogenic gases released into the atmosphere and deployment of advanced grid technologies, has spurred the penetration of renewable energy resources (RERs) at different locations and scales in the power grid. As a result, the energy system is moving away from the centralized paradigm of large, controllable power plants and toward a decentralized network based on renewables. Microgrids, either grid-connected or islanded, provide a key solution for integrating RERs, load demand flexibility, and energy storage systems within this framework. Nonetheless, renewable energy resources, such as solar and wind energy, can be extremely stochastic as they are weather dependent. These resources coupled with load demand uncertainties lead to random variations on both the generation and load sides, thus challenging optimal energy management. This thesis develops an optimal energy management system (EMS) for a grid-tied solar PV-battery microgrid. The goal of the EMS is to obtain the minimum operational costs (cost of power exchange with the utility and battery wear cost) while still considering network constraints, which ensure grid violations are avoided. A reinforcement learning (RL) approach is proposed to minimize the operational cost of the microgrid under this stochastic setting. RL is a reward-motivated optimization technique derived from how animals learn to optimize their behaviour in new environments. Unlike other conventional model-based optimization approaches, RL doesn't need an explicit model of the optimization system to get optimal solutions. The EMS is modelled as a Markov Decision Process (MDP) to achieve optimality considering the state, action, and reward function. The feasibility of two RL algorithms, namely, conventional Q-learning algorithm and deep Q network algorithm, are developed, and their efficacy in performing optimal energy management for the designed system is evaluated in this thesis. First, the energy management problem is expressed as a sequential decision-making process, after which two algorithms, trading, and non-trading algorithm, are developed. In the trading algorithm case, excess microgrid's energy can be sold back to the utility to increase revenue, while in the latter case constraining rules are embedded in the designed EMS to ensure that no excess energy is sold back to the utility. Then a Q-learning algorithm is developed to minimize the operational cost of the microgrid under unknown future information. Finally, to evaluate the performance of the proposed EMS, a comparison study between a trading case EMS model and a non-trading case is performed using a typical commercial load curve and PV generation profile over a 24- hour horizon. Numerical simulation results indicated that the algorithm learned to select an optimized energy schedule that minimizes energy cost (cost of power purchased from the utility based on the time-varying tariff and battery wear cost) in both summer and winter case studies. However, comparing the non-trading EMS to the trading EMS model operational costs, the latter one decreased cost by 4.033% in the summer season and 2.199% in the winter season. Secondly, a deep Q network (DQN) method that uses recent learning algorithm enhancements, including experience replay and target network, is developed to learn the system uncertainties, including load demand, grid prices and volatile power supply from the renewables solve the optimal energy management problem. Unlike the Q-learning method, which updates the Q-function using a lookup table (which limits its scalability and overall performance in stochastic optimization), the DQN method uses a deep neural network that approximates the Q- function via statistical regression. The performance of the proposed method is evaluated with differently fluctuating load profiles, i.e., slow, medium, and fast. Simulation results substantiated the efficacy of the proposed method as the algorithm was established to learn from experience to raise the battery state of charge and optimally shift loads from a one-time instance, thus supporting the utility grid in reducing aggregate peak load. Furthermore, the performance of the proposed DQN approach was compared to the conventional Q-learning algorithm in terms of achieving a minimum global cost. Simulation results showed that the DQN algorithm outperformed the conventional Q-learning approach, reducing system operational costs by 15%, 24%, and 26% for the slow, medium, and fast fluctuating load profiles in the studied cases
Reinforcement Learning Based Cooperative P2P Energy Trading between DC Nanogrid Clusters with Wind and PV Energy Resources
In order to replace fossil fuels with the use of renewable energy resources,
unbalanced resource production of intermittent wind and photovoltaic (PV) power
is a critical issue for peer-to-peer (P2P) power trading. To resolve this
problem, a reinforcement learning (RL) technique is introduced in this paper.
For RL, graph convolutional network (GCN) and bi-directional long short-term
memory (Bi-LSTM) network are jointly applied to P2P power trading between
nanogrid clusters based on cooperative game theory. The flexible and reliable
DC nanogrid is suitable to integrate renewable energy for distribution system.
Each local nanogrid cluster takes the position of prosumer, focusing on power
production and consumption simultaneously. For the power management of nanogrid
clusters, multi-objective optimization is applied to each local nanogrid
cluster with the Internet of Things (IoT) technology. Charging/discharging of
electric vehicle (EV) is performed considering the intermittent characteristics
of wind and PV power production. RL algorithms, such as deep Q-learning network
(DQN), deep recurrent Q-learning network (DRQN), Bi-DRQN, proximal policy
optimization (PPO), GCN-DQN, GCN-DRQN, GCN-Bi-DRQN, and GCN-PPO, are used for
simulations. Consequently, the cooperative P2P power trading system maximizes
the profit utilizing the time of use (ToU) tariff-based electricity cost and
system marginal price (SMP), and minimizes the amount of grid power
consumption. Power management of nanogrid clusters with P2P power trading is
simulated on the distribution test feeder in real-time and proposed GCN-PPO
technique reduces the electricity cost of nanogrid clusters by 36.7%.Comment: 22 pages, 8 figures, to be submitted to Applied Energy of Elsevie
Optimal Energy Management of a Grid-Tied Solar PV-Battery Microgrid: A Reinforcement Learning Approach
In the near future, microgrids will become more prevalent as they play a critical role in integrating distributed renewable energy resources into the main grid. Nevertheless, renewable energy sources, such as solar and wind energy can be extremely volatile as they are weather dependent. These resources coupled with demand can lead to random variations on both the generation and load sides, thus complicating optimal energy management. In this article, a reinforcement learning approach has been proposed to deal with this non-stationary scenario, in which the energy management system (EMS) is modelled as a Markov decision process (MDP). A novel modification of the control problem has been presented that improves the use of energy stored in the battery such that the dynamic demand is not subjected to future high grid tariffs. A comprehensive reward function has also been developed which decreases infeasible action explorations thus improving the performance of the data-driven technique. A Q-learning algorithm is then proposed to minimize the operational cost of the microgrid under unknown future information. To assess the performance of the proposed EMS, a comparison study between a trading EMS model and a non-trading case is performed using a typical commercial load curve and PV profile over a 24-h horizon. Numerical simulation results indicate that the agent learns to select an optimized energy schedule that minimizes energy cost (cost of power purchased from the utility and battery wear cost) in all the studied cases. However, comparing the non-trading EMS to the trading EMS model operational costs, the latter one was found to decrease costs by 4.033% in summer season and 2.199% in winter season
- …