Decentralized Microgrid Energy Management: A Multi-agent Correlated Q-learning Approach

Microgrids (MG) are anticipated to be important players in the future smart grid. For proper operation of MGs an Energy Management System (EMS) is essential. The EMS of an MG could be rather complicated when renewable energy resources (RER), energy storage system (ESS) and demand side management (DSM) need to be orchestrated. Furthermore, these systems may belong to different entities and competition may exist between them. Nash equilibrium is most commonly used for coordination of such entities however the convergence and existence of Nash equilibrium can not always be guaranteed. To this end, we use the correlated equilibrium to coordinate agents, whose convergence can be guaranteed. In this paper, we build an energy trading model based on mid-market rate, and propose a correlated Q-learning (CEQ) algorithm to maximize the revenue of each agent. Our results show that CEQ is able to balance the revenue of agents without harming total benefit. In addition, compared with Q-learning without correlation, CEQ could save 19.3% cost for the DSM agent and 44.2% more benefits for the ESS agent.Comment: Accepted by 2020 IEEE International Conference on SmartGridComm, 978-1-7281-6127-3/20/$31.00 copyright 2020 IEE

Reinforcement Learning Based Cooperative P2P Energy Trading between DC Nanogrid Clusters with Wind and PV Energy Resources

In order to replace fossil fuels with the use of renewable energy resources, unbalanced resource production of intermittent wind and photovoltaic (PV) power is a critical issue for peer-to-peer (P2P) power trading. To resolve this problem, a reinforcement learning (RL) technique is introduced in this paper. For RL, graph convolutional network (GCN) and bi-directional long short-term memory (Bi-LSTM) network are jointly applied to P2P power trading between nanogrid clusters based on cooperative game theory. The flexible and reliable DC nanogrid is suitable to integrate renewable energy for distribution system. Each local nanogrid cluster takes the position of prosumer, focusing on power production and consumption simultaneously. For the power management of nanogrid clusters, multi-objective optimization is applied to each local nanogrid cluster with the Internet of Things (IoT) technology. Charging/discharging of electric vehicle (EV) is performed considering the intermittent characteristics of wind and PV power production. RL algorithms, such as deep Q-learning network (DQN), deep recurrent Q-learning network (DRQN), Bi-DRQN, proximal policy optimization (PPO), GCN-DQN, GCN-DRQN, GCN-Bi-DRQN, and GCN-PPO, are used for simulations. Consequently, the cooperative P2P power trading system maximizes the profit utilizing the time of use (ToU) tariff-based electricity cost and system marginal price (SMP), and minimizes the amount of grid power consumption. Power management of nanogrid clusters with P2P power trading is simulated on the distribution test feeder in real-time and proposed GCN-PPO technique reduces the electricity cost of nanogrid clusters by 36.7%.Comment: 22 pages, 8 figures, to be submitted to Applied Energy of Elsevie

Towards Agent-Based Model Specification of Smart Grid: A Cognitive Agent-Based Computing Approach

A smart grid can be considered as a complex network where each node represents a generation unit or a consumer, whereas links can be used to represent transmission lines. One way to study complex systems is by using the agent-based modeling paradigm. The agent-based modeling is a way of representing a complex system of autonomous agents interacting with each other. Previously, a number of studies have been presented in the smart grid domain making use of the agent-based modeling paradigm. However, to the best of our knowledge, none of these studies have focused on the specification aspect of the model. The model specification is important not only for understanding but also for replication of the model. To fill this gap, this study focuses on specification methods for smart grid modeling. We adopt two specification methods named as Overview, design concept, and details and Descriptive agent-based modeling. By using specification methods, we provide tutorials and guidelines for model developing of smart grid starting from conceptual modeling to validated agent-based model through simulation. The specification study is exemplified through a case study from the smart grid domain. In the case study, we consider a large set of network, in which different consumers and power generation units are connected with each other through different configuration. In such a network, communication takes place between consumers and generating units for energy transmission and data routing. We demonstrate how to effectively model a complex system such as a smart grid using specification methods. We analyze these two specification approaches qualitatively as well as quantitatively. Extensive experiments demonstrate that Descriptive agent-based modeling is a more useful approach as compared with Overview, design concept, and details method for modeling as well as for replication of models for the smart grid

Dynamic Programming based approach for Energy Trading

Bi-directional Energy Trading is going to play an essential role in facilitating the increased usage of distributed renewable energy sources. The smooth transition towards these clean sources of energy would require opening up of the energy markets to allow for a two-way electricity trade. The study proposes a dynamic programming based energy trading framework (called Dynamic Battery Charging (DBC) Algorithm) from the end-user perspective. Using the proposed energy transfer model the framework finds out the optimal battery charge state at the consumer end. To further improve the performance of the framework, the original DBC algorithm is clubbed together with a capacity fading based battery cost model. For testing and validation purpose, a case study of three different load profiles (different in scale) in three energy markets is done. The simulation results show the profitability of the proposed strategy in all the tested scenarios