Correlated Deep Q-learning based Microgrid Energy Management

Microgrid (MG) energy management is an important part of MG operation. Various entities are generally involved in the energy management of an MG, e.g., energy storage system (ESS), renewable energy resources (RER) and the load of users, and it is crucial to coordinate these entities. Considering the significant potential of machine learning techniques, this paper proposes a correlated deep Q-learning (CDQN) based technique for the MG energy management. Each electrical entity is modeled as an agent which has a neural network to predict its own Q-values, after which the correlated Q-equilibrium is used to coordinate the operation among agents. In this paper, the Long Short Term Memory networks (LSTM) based deep Q-learning algorithm is introduced and the correlated equilibrium is proposed to coordinate agents. The simulation result shows 40.9% and 9.62% higher profit for ESS agent and photovoltaic (PV) agent, respectively.Comment: Accepted by 2020 IEEE 25th International Workshop on CAMAD, 978-1-7281-6339-0/20/$31.00 \copyright 2020 IEE

Decentralized Microgrid Energy Management: A Multi-agent Correlated Q-learning Approach

Microgrids (MG) are anticipated to be important players in the future smart grid. For proper operation of MGs an Energy Management System (EMS) is essential. The EMS of an MG could be rather complicated when renewable energy resources (RER), energy storage system (ESS) and demand side management (DSM) need to be orchestrated. Furthermore, these systems may belong to different entities and competition may exist between them. Nash equilibrium is most commonly used for coordination of such entities however the convergence and existence of Nash equilibrium can not always be guaranteed. To this end, we use the correlated equilibrium to coordinate agents, whose convergence can be guaranteed. In this paper, we build an energy trading model based on mid-market rate, and propose a correlated Q-learning (CEQ) algorithm to maximize the revenue of each agent. Our results show that CEQ is able to balance the revenue of agents without harming total benefit. In addition, compared with Q-learning without correlation, CEQ could save 19.3% cost for the DSM agent and 44.2% more benefits for the ESS agent.Comment: Accepted by 2020 IEEE International Conference on SmartGridComm, 978-1-7281-6127-3/20/$31.00 copyright 2020 IEE

Peer-to-Peer Energy Trading in Smart Residential Environment with User Behavioral Modeling

Electric power systems are transforming from a centralized unidirectional market to a decentralized open market. With this shift, the end-users have the possibility to actively participate in local energy exchanges, with or without the involvement of the main grid. Rapidly reducing prices for Renewable Energy Technologies (RETs), supported by their ease of installation and operation, with the facilitation of Electric Vehicles (EV) and Smart Grid (SG) technologies to make bidirectional flow of energy possible, has contributed to this changing landscape in the distribution side of the traditional power grid. Trading energy among users in a decentralized fashion has been referred to as Peer- to-Peer (P2P) Energy Trading, which has attracted significant attention from the research and industry communities in recent times. However, previous research has mostly focused on engineering aspects of P2P energy trading systems, often neglecting the central role of users in such systems. P2P trading mechanisms require active participation from users to decide factors such as selling prices, storing versus trading energy, and selection of energy sources among others. The complexity of these tasks, paired with the limited cognitive and time capabilities of human users, can result sub-optimal decisions or even abandonment of such systems if performance is not satisfactory. Therefore, it is of paramount importance for P2P energy trading systems to incorporate user behavioral modeling that captures users’ individual trading behaviors, preferences, and perceived utility in a realistic and accurate manner. Often, such user behavioral models are not known a priori in real-world settings, and therefore need to be learned online as the P2P system is operating. In this thesis, we design novel algorithms for P2P energy trading. By exploiting a variety of statistical, algorithmic, machine learning, and behavioral economics tools, we propose solutions that are able to jointly optimize the system performance while taking into account and learning realistic model of user behavior. The results in this dissertation has been published in IEEE Transactions on Green Communications and Networking 2021, Proceedings of IEEE Global Communication Conference 2022, Proceedings of IEEE Conference on Pervasive Computing and Communications 2023 and ACM Transactions on Evolutionary Learning and Optimization 2023

Energy Sharing Models for Renewable Energy Integration: Subtransmission Level, Distribution Level, and Community Level

Distributed energy resources (DERs) are being embedded rapidly and widely in the power grid and promoting the transformation of the centralized power industry to a more deregulated mode. However, how to safely and efficiently consume renewable energy is becoming a major concern. In this regard, energy sharing at both grid-scale and community-scale has emerged as a new solution to encourage participants to actively bid instead of acting as price takers and has the potential to accelerate the integration of DERs and decrease energy costs. At the grid level, two risk-averse energy sharing models are developed to safely integrate renewable energy by considering the network constraints and overbidding risk. A risk-averse two-stage stochastic game model is proposed for the regional energy sharing market (ESM). The sample average approximation (SAA) method is used to approximate the stochastic Cournot-Nash equilibrium. In addition, a data-driven joint chance-constrained game is developed for energy sharing in the local energy market (LEM). This model considers the maximum outputs of renewable energy aggregators (REAs) are random variables whose probability distributions are unknown, but the decision-maker has access to finite samples. Case studies show that the proposed game models can effectively increase the profit of reliable players and decrease the overbidding risk. At the community level, a community server enables energy sharing among users based on the Bayesian game-based pricing mechanism. It can also control the community energy storage system (CESS) to smooth the load based on the grid's price signal. A communication-censored ADMM for sharing problems is developed to decrease the communication cost between the community and the grid. Moreover, a co-optimization model for the plan and operation of the shared CESS is developed. By introducing the price uncertainty and degradation cost, the proposed model could more accurately evaluate the performance of the CESS and tap more economic potential. This thesis provides proof of the Nash equilibrium of all game models and the convergence of all market clearing algorithms. The proposed models and methods present performance improvement compared with existing solutions. The work in this thesis indicates that energy sharing is possible to implement at different levels of the power system and could benefit the participants and promote the integration of DERs

Coalitional model predictive control for systems of systems

An aspect so far rarely contemplated in distributed control problems is the explicit consideration of individual (local) interests of the components of a complex system. Indeed, the focus of the majority of the literature about distributed control has been the overall system performance. While on one hand this permitted to address fundamental properties of centralized control, such as system-wide optimality and stability, one the other hand it implied assuming unrestricted cooperation across local controllers. However, when dealing with multi-agent systems with a strong heterogeneous character, cooperation between the agents cannot be taken for granted (due to, for example, logistics, market competition), and selfish interests may not be neglected. Another critical point that must be kept into consideration is the diversity characterizing systems of systems (SoS), yielding very complex interactions between the agents involved (one example of such system is the smart grid). In order to tackle such inherent aspects of SoS, the research presented in this thesis has been concerned with the development of a novel framework, the coalitional control, that extends the scope of advanced control methods (in particular MPC) by drawing concepts from cooperative game theory that are suited for the inherent heterogeneity of SoS, providing as well an economical interpretation useful to explicitly take into account local selfish interests. Thus, coalitional control aims at governing the association/dissociation dynamics of the agents controlling the system, according to the expected benefits of their possible cooperation. From a control theoretical perspective, this framework is founded on the theory of switched systems and variable structure/topology networked systems, topics that are recently experiencing a renewed interest within the community. The main concepts and challenges in coalitional control, and the links with cooperative network game theory are presented in this document, tracing a path from model partitioning to the control schemes whose principles delineate the idea of coalitional control. This thesis focuses on two basic architectures: (i) a hierarchically supervised evolution of the coalitional structure, and (ii) a protocol for autonomous negotiation between the agents, with specific mechanisms for benefit redistribution, leading to the emergence of cooperating clusters.Premio Extraordinario de Doctorado U

Data-Intensive Computing in Smart Microgrids

Microgrids have recently emerged as the building block of a smart grid, combining distributed renewable energy sources, energy storage devices, and load management in order to improve power system reliability, enhance sustainable development, and reduce carbon emissions. At the same time, rapid advancements in sensor and metering technologies, wireless and network communication, as well as cloud and fog computing are leading to the collection and accumulation of large amounts of data (e.g., device status data, energy generation data, consumption data). The application of big data analysis techniques (e.g., forecasting, classification, clustering) on such data can optimize the power generation and operation in real time by accurately predicting electricity demands, discovering electricity consumption patterns, and developing dynamic pricing mechanisms. An efficient and intelligent analysis of the data will enable smart microgrids to detect and recover from failures quickly, respond to electricity demand swiftly, supply more reliable and economical energy, and enable customers to have more control over their energy use. Overall, data-intensive analytics can provide effective and efficient decision support for all of the producers, operators, customers, and regulators in smart microgrids, in order to achieve holistic smart energy management, including energy generation, transmission, distribution, and demand-side management. This book contains an assortment of relevant novel research contributions that provide real-world applications of data-intensive analytics in smart grids and contribute to the dissemination of new ideas in this area

Dynamic Incentives for Optimal Control of Competitive Power Systems

Technologisch herausfordernde Transformationsprozesse wie die Energiewende können durch passende Anreizsysteme entscheidend beschleunigt werden. Ziel solcher Anreize ist es hierbei, ein Umfeld idealerweise so zu schaffen, dass das Zusammenspiel aller aus Sicht der beteiligten Wettbewerber individuell optimalen Einzelhandlungen auch global optimal im Sinne eines übergeordneten Großziels ist. Die vorliegende Dissertation schafft einen regelungstechnischen Zugang zur Frage optimaler Anreizsysteme für heutige und zukünftige Stromnetze im Zieldreieck aus Systemstabilität, ökonomischer Effizienz und Netzdienlichkeit. Entscheidende Neuheit des entwickelten Ansatzes ist die Einführung zeitlich wie örtlich differenzierter Echtzeit-Preissignale, die sich aus der Lösung statischer und dynamischer Optimierungsprobleme ergeben. Der Miteinbezug lokal verfügbarer Messinformationen, die konsequente Mitmodellierung des unterlagerten physikalischen Netzes inklusive resistiver Verluste und die durchgängig zeitkontinuierliche Formulierung aller Teilsysteme ebnen den Weg von einer reinen Anreiz-Steuerung hin zu einer echten Anreiz-Regelung. Besonderes Augenmerk der Arbeit liegt in einer durch das allgemeine Unbundling-Gebot bedingten rigorosen Trennung zwischen Markt- und Netzakteuren. Nach umfangreicher Analyse des hierbei entstehenden geschlossenen Regelkreises erfolgt die beispielhafte Anwendung der Regelungsarchitektur für den Aufbau eines neuartigen Echtzeit-Engpassmanagementsystems. Weitere praktische Vorteile des entwickelten Ansatzes im Vergleich zu bestehenden Konzepten werden anhand zweier Fallstudien deutlich. Die port-basierte Systemmodellierung, der Verzicht auf zentralisierte Regeleingriffe und nicht zuletzt die Möglichkeit zur automatischen, dezentralen Selbstregulation aller Preise über das Gesamtnetz hinweg stellen schließlich die problemlose Erweiterbarkeit um zusätzliche optionale Anreizkomponenten sicher

Dynamic Incentives for Optimal Control of Competitive Power Systems

This work presents a real-time dynamic pricing framework for future electricity markets. Deduced by first-principles analysis of physical, economic, and communication constraints within the power system, the proposed feedback control mechanism ensures both closed-loop system stability and economic efficiency at any given time. The resulting price signals are able to incentivize competitive market participants to eliminate spatio-temporal shortages in power supply quickly and purposively