Deep Reinforcement Learning for Distribution Network Operation and Electricity Market

The conventional distribution network and electricity market operation have become challenging under complicated network operating conditions, due to emerging distributed electricity generations, coupled energy networks, and new market behaviours. These challenges include increasing dynamics and stochastics, and vast problem dimensions such as control points, measurements, and multiple objectives, etc. Previously the optimization models were often formulated as conventional programming problems and then solved mathematically, which could now become highly time-consuming or sometimes infeasible. On the other hand, with the recent advancement of artificial intelligence technologies, deep reinforcement learning (DRL) algorithms have demonstrated their excellent performances in various control and optimization fields. This indicates a potential alternative to address these challenges. In this thesis, DRL-based solutions for distribution network operation and electricity market have been investigated and proposed. Firstly, a DRL-based methodology is proposed for Volt/Var Control (VVC) optimization in a large distribution network, to effectively control bus voltages and reduce network power losses. Further, this thesis proposes a multi-agent (MA)DRL-based methodology under a complex regional coordinated VVC framework, and it can address spatial and temporal uncertainties. The DRL algorithm is also improved to adapt to the applications. Then, an integrated energy and heating systems (IEHS) optimization problem is solved by a MADRL-based methodology, where conventionally this could only be solved by simplifications or iterations. Beyond the applications in distribution network operation, a new electricity market service pricing method based on a DRL algorithm is also proposed. This DRL-based method has demonstrated good performance in this virtual storage rental service pricing problem, whereas this bi-level problem could hardly be solved directly due to a non-convex and non-continuous lower-level problem. These proposed methods have demonstrated advantageous performances under comprehensive case studies, and numerical simulation results have validated the effectiveness and high efficiency under different sophisticated operation conditions, solution robustness against temporal and spatial uncertainties, and optimality under large problem dimensions

Decision support for participation in electricity markets considering the transaction of services and electricity at the local level

[EN] The growing concerns regarding the lack of fossil fuels, their costs, and their impact on the environment have led governmental institutions to launch energy policies that promote the increasing installation of technologies that use renewable energy sources to generate energy. The increasing penetration of renewable energy sources brings a great fluctuation on the generation side, which strongly affects the power and energy system management. The control of this system is moving from hierarchical and central to a smart and distributed approach. The system operators are nowadays starting to consider the final end users (consumers and prosumers) as a part of the solution in power system operation activities. In this sense, the end-users are changing their behavior from passive to active players. The role of aggregators is essential in order to empower the end-users, also contributing to those behavior changes. Although in several countries aggregators are legally recognized as an entity of the power and energy system, its role being mainly centered on representing end-users in wholesale market participation. This work contributes to the advancement of the state-of-the-art with models that enable the active involvement of the end-users in electricity markets in order to become key participants in the management of power and energy systems. Aggregators are expected to play an essential role in these models, making the connection between the residential end-users, electricity markets, and network operators. Thus, this work focuses on providing solutions to a wide variety of challenges faced by aggregators. The main results of this work include the developed models to enable consumers and prosumers participation in electricity markets and power and energy systems management. The proposed decision support models consider demand-side management applications, local electricity market models, electricity portfolio management, and local ancillary services. The proposed models are validated through case studies based on real data. The used scenarios allow a comprehensive validation of the models from different perspectives, namely end-users, aggregators, and network operators. The considered case studies were carefully selected to demonstrate the characteristics of each model, and to demonstrate how each of them contributes to answering the research questions defined to this work.[ES] La creciente preocupación por la escasez de combustibles fósiles, sus costos y su impacto en el medio ambiente ha llevado a las instituciones gubernamentales a lanzar políticas energéticas que promuevan la creciente instalación de tecnologías que utilizan fuentes de energía renovables para generar energía. La creciente penetración de las fuentes de energía renovable trae consigo una gran fluctuación en el lado de la generación, lo que afecta fuertemente la gestión del sistema de potencia y energía. El control de este sistema está pasando de un enfoque jerárquico y central a un enfoque inteligente y distribuido. Actualmente, los operadores del sistema están comenzando a considerar a los usuarios finales (consumidores y prosumidores) como parte de la solución en las actividades de operación del sistema eléctrico. En este sentido, los usuarios finales están cambiando su comportamiento de jugadores pasivos a jugadores activos. El papel de los agregadores es esencial para empoderar a los usuarios finales, contribuyendo también a esos cambios de comportamiento. Aunque en varios países los agregadores están legalmente reconocidos como una entidad del sistema eléctrico y energético, su papel se centra principalmente en representar a los usuarios finales en la participación del mercado mayorista. Este trabajo contribuye al avance del estado del arte con modelos que permiten la participación activa de los usuarios finales en los mercados eléctricos para convertirse en participantes clave en la gestión de los sistemas de potencia y energía. Se espera que los agregadores desempeñen un papel esencial en estos modelos, haciendo la conexión entre los usuarios finales residenciales, los mercados de electricidad y los operadores de red. Por lo tanto, este trabajo se enfoca en brindar soluciones a una amplia variedad de desafíos que enfrentan los agregadores. Los principales resultados de este trabajo incluyen los modelos desarrollados para permitir la participación de los consumidores y prosumidores en los mercados eléctricos y la gestión de los sistemas de potencia y energía. Los modelos de soporte de decisiones propuestos consideran aplicaciones de gestión del lado de la demanda, modelos de mercado eléctrico local, gestión de cartera de electricidad y servicios auxiliares locales. Los modelos propuestos son validan mediante estudios de casos basados en datos reales. Los escenarios utilizados permiten una validación integral de los modelos desde diferentes perspectivas, a saber, usuarios finales, agregadores y operadores de red. Los casos de estudio considerados fueron cuidadosamente seleccionados para demostrar las características de cada modelo y demostrar cómo cada uno de ellos contribuye a responder las preguntas de investigación definidas para este trabajo

PV Charging and Storage for Electric Vehicles

Electric vehicles are only ‘green’ as long as the source of electricity is ‘green’ as well. At the same time, renewable power production suffers from diurnal and seasonal variations, creating the need for energy storage technology. Moreover, overloading and voltage problems are expected in the distributed network due to the high penetration of distributed generation and increased power demand from the charging of electric vehicles. The energy and mobility transition hence calls for novel technological innovations in the field of sustainable electric mobility powered from renewable energy. This Special Issue focuses on recent advances in technology for PV charging and storage for electric vehicles

Understanding Deregulated Retail Electricity Markets in the Future: A Perspective from Machine Learning and Optimization

On top of Smart Grid technologies and new market mechanism design, the further deregulation of retail electricity market at distribution level will play a important role in promoting energy system transformation in a socioeconomic way. In today’s retail electricity market, customers have very limited ”energy choice,” or freedom to choose different types of energy services. Although the installation of distributed energy resources (DERs) has become prevalent in many regions, most customers and prosumers who have local energy generation and possible surplus can still only choose to trade with utility companies.They either purchase energy from or sell energy surplus back to the utilities directly while suffering from some price gap. The key to providing more energy trading freedom and open innovation in the retail electricity market is to develop new consumer-centric business models and possibly a localized energy trading platform. This dissertation is exactly pursuing these ideas and proposing a holistic localized electricity retail market to push the next-generation retail electricity market infrastructure to be a level playing field, where all customers have an equal opportunity to actively participate directly. This dissertation also studied and discussed opportunities of many emerging technologies, such as reinforcement learning and deep reinforcement learning, for intelligent energy system operation. Some improvement suggestion of the modeling framework and methodology are included as well.Ph.D.College of Engineering & Computer ScienceUniversity of Michigan-Dearbornhttps://deepblue.lib.umich.edu/bitstream/2027.42/145686/1/Tao Chen Final Dissertation.pdfDescription of Tao Chen Final Dissertation.pdf : Dissertatio

SALSA: A Formal Hierarchical Optimization Framework for Smart Grid

The smart grid, by the integration of advanced control and optimization technologies, provides the traditional grid with an indisputable opportunity to deliver and utilize the electricity more efficiently. Building smart grid applications is a challenging task, which requires a formal modeling, integration, and validation framework for various smart grid domains. The design flow of such applications must adapt to the grid requirements and ensure the security of supply and demand. This dissertation, by proposing a formal framework for customers and operations domains in the smart grid, aims at delivering a smooth way for: i) formalizing their interactions and functionalities, ii) upgrading their components independently, and iii) evaluating their performance quantitatively and qualitatively.The framework follows an event-driven demand response program taking no historical data and forecasting service into account. A scalable neighborhood of prosumers (inside the customers domain), which are equipped with smart appliances, photovoltaics, and battery energy storage systems, are considered. They individually schedule their appliances and sell/purchase their surplus/demand to/from the grid with the purposes of maximizing their comfort and profit at each instant of time. To orchestrate such trade relations, a bilateral multi-issue negotiation approach between a virtual power plant (on behalf of prosumers) and an aggregator (inside the operations domain) in a non-cooperative environment is employed. The aggregator, with the objectives of maximizing its profit and minimizing the grid purchase, intends to match prosumers' supply with demand. As a result, this framework particularly addresses the challenges of: i) scalable and hierarchical load demand scheduling, and ii) the match between the large penetration of renewable energy sources being produced and consumed. It is comprised of two generic multi-objective mixed integer nonlinear programming models for prosumers and the aggregator. These models support different scheduling mechanisms and electricity consumption threshold policies.The effectiveness of the framework is evaluated through various case studies based on economic and environmental assessment metrics. An interactive web service for the framework has also been developed and demonstrated

A novel multi-level and community-based agent ecosystem to support customers dynamic decision-making in smart grids

Electrical systems have evolved at a fast pace over the past years, particularly in response to the current environmental and climate challenges. Consequently, the European Union and the United Nations have encouraged the development of a more sustainable energy strategy. This strategy triggered a paradigm shift in energy consumption and production, which becoming increasingly distributed, resulted in the development and emergence of smart energy grids. Multi-agent systems are one of the most widely used artificial intelligence concepts in smart grids. Both multi-agent systems and smart grids are distributed, so there is correspondence between the used technology and the network's complex reality. Due to the wide variety of multi-agent systems applied to smart grids, which typically have very specific goals, the ability to model the network as a whole may be compromised, as communication between systems is typically non-existent. This dissertation, therefore, proposes an agent-based ecosystem to model smart grids in which different agent-based systems can coexist. This dissertation aims to conceive, implement, test, and validate a new agent-based ecosystem, entitled A4SG (agent-based ecosystem for smart grids modelling), which combines the concepts of multi-agent systems and agent communities to enable the modelling and representation of smart grids and the entities that compose them. The proposed ecosystem employs an innovative methodology for managing static or dynamic interactions present in smart grids. The creation of a solution that allows the integration of existing systems into an ecosystem, enables the representation of smart grids in a realistic and comprehensive manner. A4SG integrates several functionalities that support the ecosystem's management, also conceived, implemented, tested, and validated in this dissertation. Two mobility functionalities are proposed: one that allows agents to move between physical machines and another that allows "virtual" mobility, where agents move between agent communities to improve the context for the achievement of their objectives. In order to prevent an agent from becoming overloaded, a novel functionality is proposed to enable the creation of agents that function as extensions of the main agent (i.e., branch agents), allowing the distribution of objectives among the various extensions of the main agent. Several case studies, which test the proposed services and functionalities individually and the ecosystem as a whole, were used to test and validate the proposed solution. These case studies were conducted in realistic contexts using data from multiple sources, including energy communities. The results indicate that the used methodologies can increase participation in demand response events, increasing the fitting between consumers and aggregators from 12 % to 69 %, and improve the strategies used in energy transaction markets, allowing an energy community of 50 customers to save 77.0 EUR per week.Os últimos anos têm sido de mudança nos sistemas elétricos, especialmente devido aos atuais desafios ambientais e climáticos. A procura por uma estratégia mais sustentável para o domínio da energia tem sido promovida pela União Europeia e pela Organização das Nações Unidas. A mudança de paradigma no que toca ao consumo e produção de energia, que acontece, cada vez mais, de forma distribuída, tem levado à emergência das redes elétricas inteligentes. Os sistemas multi-agente são um dos conceitos, no domínio da inteligência artificial, mais aplicados em redes inteligentes. Tanto os sistemas multi-agente como as redes inteligentes têm uma natureza distribuída, existindo por isso um alinhamento entre a tecnologia usada e a realidade complexa da rede. Devido a existir uma vasta oferta de sistemas multi-agente aplicados a redes inteligentes, normalmente com objetivos bastante específicos, a capacidade de modelar a rede como um todo pode ficar comprometida, porque a comunicação entre sistemas é, geralmente, inexistente. Por isso, esta dissertação propõe um ecossistema baseado em agentes para modelar as redes inteligentes, onde vários sistemas de agentes coexistem. Esta dissertação pretende conceber, implementar, testar, e validar um novo ecossistema multiagente, intitulado A4SG (agent-based ecosystem for smart grids modelling), que combina os conceitos de sistemas multi-agente e comunidades de agentes, permitindo a modelação e representação de redes inteligentes e das suas entidades. O ecossistema proposto utiliza uma metodologia inovadora para gerir as interações presentes nas redes inteligentes, sejam elas estáticas ou dinâmicas. A criação de um ecossistema que permite a integração de sistemas já existentes, cria a possibilidade de uma representação realista e detalhada das redes de energia. O A4SG integra diversas funcionalidades, também estas concebidas, implementadas, testadas, e validadas nesta dissertação, que suportam a gestão do próprio ecossistema. São propostas duas funcionalidades de mobilidade, uma que permite aos agentes mover-se entre máquinas físicas, e uma que permite uma mobilidade “virtual”, onde os agentes se movem entre comunidades de agentes, de forma a melhorar o contexto para a execução dos seus objetivos. É também proposta uma nova funcionalidade que permite a criação de agentes que funcionam como uma extensão de um agente principal, com o objetivo de evitar a sobrecarga de um agente, permitindo a distribuição de objetivos entre as várias extensões do agente principal. A solução proposta foi testada e validada por vários casos de estudo, que testam os serviços e funcionalidades propostas individualmente, e o ecossistema como um todo. Estes casos de estudo foram executados em contextos realistas, usando dados provenientes de diversas fontes, tais como comunidades de energia. Os resultados demonstram que as metodologias utilizadas podem melhorar a participação em eventos de demand response, subindo a adequação entre consumidores e agregadores de 12 % para 69 %, e melhorar as estratégias utilizadas em mercados de transações de energia, permitindo a uma comunidade de energia com 50 consumidores poupar 77,0 EUR por semana

Distributed Power Generation Scheduling, Modelling and Expansion Planning

Distributed generation is becoming more important in electrical power systems due to the decentralization of energy production. Within this new paradigm, new approaches for the operation and planning of distributed power generation are yet to be explored. This book deals with distributed energy resources, such as renewable-based distributed generators and energy storage units, among others, considering their operation, scheduling, and planning. Moreover, other interesting aspects such as demand response, electric vehicles, aggregators, and microgrid are also analyzed. All these aspects constitute a new paradigm that is explored in this Special Issue