Deep Reinforcement Learning for Distribution Network Operation and Electricity Market

The conventional distribution network and electricity market operation have become challenging under complicated network operating conditions, due to emerging distributed electricity generations, coupled energy networks, and new market behaviours. These challenges include increasing dynamics and stochastics, and vast problem dimensions such as control points, measurements, and multiple objectives, etc. Previously the optimization models were often formulated as conventional programming problems and then solved mathematically, which could now become highly time-consuming or sometimes infeasible. On the other hand, with the recent advancement of artificial intelligence technologies, deep reinforcement learning (DRL) algorithms have demonstrated their excellent performances in various control and optimization fields. This indicates a potential alternative to address these challenges. In this thesis, DRL-based solutions for distribution network operation and electricity market have been investigated and proposed. Firstly, a DRL-based methodology is proposed for Volt/Var Control (VVC) optimization in a large distribution network, to effectively control bus voltages and reduce network power losses. Further, this thesis proposes a multi-agent (MA)DRL-based methodology under a complex regional coordinated VVC framework, and it can address spatial and temporal uncertainties. The DRL algorithm is also improved to adapt to the applications. Then, an integrated energy and heating systems (IEHS) optimization problem is solved by a MADRL-based methodology, where conventionally this could only be solved by simplifications or iterations. Beyond the applications in distribution network operation, a new electricity market service pricing method based on a DRL algorithm is also proposed. This DRL-based method has demonstrated good performance in this virtual storage rental service pricing problem, whereas this bi-level problem could hardly be solved directly due to a non-convex and non-continuous lower-level problem. These proposed methods have demonstrated advantageous performances under comprehensive case studies, and numerical simulation results have validated the effectiveness and high efficiency under different sophisticated operation conditions, solution robustness against temporal and spatial uncertainties, and optimality under large problem dimensions

Optimization Based Control for Multi-agent System with Interaction

Recently, the artificial intelligence has achieved a significant success with applications in various domains including transportation, smart building, robotics, economy and so on. More and more traditional system entities have been entitled with full or partial autonomy, allowing them to make their own decisions and moves based on the specific surrounding environments. An integration of multiple such intelligent entities is called a multi-agent system (MAS) where the agents need to interact with each other effectively and efficiently to attain cooperation and optimal system performance. As to fulfill this more challenging intelligent interaction objective, the traditional control approaches will not suffice and more advanced algorithms become essential.In this dissertation, three system structures for interactive control systems, centralized, distributed and decentralized, are discussed with application in intelligent building and autonomous driving. Several concrete interactive control algorithms are proposed and verified.In the centralized control system, a single central agent with the whole system information available is in charge of making decisions for all the agents. The systemwise cooperation solution is thus directly obtained and all the interactions involved are optimally addressed. Chapter 3 and 4 adopt such centralized control strategy for the intelligent building system. In order to save energy consumption and satisfy the occupants' thermal comfort demand, a combination of feedforward iterative learning control (ILC) and iteratively tuned feedback controller is designed to compensate both repetitive and non-repetitive disturbance components. Chapter 3 proposes an iterative controller design algorithm via optimization solving and stabilizing feedback projection. In Chapter 4, the concurrent design of feedforward ILC and causal stabilizing feedback controller is introduced, where both controllers are simultaneously solved by one optimization.However, the centralized approach's complexity grows with the problem size, which leads to failure for large-scale systems. The distributed control strategy is introduced as an alternative for such high-dimensional control problems. In the distributed system, a communication network enables the information exchange among agents. Therefore, each agent can keep broadcasting and updating its local controller until a convergence to the cooperative solution is reached. In Chapter 5, a distributed cooperative controller design method is developed for intelligent building thermal control with convergence property theoretically proven.For a system with no global communication, agents of which follow different control policies, the decentralized control structure is the only valid solution, where each agent designs its local controller independently based on estimated information of others. In Part II of the dissertation, several decentralized interactive control algorithms are proposed for the autonomous driving system. In Chapter 6, an optimization-based negotiation with both concession and persuasion is formulated for vehicle agent's decision making in various interactive scenarios. A Bayesian persuasion based algorithm for interactive driving is explored in Chapter 7. In the algorithm, the ego vehicle agent (persuader) intends to manipulate the interacting vehicle agent (information receiver)'s belief about the current driving situation via observable driving behavior. In Chapter 8, the interaction between two vehicle agents is defined as a two-player persuasion game, the mixed Nash equilibrium of which denotes the agents' optimal intention probabilities. The optimal intention is then expressed via the ego vehicle's driving trajectory planned by an optimization with the intention expression constraint

Decision support for participation in electricity markets considering the transaction of services and electricity at the local level

[EN] The growing concerns regarding the lack of fossil fuels, their costs, and their impact on the environment have led governmental institutions to launch energy policies that promote the increasing installation of technologies that use renewable energy sources to generate energy. The increasing penetration of renewable energy sources brings a great fluctuation on the generation side, which strongly affects the power and energy system management. The control of this system is moving from hierarchical and central to a smart and distributed approach. The system operators are nowadays starting to consider the final end users (consumers and prosumers) as a part of the solution in power system operation activities. In this sense, the end-users are changing their behavior from passive to active players. The role of aggregators is essential in order to empower the end-users, also contributing to those behavior changes. Although in several countries aggregators are legally recognized as an entity of the power and energy system, its role being mainly centered on representing end-users in wholesale market participation. This work contributes to the advancement of the state-of-the-art with models that enable the active involvement of the end-users in electricity markets in order to become key participants in the management of power and energy systems. Aggregators are expected to play an essential role in these models, making the connection between the residential end-users, electricity markets, and network operators. Thus, this work focuses on providing solutions to a wide variety of challenges faced by aggregators. The main results of this work include the developed models to enable consumers and prosumers participation in electricity markets and power and energy systems management. The proposed decision support models consider demand-side management applications, local electricity market models, electricity portfolio management, and local ancillary services. The proposed models are validated through case studies based on real data. The used scenarios allow a comprehensive validation of the models from different perspectives, namely end-users, aggregators, and network operators. The considered case studies were carefully selected to demonstrate the characteristics of each model, and to demonstrate how each of them contributes to answering the research questions defined to this work.[ES] La creciente preocupación por la escasez de combustibles fósiles, sus costos y su impacto en el medio ambiente ha llevado a las instituciones gubernamentales a lanzar políticas energéticas que promuevan la creciente instalación de tecnologías que utilizan fuentes de energía renovables para generar energía. La creciente penetración de las fuentes de energía renovable trae consigo una gran fluctuación en el lado de la generación, lo que afecta fuertemente la gestión del sistema de potencia y energía. El control de este sistema está pasando de un enfoque jerárquico y central a un enfoque inteligente y distribuido. Actualmente, los operadores del sistema están comenzando a considerar a los usuarios finales (consumidores y prosumidores) como parte de la solución en las actividades de operación del sistema eléctrico. En este sentido, los usuarios finales están cambiando su comportamiento de jugadores pasivos a jugadores activos. El papel de los agregadores es esencial para empoderar a los usuarios finales, contribuyendo también a esos cambios de comportamiento. Aunque en varios países los agregadores están legalmente reconocidos como una entidad del sistema eléctrico y energético, su papel se centra principalmente en representar a los usuarios finales en la participación del mercado mayorista. Este trabajo contribuye al avance del estado del arte con modelos que permiten la participación activa de los usuarios finales en los mercados eléctricos para convertirse en participantes clave en la gestión de los sistemas de potencia y energía. Se espera que los agregadores desempeñen un papel esencial en estos modelos, haciendo la conexión entre los usuarios finales residenciales, los mercados de electricidad y los operadores de red. Por lo tanto, este trabajo se enfoca en brindar soluciones a una amplia variedad de desafíos que enfrentan los agregadores. Los principales resultados de este trabajo incluyen los modelos desarrollados para permitir la participación de los consumidores y prosumidores en los mercados eléctricos y la gestión de los sistemas de potencia y energía. Los modelos de soporte de decisiones propuestos consideran aplicaciones de gestión del lado de la demanda, modelos de mercado eléctrico local, gestión de cartera de electricidad y servicios auxiliares locales. Los modelos propuestos son validan mediante estudios de casos basados en datos reales. Los escenarios utilizados permiten una validación integral de los modelos desde diferentes perspectivas, a saber, usuarios finales, agregadores y operadores de red. Los casos de estudio considerados fueron cuidadosamente seleccionados para demostrar las características de cada modelo y demostrar cómo cada uno de ellos contribuye a responder las preguntas de investigación definidas para este trabajo

Computational Intelligence Approaches for Energy Optimization in Microgrids

The future electrical system termed as smart grid represents a significant paradigm shift for power industry. Nowadays, microgrids are becoming smarter with the integration of renewable energy resources (RESs) , diesel generators , energy storage systems (ESS), and plug-in electric vehicles (PEV or EV) . However, these integration bring with new challenges for intelligent management systems. The classical power generation approaches can no longer be applied to a microgrid with unpredictable renewable energy resources. To relive these problem, a proper power system optimization and a suitable coordination strategy are needed to balance the supply and demand. This thesis presents three projects to study the optimization and control for smart community and to investigate the strategic impact and the energy trading techniques for interconnected microgrids. The first goal of this thesis is to propose a new game-theoretic framework to study the optimization and decision making of multi-players in the distributed power system. The proposed game theoretic special concept-rational reaction set (RRS) is capable to model the game of the distributed energy providers and the large residential consumers. Meanwhile, the residential consumers are able to participate in the retail electricity market to control the market price. Case studies are conducted to validate the system framework using the proposed game theoretic method. The simulation results show the effectiveness and the accuracy of the proposed strategic framework for obtaining the optimum profits for players participating in this market. The second goal of the thesis is to study a distributed convex optimization framework for energy trading of interconnected microgrids to improve the reliability of system operation. In this work, a distributed energy trading approach for interconnected operation of islanded microgrids is studied. Specifically, the system includes several islanded microgrids that can trade energy in a given topology. A distributed iterative deep cut ellipsoid (DCE) algorithm is implemented with limited information exchange. This approach will address the scalability issue and also secure local information on cost functions. During the iterative process, the information exchange among interconnected microgrids is restricted to electricity prices and expected trading energy. Numerical results are presented in terms of the convergent rate of the algorithm for different topologies, and the performance of the DCE algorithm is compared with sub-gradient algorithm. The third goal of this thesis is to use proper optimization approaches to motivate the household consumers to either shift their loads from peaking periods or reduce their consumption. Genetic algorithm (GA) and dynamic programming (DP) based smart appliance scheduling schemes and time-of-use pricing are investigated for comparative studies with demand response

Artificial Intelligence and Machine Learning Approaches to Energy Demand-Side Response: A Systematic Review

Recent years have seen an increasing interest in Demand Response (DR) as a means to provide flexibility, and hence improve the reliability of energy systems in a cost-effective way. Yet, the high complexity of the tasks associated with DR, combined with their use of large-scale data and the frequent need for near real-time de-cisions, means that Artificial Intelligence (AI) and Machine Learning (ML) — a branch of AI — have recently emerged as key technologies for enabling demand-side response. AI methods can be used to tackle various challenges, ranging from selecting the optimal set of consumers to respond, learning their attributes and pref-erences, dynamic pricing, scheduling and control of devices, learning how to incentivise participants in the DR schemes and how to reward them in a fair and economically efficient way. This work provides an overview of AI methods utilised for DR applications, based on a systematic review of over 160 papers, 40 companies and commercial initiatives, and 21 large-scale projects. The papers are classified with regards to both the AI/ML algorithm(s) used and the application area in energy DR. Next, commercial initiatives are presented (including both start-ups and established companies) and large-scale innovation projects, where AI methods have been used for energy DR. The paper concludes with a discussion of advantages and potential limitations of reviewed AI techniques for different DR tasks, and outlines directions for future research in this fast-growing area