Multi-Agent Reinforcement Learning for Optimal Network and Market Operations in Active Distribution Networks

Abstract

The global energy transition toward decarbonization, decentralisation, and digitalisation is driving rapid growth of distributed energy resources (DERs) such as photovoltaics, battery energy storage, electric vehicles, and flexible loads. Their widespread adoption reshapes electricity distribution networks, enabling higher renewable utilisation, local flexibility, and active prosumer participation. Yet it also introduces new challenges in system coordination, operational reliability, and economic efficiency. To fully capture DER potential, tightly coupled system–market frameworks are urgently needed. Market operations not only guide resource allocation and price formation but also provide a scalable basis for coordinating numerous heterogeneous, agent-based entities at the distribution level. This thesis proposes a layered market design that distinguishes between internal and external markets within virtual power plants (VPPs), enabling hierarchical and flexible energy trading. A carbon-aware market-clearing mechanism is further developed to jointly optimise economic cost and carbon emissions, embedding environmental considerations into operational decision-making. However, advanced market mechanisms create additional technical hurdles, including unbalanced power flow management, constraint satisfaction under decentralised control, and learning under uncertainty. To address these issues, the thesis develops a multi-agent reinforcement learning (MARL) suite featuring: safe policy learning to satisfy nonlinear voltage and emission constraints, decentralised training to preserve data privacy, and large language model assistance to enhance robustness against exogenous uncertainties. All proposed mechanisms and algorithms are validated on standard IEEE distribution test systems. Results show that coordinated system and market operations empowered by MARL can significantly improve the scalability, security, and sustainability of next-generation distribution networks