Centralised rehearsal of decentralised cooperation: Multi-agent reinforcement learning for the scalable coordination of residential energy flexibility

This paper investigates how deep multi-agent reinforcement learning can enable the scalable and privacy-preserving coordination of residential energy flexibility. The coordination of distributed resources such as electric vehicles and heating will be critical to the successful integration of large shares of renewable energy in our electricity grid and, thus, to help mitigate climate change. The pre-learning of individual reinforcement learning policies can enable distributed control with no sharing of personal data required during execution. However, previous approaches for multi-agent reinforcement learning-based distributed energy resources coordination impose an ever greater training computational burden as the size of the system increases. We therefore adopt a deep multi-agent actor-critic method which uses a \emph{centralised but factored critic} to rehearse coordination ahead of execution. Results show that coordination is achieved at scale, with minimal information and communication infrastructure requirements, no interference with daily activities, and privacy protection. Significant savings are obtained for energy users, the distribution network and greenhouse gas emissions. Moreover, training times are nearly 40 times shorter than with a previous state-of-the-art reinforcement learning approach without the factored critic for 30 homes

Multi-time scale control of demand flexibility in smart distribution networks

This paper presents a multi-timescale control strategy to deploy electric vehicle (EV) demand flexibility for simultaneously providing power balancing, grid congestion management, and economic benefits to participating actors. First, an EV charging problem is investigated from consumer, aggregator, and distribution system operator’s perspectives. A hierarchical control architecture (HCA) comprising scheduling, coordinative, and adaptive layers is then designed to realize their coordinative goal. This is realized by integrating multi-time scale controls that work from a day-ahead scheduling up to real-time adaptive control. The performance of the developed method is investigated with high EV penetration in a typical residential distribution grid. The simulation results demonstrate that HCA efficiently utilizes demand flexibility stemming from EVs to solve grid unbalancing and congestions with simultaneous maximization of economic benefits to the participating actors. This is ensured by enabling EV participation in day-ahead, balancing, and regulation markets. For the given network configuration and pricing structure, HCA ensures the EV owners to get paid up to five times the cost they were paying without control

Intermediation in Future Energy Markets: Innovative Product Design and Pricing

In order to mitigate the impacts of climate change, the international community envisages significant investments in electricity generation from renewable energy sources (RES). The integration of this decentralized and fluctuating type electricity generation poses several challenges to planning, operation, and economics of power systems. The established energy systems were originally designed for a centralized electricity generation that follows the uncontrolled but well predictable demand. However, for large shares of RES, relying only on the flexibility of the generation side would be economically inefficient. Furthermore, the environmental benefits of using RES would be depleted by additional carbon emissions from ramping highly flexible fossil-fueled power plants. An appealing alternative to facilitate the efficient integration of large shares of RES is to exploit the so far mainly passive demand side as an additional source of flexibility. The established centralized approaches can hardly handle the fine-grained and decentralized nature of demand side flexibility. Therefore, the intermediation between centralized control and decentralized demand will play a major role in future energy markets, which constitutes the overarching topic of this dissertation. Typically electricity generation from RES is capital-intensive but has near zero marginal costs. On this account, novel services need to be offered in order to transmit the right economic signals. To this end, the concept of the differentiable good electricity is refined in this dissertation. Embedded into the so-called energy service, characteristics such as temporal and spatial price differentiation or the risk of interruption can be specified to differentiate the so far homogeneous good. Based on the morphological design theory a framework for the notion of energy services is established and subsequently implemented as a decision support system. This supports a systematic and structured product development process to design innovative energy services. Such an innovative energy service is, e.g., the charging of electric vehicles in car parks, where prices are differentiated by job completion deadline. This allows the car park operator to control the aggregated load of all charging jobs to follow local RES generation. Based on this energy service the downstream activity of an intermediary is formally modeled as an optimization problem and evaluated by means of an empirical simulation experiment. The results provide insights on pricing policy and the value of demand side flexibility with regard to both the integration of local RES generation and operative profit optimization. In order to illustrate another innovative energy service the presented model is extended by the upstream activity of the intermediary. Household consumers are offered monetary incentives if they allow the intermediary to control their appliances. The results indicate the cost saving potential from demand side flexibility for the intermediary\u27s procurement of electricity. Beyond that, this model formulation constitutes the foundation for further examinations, e.g., to study the strategic behavior of intermediaries on real-time electricity markets that are prone to market power abuse due to low market liquidity