Designing Efficient Local Flexibility Markets in the Presence of Reinforcement-Learning Agents

Local Flexibility Markets (LFMs) are considered a promising framework towards resolving voltage and congestion issues of power distribution systems in an economically efficient manner. However, the need for location-specific flexibility services renders LFMs naturally imperfectly competitive and market efficiency is severely challenged by strategic participants that exploit their locally monopolistic power. Previous works have been considering either non-strategic participants, or strategic participants with perfect information (e.g. about the network characteristics etc) that can readily compute their payoffmaximizing bidding strategy. In this paper, we take on the problem of designing an efficient LFM in the more realistic case where market participants do not possess this information and, instead, learn to improve their bidding policies through experience. To that end, we develop a multi-agent reinforcement learning algorithm to model the participants' learning-to-bid process. In this framework, we first present two popular LFM pricing schemes (pay-as-bid and distribution locational marginal pricing) and expose that learning agents can discover ways to exploit them, resulting in severe dispatch inefficiency. We then present a gametheoretic pricing scheme that theoretically incentivizes truthful bidding and empirically demonstrate that this property improves the efficiency of the resulting dispatch also in the presence of learning agents. In particular, the proposed scheme is able to outperform the popular distribution locational marginal pricing (DLMP) scheme, in terms of efficiency, by a factor of 15 − 23%

Adaptive flexibility function in smart energy systems: A linearized price-demand mapping approach

This paper proposes an adaptive mechanism for price signal generation using a piecewise linear approximation of a flexibility function with unknown parameters. In this adaptive approach, the price signal is parameterized and the parameters are changed adaptively such that the output of the flexibility function follows the reference demand signal provided by the involved aggregator. This is guaranteed using the Lyapunov stability theorem. The proposed method does not require an estimation algorithm for unknown parameters, that eliminates the need for persistency of excitation of signals, and consequently, simplifies offering the flexibility services. Furthermore, boundedness of the price signal is ensured using a projection algorithm in the adaptive system. We present simulation results that demonstrate the price generation results using the proposed approaches

Flexibility Aggregation of Temporally Coupled Resources in Real Time Balancing Markets Using Machine Learning

In modern power systems with high penetration of renewable energy sources, the flexibility provided by distributed energy resources is becoming invaluable. Demand aggregators offer balancing energy in the real-time balancing market on behalf of flexible resources. A challenging task is the design of the offering strategy of an aggregator. In particular, it is difficult to capture the flexibility cost of a portfolio of flexibility assets within a price-quantity offer, since the costs and constraints of flexibility resources exhibit inter-temporal dependencies. In this article, we propose a generic method for constructing aggregated balancing energy offers that best represent the portfolio's actual flexibility costs, while accounting for uncertainty in future timeslots. For the case study presented, we use offline simulations to train and compare different machine learning (ML) algorithms that receive the information about the state of the flexible resources and calculate the aggregator's offer. Once trained, the ML algorithms can make fast decisions about the portfolio's balancing energy offer in the real-time balancing market. Our simulations show that the proposed method performs reliably towards capturing the flexibility of the Aggregator's portfolio and minimizing the aggregator's imbalances.</p

Operation and Planning of Energy Hubs Under Uncertainty - a Review of Mathematical Optimization Approaches

Co-designing energy systems across multiple energy carriers is increasingly attracting attention of researchers and policy makers, since it is a prominent means of increasing the overall efficiency of the energy sector. Special attention is attributed to the so-called energy hubs, i.e., clusters of energy communities featuring electricity, gas, heat, hydrogen, and also water generation and consumption facilities. Managing an energy hub entails dealing with multiple sources of uncertainty, such as renewable generation, energy demands, wholesale market prices, etc. Such uncertainties call for sophisticated decision-making techniques, with mathematical optimization being the predominant family of decision-making methods proposed in the literature of recent years. In this paper, we summarize, review, and categorize research studies that have applied mathematical optimization approaches towards making operational and planning decisions for energy hubs. Relevant methods include robust optimization, information gap decision theory, stochastic programming, and chance-constrained optimization. The results of the review indicate the increasing adoption of robust and, more recently, hybrid methods to deal with the multi-dimensional uncertainties of energy hubs

Operation and planning of energy hubs under uncertainty - A review of mathematical optimization approaches

Co-designing energy systems across multiple energy carriers is increasingly attracting attention of researchers and policy makers, since it is a prominent means of increasing the overall efficiency of the energy sector. Special attention is attributed to the so-called energy hubs, i.e., clusters of energy communities featuring electricity, gas, heat, hydrogen, and also water generation and consumption facilities. Managing an energy hub entails dealing with multiple sources of uncertainty, such as renewable generation, energy demands, wholesale market prices, etc. Such uncertainties call for sophisticated decision-making techniques, with mathematical optimization being the predominant family of decision-making methods proposed in the literature of recent years. In this paper, we summarize, review, and categorize research studies that have applied mathematical optimization approaches towards making operational and planning decisions for energy hubs. Relevant methods include robust optimization, information gap decision theory, stochastic programming, and chance-constrained optimization. The results of the review indicate the increasing adoption of robust and, more recently, hybrid methods to deal with the multi-dimensional uncertainties of energy hubs.Web of Science117228720

Correlated Equilibrium Power Flow in Distribution Networks using Graphical Game Theory

Modern power systems necessitate active management of distribution networks by Distribution System Operators (DSOs). A DSO can harness the flexibility capabilities of the nodes towards ensuring the safe operation of the distribution system, while optimizing a certain system objective. The objective-optimizing dispatch can be obtained by solving the Optimal Power Flow (OPF) problem. However, selfish nodes may deviate from the dispatch instruction by calculating a profitable deviation. Thus, an implementable solution needs to be not only optimal, but also an equilibrium, i.e., a point from which no node is willing to deviate. In this paper, we propose the adoption of correlated equilibrium as a solution concept that is generally more efficient than Nash and more relevant to this setting where the DSO acts as a coordinating entity. We formulate the problem of finding an efficient correlated equilibrium for distribution networks with discrete resources. The problem’s complexity is managed by exploiting the graphical structure and sparse connectivity of distribution networks, drawing on the methodology of graphical games. Simulations showcase that the proposed approach achieves an equilibrium with near-optimal efficiency, in contrast to the standard OPF approach which is shown to be unstable when each node selfishly optimizes its own objective