New actor types in electricity market simulation models: Deliverable D4.4

Project TradeRES - New Markets Design & Models for 100% Renewable Power Systems: https://traderes.eu/about/ABSTRACT: The modelling of agents in the simulation models and tools is of primary importance if the quality and the validity of the simulation outcomes are at stake. This is the first version of the report that deals with the representation of electricity market actors’ in the agent based models (ABMs) used in TradeRES project. With the AMIRIS, the EMLab-Generation (EMLab), the MASCEM and the RESTrade models being in the centre of the analysis, the subject matter of this report has been the identification of the actors’ characteristics that are already covered by the initial (with respect to the project) version of the models and the presentation of the foreseen modelling enhancements. For serving these goals, agent attributes and representation methods, as found in the literature of agent-driven models, are considered initially. The detailed review of such aspects offers the necessary background and supports the formation of a context that facilitates the mapping of actors’ characteristics to agent modelling approaches. Emphasis is given in several approaches and technics found in the literature for the development of a broader environment, on which part of the later analysis is deployed. Although the ABMs that are used in the project constitute an important part of the literature, they have not been included in the review since they are the subject of another section.N/

Deep Reinforcement Learning for Power Trading

The Dutch power market includes a day-ahead market and an auction-like intraday balancing market. The varying supply and demand of power and its uncertainty induces an imbalance, which causes differing power prices in these two markets and creates an opportunity for arbitrage. In this paper, we present collaborative dual-agent reinforcement learning (RL) for bi-level simulation and optimization of European power arbitrage trading. Moreover, we propose two novel practical implementations specifically addressing the electricity power market. Leveraging the concept of imitation learning, the RL agent's reward is reformed by taking into account prior domain knowledge results in better convergence during training and, moreover, improves and generalizes performance. In addition, tranching of orders improves the bidding success rate and significantly raises the P&L. We show that each method contributes significantly to the overall performance uplifting, and the integrated methodology achieves about three-fold improvement in cumulative P&L over the original agent, as well as outperforms the highest benchmark policy by around 50% while exhibits efficient computational performance

Reinforcement learning for optimization of energy trading strategy

An increasing part of energy is produced from renewable sources by a large number of small producers. The efficiency of these sources is volatile and, to some extent, random, exacerbating the energy market balance problem. In many countries, that balancing is performed on day-ahead (DA) energy markets. In this paper, we consider automated trading on a DA energy market by a medium size prosumer. We model this activity as a Markov Decision Process and formalize a framework in which a ready-to-use strategy can be optimized with real-life data. We synthesize parametric trading strategies and optimize them with an evolutionary algorithm. We also use state-of-the-art reinforcement learning algorithms to optimize a black-box trading strategy fed with available information from the environment that can impact future prices

Proximal Policy Optimization Based Reinforcement Learning for Joint Bidding in Energy and Frequency Regulation Markets

Driven by the global decarbonization effort, the rapid integration of renewable energy into the conventional electricity grid presents new challenges and opportunities for the battery energy storage system (BESS) participating in the energy market. Energy arbitrage can be a significant source of revenue for the BESS due to the increasing price volatility in the spot market caused by the mismatch between renewable generation and electricity demand. In addition, the Frequency Control Ancillary Services (FCAS) markets established to stabilize the grid can offer higher returns for the BESS due to their capability to respond within milliseconds. Therefore, it is crucial for the BESS to carefully decide how much capacity to assign to each market to maximize the total profit under uncertain market conditions. This paper formulates the bidding problem of the BESS as a Markov Decision Process, which enables the BESS to participate in both the spot market and the FCAS market to maximize profit. Then, Proximal Policy Optimization, a model-free deep reinforcement learning algorithm, is employed to learn the optimal bidding strategy from the dynamic environment of the energy market under a continuous bidding scale. The proposed model is trained and validated using real-world historical data of the Australian National Electricity Market. The results demonstrate that our developed joint bidding strategy in both markets is significantly profitable compared to individual markets

How to trade electricity flexibility using artificial intelligence - An integrated algorithmic framework

In course of the energy transition, the growing share of Renewable Energy Sources (RES) makes electricity generation more decentralized and intermittent. This increases the relevance of exploiting flexibility potentials that help balancing intermittent RES supply and demand and, thus, contribute to overall system resilience. Digital technologies, in the form of automated trading algorithms, may considerably contribute to flexibility exploitation, as they enable faster and more accurate market interactions. In this paper, we develop an integrated algorithmic framework that finds an optimal trading strategy for flexibility on multiple markets. Hence, our work supports the trading of flexibility in a multi-market environment that results in enhanced market integration and harmonization of economically traded and physically delivered electricity, which finally promotes resilience in highly complex electricity systems

Demonstration of ALBidS: Adaptive Learning Strategic Bidding System

International Conference on Practical Applications of Agents and Multi-Agent SystemsCurrent worldwide electricity markets are strongly affected by the increasing use of renewable energy sources [1]. This increase has been stimulated by new energy policies that result from the growing concerns regarding the scarcity of fossil fuels and their impact in the environment. This has also led to an unavoidable restructuring of the power and energy sector, which was forced to adapt to the new paradigm [2]. The restructuring process resulted in a deep change in the operation of competitive electricity markets. The restructuring made the market more competitive, but also more complex, placing new challenges to the participants, which increases the difficulty of decision making. This is exacerbated by the increasing number of new market types that are being implemented to deal with the new challenges. Therefore, the intervenient entities are relentlessly forced to rethink their behaviour and market strategies in order to cope with such a constantly changing environment [2].This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 641794.info:eu-repo/semantics/publishedVersio