Contextual Simulated Annealing Q-Learning for Pre-negotiation of Agent-Based Bilateral Negotiations

Electricity markets are complex environments, which have been suffering continuous transformations due to the increase of renewable based generation and the introduction of new players in the system. In this context, players are forced to re-think their behavior and learn how to act in this dynamic environment in order to get as much benefit as possible from market negotiations. This paper introduces a new learning model to enable players identifying the expected prices of future bilateral agreements, as a way to improve the decision-making process in deciding the opponent players to approach for actual negotiations. The proposed model introduces a con-textual dimension in the well-known Q-Learning algorithm, and includes a simulated annealing process to accelerate the convergence process. The proposed model is integrated in a multi-agent decision support system for electricity market players negotiations, enabling the experimentation of results using real data from the Iberian electricity market.This work has received funding from the European Union's Horizon 2020 research and innovation programme under project DOMINOES (grant agreement No 771066) and from FEDER Funds through COMPETE program and from National Funds through FCT under the project UID/EEA/00760/2019.info:eu-repo/semantics/publishedVersio

Contextual Q-Learning

This paper highlights a new learning model that introduces a contextual dimension to the well-known Q-Learning algorithm. Through the identification of different contexts, the learning process is adapted accordingly, thus converging to enhanced results. The proposed learning model includes a simulated annealing (SA) process that accelerates the convergence process. The model is integrated in a multi-agent decision support system for electricity market players negotiations, enabling the experimentation of results using real electricity market data.This work has received funding from the EU Horizon 2020 research and innovation program under project DOMINOES (grant agreement No 771066) and from FEDER Funds through COMPETE program and from National Funds through FCT under projects CEECIND/01811/2017 and UIDB/00760/2020N/

Modelling the transition to a low-carbon energy supply

PhD ThesisA transition to a low-carbon electricity supply is crucial to limit the impacts of climate change. Reducing carbon emissions could help prevent the world from reaching a tipping point, where runaway emissions are likely. Runaway emissions could lead to extremes in weather conditions around the world - especially in problematic regions unable to cope with these conditions. However, the movement to a low-carbon energy supply can not happen instantaneously due to the existing fossil-fuel infrastructure and the requirement to maintain a reliable energy supply. Therefore, a low-carbon transition is required, however, the decisions various stakeholders should make over the coming decades to reduce these carbon emissions are not obvious. This is due to many long-term uncertainties, such as electricity, fuel and generation costs, human behaviour and the size of electricity demand. A well choreographed low-carbon transition is, therefore, required between all of the heterogenous actors in the system, as opposed to changing the behaviour of a single, centralised actor. The objective of this thesis is to create a novel, open-source agent-based model to better understand the manner in which the whole electricity market reacts to different factors using state-of-the-art machine learning and artificial intelligence methods. In contrast to other works, this thesis looks at both the long-term and short-term impact that different behaviours have on the electricity market by using these state-of-the-art methods. Specifically, we investigate the following applications: 1. Predictions are made to predict electricity demand in the short-term. We model the impact that poor predictions have on investments in electricity generators and utilisation over the long-term. We find that poor short-term predictions lead to a higher proportion of coal, gas, and nuclear power plants. 2. We devise a long-term carbon tax for the United Kingdom using a genetic algorithm approach. We find multiple strategies that can minimise both long-term carbon emissions and electricity cost. 3. Oligopolies can have a detrimental effect on an electricity market by raising electricity prices without an increase in benefit to users. Reinforcement learning can be used to devise intelligent bidding strategies which are based upon forecasts and predictions of other agent behaviour to maximise revenues. These behaviours can not be modelled through traditional rule-based algorithms. We use reinforcement learning to model strategic bidding into the electricity market, and find ways to limit the impact of this strategic bidding through a market cap. We find that introducing a market cap can significantly reduce the ability for oligopolies to manipulate the market. These studies require a number of core challenges to be addressed to ensure our agent-based model, ElecSim, is fit for purpose. These are: 1. Development of the ElecSim model, where the replication of the pertinent features of the electricity market was required. For example, generation company investment behaviour, electricity market design and temporal granularity. We find that the temporal granularity of the model has a large impact on accuracy of the model, but with suitably chosen representative days calibration is possible to accurately model a time period. 2. The complexity of a model increases with the replication of increasing market features. Therefore, optimisation of the code was required to maintain computational tractability, to allow for multiple scenario runs. This enabled us to run multiple iterations to train different machine learning techniques. 3. Once the model has been developed, its long-term behaviour must be verified to ensure accuracy. In this work, cross-validation was used to both validate and calibrate ElecSim. We are able to accurately model a historic period observed in the real-world with this approach 4. To ensure that the salient parameters are found, a sensitivity analysis was run. In addition, various example scenarios were generated to show the behaviour of the model. We find that the input parameters, such as the cost of capital have a disproportionate effect on the long-term electricity mix. The findings outlined previously demonstrate the ability for artificial intelligence, machine learning and agent-based models to perform complex analyses in an uncertain system. We find that solely focusing on the accuracy of machine learning techniques, for instance, misses out on a significant amount research potential. We instead argue, that by further developing these research themes, we are able to better understand the electricity market system of the United Kingdom