Policy-based power consumption management in smart energy community using single agent and multi agent Q learning algorithms

Power consumption in residential sector has increased due to growing population, economic growth, invention of many electrical appliances and therefore is becoming a growing concern in the power industry. Managing power consumption in residential sector without sacrificing user comfort has become one of the main research areas recently. The complexity of the power system keeps growing due to the penetration of alternative sources of electric energy such as solar plant, Hydro, Biomass, Geothermal and wind farm to meet the growing demand for electricity. To overcome the challenges due to complexity, the power grid needs to be intelligent in all aspects. As the grid gets smarter and smarter, considerable efforts are being undertaken to make the houses and businesses smarter in consuming the electrical energy to minimize and level the electricity demand which is also known as Demand Side Management (DSM). It also necessitates that the conventional way of modelling, control and energy management in all sectors needs to be enhanced or replaced by intelligent information processing techniques. In our research work, it has been done in several stages. (Purpose of Study and Results) We proposed a policy-based framework which allows intelligent and flexible energy management of home appliances in a smart home which is complex and dynamic in ways that saves energy automatically. We considered the challenges in formalizing the behaviour of the appliances using their states and managing the energy consumption using policies. Policies are rules which are created and edited by a house agent to deal with situations or power problems that are likely to occur. Each time the power problem arises the house agent will refer to policy and one or a set of rules will be executed to overcome that situation. Our policy-based smart home can manage energy efficiently and can significantly participate in reducing peak energy demand (thereby may reduce carbon emission). Our proposed policy-based framework achieves peak shaving so that power consumption adapts to available power, while ensuring the comfort level of the inhabitants and taking device characteristics in to account. Our simulation results on MATLAB indicate that the proposed Policy driven homes can effectively contribute to Demand side power management by decreasing the peak hour usage of the appliances and can efficiently manage energy in a smart home in a user-friendly way. We propounded and developed peak demand management algorithms for a Smart Energy Community using different types of coordination mechanisms for coordination of multiple house agents working in the same environment. These algorithms use centralized model, decentralized model, hybrid model and Pareto resource allocation model for resource allocation. We modelled user comfort for the appliance based on user preference, the power reduction capability and the important activities that run around the house associated with that appliance. Moreover, we compared these algorithms with respect to their peak reduction capability, overall comfort of the community, simplicity of the algorithm and community involvement and finally able to find the best performing algorithm among them. Our simulation results show that the proposed coordination algorithms can effectively reduce peak demand while maintaining user comfort. With the help of our proposed algorithms, the demand for electricity of a smart community can be managed intelligently and sustainably. This work is not only aiming for peak reduction management it aims for achieving it while keeping the comfort level of the inhabitants is minimum. It can learn user’s behaviour and establish the set of optimal rules dynamically. If the available power to a house is kept at a certain level the house agent will learn to use this notional power to operate all the appliances according to the requirements and comfort level of the household. This way the consumers are forced to use the power below the set level which can result in the over-all power consumption be maintained at a certain rate or level which means sustainability is possible or depletion of natural resources for electricity can be reduced. Temporal interactions of Energy Demand by local users and renewable energy sources can also be done more efficiently by having a set of new policy rules to switch between the utility and the renewable source of energy but it is beyond the scope of this thesis. We applied Q learning techniques to a home energy management agent where the agent learns to find the optimal sequence of turning off appliances so that the appliances with higher priority will not be switched off during peak demand period or power consumption management. The policy-based home energy management determines the optimal policy at every instant dynamically by learning through the interaction with the environment using one of the reinforcement learning approaches called Q-learning. The Q-learning home power consumption problem formulation consisting of state space, actions and reward function is presented. The implications of these simulation results are that the proposed Q- learning based power consumption management is very effective and enables the users to have minimum discomfort during participation in peak demand management or at the time when power consumption management is essential when the available power is rationale. This work is extended to a group of 10 houses and three multi agent Q- learning algorithms are proposed and developed for improving the individual and community comfort while at the same time keeping the power consumption below the available power level or electricity price below the set price. The proposed algorithms are weighted strategy sharing algorithm, concurrent Q learning algorithm and cooperative distributive learning algorithm. These proposed algorithms are coded and tested for managing power consumption of a group of 10 houses and the performance of all three algorithms with respect to power management and community comfort is studied and compared. Actual power consumption of a community and modified power consumption curves using Weighted Strategy Sharing algorithm, Concurrent learning and Distributive Q Learning and user comfort results are presented, and the results are analysed in this thesis

Multi-agent Electricity Markets and Smart Grids Simulation with Connection to Real Physical Resources

The increasing penetration of distributed energy sources, mainly based on renewable generation, calls for an urgent emergence of novel advanced methods to deal with the associated problems. The consensus behind smart grids (SGs) as one of the most promising solutions for the massive integration of renewable energy sources in power systems has led to the development of several prototypes that aim at testing and validating SG methodologies. The urgent need to accommodate such resources require alternative solutions. This chapter presents a multi-agent based SG simulation platform connected to physical resources, so that realistic scenarios can be simulated. The SG simulator is also connected to the Multi-Agent Simulator of Competitive Electricity Markets, which provides a solid framework for the simulation of electricity markets. The cooperation between the two simulation platforms provides huge studying opportunities under different perspectives, resulting in an important contribution to the fields of transactive energy, electricity markets, and SGs. A case study is presented, showing the potentialities for interaction between players of the two ecosystems: a SG operator, which manages the internal resources of a SG, is able to participate in electricity market negotiations to trade the necessary amounts of power to fulfill the needs of SG consumers.This work has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement N. 641794 (project DREAM-GO). It has also received FEDER Funds through the COMPETE program and National Funds through FCT under the project UID/EEA/00760/2013. The authors gratefully acknowledge the valuable contribution of Bruno Canizes, Daniel Paiva, Gabriel Santos and Marco Silva to the work presented in the chapter.info:eu-repo/semantics/publishedVersio

A Multi-Agent Energy Trading Competition

The energy sector will undergo fundamental changes over the next ten years. Prices for fossil energy resources are continuously increasing, there is an urgent need to reduce CO2 emissions, and the United States and European Union are strongly motivated to become more independent from foreign energy imports. These factors will lead to installation of large numbers of distributed renewable energy generators, which are often intermittent in nature. This trend conflicts with the current power grid control infrastructure and strategies, where a few centralized control centers manage a limited number of large power plants such that their output meets the energy demands in real time. As the proportion of distributed and intermittent generation capacity increases, this task becomes much harder, especially as the local and regional distribution grids where renewable energy generators are usually installed are currently virtually unmanaged, lack real time metering and are not built to cope with power flow inversions (yet). All this is about to change, and so the control strategies must be adapted accordingly. While the hierarchical command-and-control approach served well in a world with a few large scale generation facilities and many small consumers, a more flexible, decentralized, and self-organizing control infrastructure will have to be developed that can be actively managed to balance both the large grid as a whole, as well as the many lower voltage sub-grids. We propose a competitive simulation test bed to stimulate research and development of electronic agents that help manage these tasks. Participants in the competition will develop intelligent agents that are responsible to level energy supply from generators with energy demand from consumers. The competition is designed to closely model reality by bootstrapping the simulation environment with real historic load, generation, and weather data. The simulation environment will provide a low-risk platform that combines simulated markets and real-world data to develop solutions that can be applied to help building the self-organizing intelligent energy grid of the future

A Systematic Literature Review of Peer-to-Peer, Community Self-Consumption, and Transactive Energy Market Models

Capper, T., Gorbatcheva, A., Mustafa, M. A., Bahloul, M., Schwidtal, J. M., Chitchyan, R., Andoni, M., Robu, V., Montakhabi, M., Scott, I., Francis, C., Mbavarira, T., Espana, J. M., & Kiesling, L. (2021). A Systematic Literature Review of Peer-to-Peer, Community Self-Consumption, and Transactive Energy Market Models. Social Science Research Network (SSRN), Elsevier. https://doi.org/10.2139/ssrn.3959620Peer-to-peer and transactive energy markets, and community or collective self-consumption offer new models for trading energy locally. Over the past 10 years there has been significant growth in the amount of academic literature and trial projects examining how these energy trading models might function. This systematic literature review of 139 peer-reviewed journal articles examines the market designs used in these energy trading models. The Business Ecosystem Architecture Modelling framework is used to extract information about the market models used in the literature and identify differences and similarities between the models. This paper identifies six archetypal market designs and three archetypal auction mechanisms used in markets presented in the reviewed literature. It classifies the types of commodities being traded, the benefits of the markets and other features such as the types of grid models. Finally, this paper identifies five evidence gaps which need future research before these markets can be widely adopted.publishersversionpublishe

A generic holonic control architecture for heterogeneous multi-scale and multi-objective smart microgrids

Designing the control infrastructure of future “smart” power grids is a challenging task. Future grids will integrate a wide variety of heterogeneous producers and consumers that are unpredictable and operate at various scales. Information and Communication Technology (ICT) solutions will have to control these in order to attain global objectives at the macrolevel, while also considering private interests at the microlevel. This article proposes a generic holonic architecture to help the development of ICT control systems that meet these requirements. We show how this architecture can integrate heterogeneous control designs, including state-of-the-art smart grid solutions. To illustrate the applicability and utility of this generic architecture, we exemplify its use via a concrete proof-of-concept implementation for a holonic controller, which integrates two types of control solutions and manages a multiscale, multiobjective grid simulator in several scenarios. We believe that the proposed contribution is essential for helping to understand, to reason about, and to develop the “smart” side of future power grids

An adaptive agent-based system for deregulated smart grids

The power grid is undergoing a major change due mainly to the increased penetration of renewables and novel digital instruments in the hands of the end users that help to monitor and shift their loads. Such transformation is only possible with the coupling of an information and communication technology infrastructure to the existing power distribution grid. Given the scale and the interoperability requirements of such future system, service-oriented architectures (SOAs) are seen as one of the reference models and are considered already in many of the proposed standards for the smart grid (e.g., IEC-62325 and OASIS eMIX). Beyond the technical issues of what the service-oriented architectures of the smart grid will look like, there is a pressing question about what the added value for the end user could be. Clearly, the operators need to guarantee availability and security of supply, but why should the end users care? In this paper, we explore a scenario in which the end users can both consume and produce small quantities of energy and can trade these quantities in an open and deregulated market. For the trading, they delegate software agents that can fully interoperate and interact with one another thus taking advantage of the SOA. In particular, the agents have strategies, inspired from game theory, to take advantage of a service-oriented smart grid market and give profit to their delegators, while implicitly helping balancing the power grid. The proposal is implemented with simulated agents and interaction with existing Web services. To show the advantage of the agent with strategies, we compare our approach with the “base” agent one by means of simulations, highlighting the advantages of the proposal