Demand Response Strategy Based on Reinforcement Learning and Fuzzy Reasoning for Home Energy Management

As energy demand continues to increase, demand response (DR) programs in the electricity distribution grid are gaining momentum and their adoption is set to grow gradually over the years ahead. Demand response schemes seek to incentivise consumers to use green energy and reduce their electricity usage during peak periods which helps support grid balancing of supply-demand and generate revenue by selling surplus of energy back to the grid. This paper proposes an effective energy management system for residential demand response using Reinforcement Learning (RL) and Fuzzy Reasoning (FR). RL is considered as a model-free control strategy which learns from the interaction with its environment by performing actions and evaluating the results. The proposed algorithm considers human preference by directly integrating user feedback into its control logic using fuzzy reasoning as reward functions. Q-learning, a RL strategy based on a reward mechanism, is used to make optimal decisions to schedule the operation of smart home appliances by shifting controllable appliances from peak periods, when electricity prices are high, to off-peak hours, when electricity prices are lower without affecting the customer’s preferences. The proposed approach works with a single agent to control 14 household appliances and uses a reduced number of state-action pairs and fuzzy logic for rewards functions to evaluate an action taken for a certain state. The simulation results show that the proposed appliances scheduling approach can smooth the power consumption profile and minimise the electricity cost while considering user’s preferences, user’s feedbacks on each action taken and his/her preference settings. A user-interface is developed in MATLAB/Simulink for the Home Energy Management System (HEMS) to demonstrate the proposed DR scheme. The simulation tool includes features such as smart appliances, electricity pricing signals, smart meters, solar photovoltaic generation, battery energy storage, electric vehicle and grid supply.Peer reviewe

Energy Optimization and Coordination Frameworks for Smart Homes Considering Incentives From Discomfort and Market Analysis

The electricity demand is increasing with the growing use of electricity-based appliances in today’s world. The residential sector’s electricity consumption share is also increasing. Demand response (DR) is a typical way to schedule consumers’ energy consumption and help utility to reduce the peak load demand. Residential demand management can contribute to reduce peak electric demand, decrease electricity costs, and maintain grid reliability. Though the demand management has benefits to the utility and the consumers, controlling the consumers electricity consumption provides inconvenience to the consumers. The challenge here is to properly address the customers’ inconvenience to encourage them to participate and meanwhile satisfy the required demand reduction efficiently. In this work, new incentive-based demand management schemes for residential houses are designed and implemented. This work investigates two separate DR frameworks designed with different demand reduction coordination strategies. The first framework design constitutes a utility, several aggregators, and residential houses participating in DR program. Demand response potential (DRP), an indicator of whether an appliance can contribute to the DR, guides the strategic allocation of the demand limit to the aggregators. Each aggregator aggregates the DRP of all the controllable appliances under it and sends to the utility. The utility allocates different demand limits to the aggregators based on their respective DRP ratios. Participating residential customers are benefited with financial compensation with consideration of their inconvenience. Two scenarios are discussed in this approach with DRP. One where the thermostatically controlled loads (TCLs) are controlled. The thermal comfort of residents and rewards are used to evaluate the demand response performance. The other scenario includes the time-shiftable appliances control with the same framework. The second framework is a three-level hierarchical control framework for large-scale residential DR with a novel bidding scheme and market-level analysis. It comprises of several residential communities, local controllers (LCs), a central controller (CC), and the electricity market. A demand reduction bidding strategy is introduced for the coordination among several LCs under a CC in this framework. Incentives are provided to the participating residential consumers, while considering their preferences, using a continuous reward structure. A simulation study on the 6-bus Roy Billinton Test System with 1;200 residential consumers demonstrates the financial benefits to both the electric utility and consumers

Artificial Intelligence and Machine Learning Approaches to Energy Demand-Side Response: A Systematic Review

Recent years have seen an increasing interest in Demand Response (DR) as a means to provide flexibility, and hence improve the reliability of energy systems in a cost-effective way. Yet, the high complexity of the tasks associated with DR, combined with their use of large-scale data and the frequent need for near real-time de-cisions, means that Artificial Intelligence (AI) and Machine Learning (ML) — a branch of AI — have recently emerged as key technologies for enabling demand-side response. AI methods can be used to tackle various challenges, ranging from selecting the optimal set of consumers to respond, learning their attributes and pref-erences, dynamic pricing, scheduling and control of devices, learning how to incentivise participants in the DR schemes and how to reward them in a fair and economically efficient way. This work provides an overview of AI methods utilised for DR applications, based on a systematic review of over 160 papers, 40 companies and commercial initiatives, and 21 large-scale projects. The papers are classified with regards to both the AI/ML algorithm(s) used and the application area in energy DR. Next, commercial initiatives are presented (including both start-ups and established companies) and large-scale innovation projects, where AI methods have been used for energy DR. The paper concludes with a discussion of advantages and potential limitations of reviewed AI techniques for different DR tasks, and outlines directions for future research in this fast-growing area

MARL-iDR: Multi-Agent Reinforcement Learning for Incentive-based Residential Demand Response

This paper presents a decentralized Multi-Agent Reinforcement Learning (MARL) approach to an incentive-based Demand Response (DR) program, which aims to maintain the capacity limits of the electricity grid and prevent grid congestion by financially incentivizing residential consumers to reduce their energy consumption. The proposed approach addresses the key challenge of coordinating heterogeneous preferences and requirements from multiple participants while preserving their privacy and minimizing financial costs for the aggregator. The participant agents use a novel Disjunctively Constrained Knapsack Problem optimization to curtail or shift the requested household appliances based on the selected demand reduction. Through case studies with electricity data from $25$ households, the proposed approach effectively reduced energy consumption's Peak-to-Average ratio (PAR) by $14.48$% compared to the original PAR while fully preserving participant privacy. This approach has the potential to significantly improve the efficiency and reliability of the electricity grid, making it an important contribution to the management of renewable energy resources and the growing electricity demand.Comment: 8 pages, IEEE Belgrade PowerTech, 202

Coordinated Smart Home Thermal and Energy Management System Using a Co-simulation Framework

The increasing demand for electricity especially during the peak hours threaten the grid reliability. Demand response (DR), changing the load pattern of the consumer in response to system conditions, can decrease energy consumption during periods of high wholesale market price and also maintain system reliability. Residential homes consume 38% of the total electric energy in the U.S., making them promising for DR participation. Consumers can be motivated to participate in DR programs by providing incentives (incentive-based DR), or by introducing a time-varying tariff for electricity consumption (price-based DR). A home energy management system (HEMS), an automated system which can alter the residential consumer’s energy consumption pattern based on the price of electricity or financial incentives, enables the consumers to participate in such DR programs. HEMS also should consider consumer comfort during the scheduling of the heating, ventilation, and air conditioning (HVAC) and other appliances. As internal heat gain of appliances and people have a significant effect in the HVAC energy consumption, an integrated HVAC and appliance scheduling are necessary to properly evaluate potential benefits of HEMS. This work presents the formulation of HEMS considering combined scheduling of HVAC and appliances in time-varying tariff. The HEMS also considers the consumer comfort for the HVAC and appliances while minimizing the total electricity cost. Similarly, the HEMS also considers the detailed building model in EnergyPlus, a building energy analysis tool, to evaluate the effectiveness of the HEMS. HEMS+, a communication interface to EnergyPlus, is designed to couple HEMS and EnergyPlus in this work. Furthermore, a co-simulation framework coupling EnergyPlus and GridLAB-D, a distribution system simulation tool, is developed. This framework enables incorporation of the controllers such as HEMS and aggregator, allowing controllers to be tested in detail in both building and power system domains. Lack of coordination among a large number of HEMS responding to same price signal results in peak more severe than the normal operating condition. This work presents an incentive-based hierarchical control framework for coordinating and controlling a large number of residential consumers’ thermostatically controlled loads (TCLs) such as HVAC and electric water heater (EWH). The potential market-level economic benefits of the residential demand reduction are also quantified

Demand response performance and uncertainty: A systematic literature review

The present review has been carried out, resorting to the PRISMA methodology, analyzing 218 published articles. A comprehensive analysis has been conducted regarding the consumer's role in the energy market. Moreover, the methods used to address demand response uncertainty and the strategies used to enhance performance and motivate participation have been reviewed. The authors find that participants will be willing to change their consumption pattern and behavior given that they have a complete awareness of the market environment, seeking the optimal decision. The authors also find that a contextual solution, giving the right signals according to the different behaviors and to the different types of participants in the DR event, can improve the performance of consumers' participation, providing a reliable response. DR is a mean of demand-side management, so both these concepts are addressed in the present paper. Finally, the pathways for future research are discussed.This article is a result of the project RETINA (NORTE-01-0145- FEDER-000062), supported by Norte Portugal Regional Operational Programme (NORTE 2020), under the PORTUGAL 2020 Partnership Agreement, through the European Regional Development Fund (ERDF). We also acknowledge the work facilities and equipment provided by GECAD research center (UIDB/00760/2020) to the project team, and grants CEECIND/02887/2017 and SFRH/BD/144200/2019.info:eu-repo/semantics/publishedVersio

Efficient Energy Optimization for Smart Grid and Smart Community

The electric power industry has undergone significant changes in response to the environmental concerns during the past decades. Nowadays, due to the integration of different distributed energy systems in the smart grid, the balancing between power generation and load demand becomes a critical problem. Specifically, due to the intermittent nature of renewable energy sources (RESs) , power system optimization becomes significantly complicated. Due to the uncertain nature of RESs, the system may fail to ensure the power quality which may cause increased operating costs for committing costly reserve units or penalty costs for curtailing load demands. This dissertation presents three projects to study the optimization and control for smart grid and smart community. First, optimal operation of battery energy storage system (BESS) in grid-connected microgrid is studied. Near optimal operation/allocation of the BESS is investigated with the consideration of battery lifetime characteristics. Approximate dynamic programming (ADP) is proposed to solve optimal control policy for time-dependent and finite-horizon BESS problems and performance comparison is done with classical dynamic programming approach. The results show that the ADP can optimize the system operation under different scenarios to maximize the total system revenue. Second, optimal operation of the BESS in islanded microgrid is also studied. Specifically, a new islanded microgrid model is formulated based on Markov decision process. A computationally efficient ADP approach is proposed to solve this energy optimization problem, and achieve near minimum operational cost efficiently. Simulation results show that the proposed ADP can achieve 100% and at least 98% of optimality for deterministic and stochastic case studies, respectively. The performance of the proposed ADP approach also achieved 18:69 times faster response than that of the traditional DP approach for 0:5 million of data samples. Third, a demand side management technique is proposed for the optimization of residential demands with financial incentives. A new design of comfort indicator is proposed considering both thermal and other electric appliances based on consumers’ comfort level. The proposed approach is compared with two existing demand response approaches for both 10-houses and 100-houses simulation studies. For both cases, the proposed approach outperformed the existing approaches in terms of reward incentives and comfort levels

Optimal Scheduling of Electric Vehicle Charging with Deep Reinforcement Learning considering End Users Flexibility

The rapid growth of decentralized energy resources and especially Electric Vehicles (EV), that are expected to increase sharply over the next decade, will put further stress on existing power distribution networks, increasing the need for higher system reliability and flexibility. In an attempt to avoid unnecessary network investments and to increase the controllability over distribution networks, network operators develop demand response (DR) programs that incentivize end users to shift their consumption in return for financial or other benefits. Artificial intelligence (AI) methods are in the research forefront for residential load scheduling applications, mainly due to their high accuracy, high computational speed and lower dependence on the physical characteristics of the models under development. The aim of this work is to identify households' EV cost-reducing charging policy under a Time-of-Use tariff scheme, with the use of Deep Reinforcement Learning, and more specifically Deep Q-Networks (DQN). A novel end users flexibility potential reward is inferred from historical data analysis, where households with solar power generation have been used to train and test the designed algorithm. The suggested DQN EV charging policy can lead to more than 20% of savings in end users electricity bills