Demand Response Strategy Based on Reinforcement Learning and Fuzzy Reasoning for Home Energy Management

As energy demand continues to increase, demand response (DR) programs in the electricity distribution grid are gaining momentum and their adoption is set to grow gradually over the years ahead. Demand response schemes seek to incentivise consumers to use green energy and reduce their electricity usage during peak periods which helps support grid balancing of supply-demand and generate revenue by selling surplus of energy back to the grid. This paper proposes an effective energy management system for residential demand response using Reinforcement Learning (RL) and Fuzzy Reasoning (FR). RL is considered as a model-free control strategy which learns from the interaction with its environment by performing actions and evaluating the results. The proposed algorithm considers human preference by directly integrating user feedback into its control logic using fuzzy reasoning as reward functions. Q-learning, a RL strategy based on a reward mechanism, is used to make optimal decisions to schedule the operation of smart home appliances by shifting controllable appliances from peak periods, when electricity prices are high, to off-peak hours, when electricity prices are lower without affecting the customer’s preferences. The proposed approach works with a single agent to control 14 household appliances and uses a reduced number of state-action pairs and fuzzy logic for rewards functions to evaluate an action taken for a certain state. The simulation results show that the proposed appliances scheduling approach can smooth the power consumption profile and minimise the electricity cost while considering user’s preferences, user’s feedbacks on each action taken and his/her preference settings. A user-interface is developed in MATLAB/Simulink for the Home Energy Management System (HEMS) to demonstrate the proposed DR scheme. The simulation tool includes features such as smart appliances, electricity pricing signals, smart meters, solar photovoltaic generation, battery energy storage, electric vehicle and grid supply.Peer reviewe

Assessment of Optimal Flexibility in Ensemble of Frequency Responsive Loads

Potential of electrical loads in providing grid ancillary services is often limited due to the uncertainties associated with the load behavior. A knowledge of the expected uncertainties with a load control program would invariably yield to better informed control policies, opening up the possibility of extracting the maximal load control potential without affecting grid operations. In the context of frequency responsive load control, a probabilistic uncertainty analysis framework is presented to quantify the expected error between the target and actual load response, under uncertainties in the load dynamics. A closed-form expression of an optimal demand flexibility, minimizing the expected error in actual and committed flexibility, is provided. Analytical results are validated through Monte Carlo simulations of ensembles of electric water heaters.Comment: IEEE International Conference on Smart Grid Communication

A Practical Integration of Automatic Generation Control and Demand Response

For a power grid to operate properly, electrical frequency must be continuously maintained close to its nominal value. Increasing penetration of distributed generation, such as solar and wind generation, introduces fluctuations in active power while also reducing the natural inertial response of the electricity grid, creating reliability concerns. While frequency regulation has traditionally been achieved by controlling generators, the control of Demand Response resources has been recognized in recent smart grid literature as an efficient means for providing additional regulation capability. To this end, several control methodologies have been proposed recently, but various features of these proposals make their practical implementations difficult. In this paper, we propose a new control algorithm that facilitates optimal frequency regulation through direct control of both generators and Demand Response, while addressing several issues that prevent practical implementation of other proposals. In particular, i) our algorithm is ideal for control over a large, low-bandwidth network as communication and measurement is only required every 2 seconds, ii) it enables Demand Response resources to recover energy lost during system transients, and iii) it accommodates both measured disturbances and unmeasured disturbances. We demonstrate the viability of our approach through dynamic simulations on a 118-bus grid model.NSF initiative, Award no. EFRI-144130

Efficient Energy Optimization for Smart Grid and Smart Community

The electric power industry has undergone significant changes in response to the environmental concerns during the past decades. Nowadays, due to the integration of different distributed energy systems in the smart grid, the balancing between power generation and load demand becomes a critical problem. Specifically, due to the intermittent nature of renewable energy sources (RESs) , power system optimization becomes significantly complicated. Due to the uncertain nature of RESs, the system may fail to ensure the power quality which may cause increased operating costs for committing costly reserve units or penalty costs for curtailing load demands. This dissertation presents three projects to study the optimization and control for smart grid and smart community. First, optimal operation of battery energy storage system (BESS) in grid-connected microgrid is studied. Near optimal operation/allocation of the BESS is investigated with the consideration of battery lifetime characteristics. Approximate dynamic programming (ADP) is proposed to solve optimal control policy for time-dependent and finite-horizon BESS problems and performance comparison is done with classical dynamic programming approach. The results show that the ADP can optimize the system operation under different scenarios to maximize the total system revenue. Second, optimal operation of the BESS in islanded microgrid is also studied. Specifically, a new islanded microgrid model is formulated based on Markov decision process. A computationally efficient ADP approach is proposed to solve this energy optimization problem, and achieve near minimum operational cost efficiently. Simulation results show that the proposed ADP can achieve 100% and at least 98% of optimality for deterministic and stochastic case studies, respectively. The performance of the proposed ADP approach also achieved 18:69 times faster response than that of the traditional DP approach for 0:5 million of data samples. Third, a demand side management technique is proposed for the optimization of residential demands with financial incentives. A new design of comfort indicator is proposed considering both thermal and other electric appliances based on consumers’ comfort level. The proposed approach is compared with two existing demand response approaches for both 10-houses and 100-houses simulation studies. For both cases, the proposed approach outperformed the existing approaches in terms of reward incentives and comfort levels

Joint Model Predictive Control of Electric and Heating Resources in a Smart Building

The new challenge in power systems design and operation is to organize and control smart micro grids supplying aggregation of users and special loads as electric vehicles charging stations. The presence of renewable and storage can help the optimal operation only if a good control manages all the elements of the grid. New models of green buildings and energy communities are proposed. For a real application they need an appropriate and advanced power system equipped with a building automation control system. This article presents an economic model predictive control approach to the problem of managing the electric and heating resources in a smart building in a coordinated way, for the purpose of achieving in real time nearly zero energy consumption and automated participation to demand response programs. The proposed control, leveraging a mixed integer quadratic programming problem, allows to meet manifold thermal and electric users' requirements and react to inbound demand response signals, while still guaranteeing stable operation of the building's electric and thermal storage equipment. The simulation results, performed for a real case study in Italy, highlight the peculiarities of the proposed approach in the joint handling of electric and thermal building flexibility

Incentive based Residential Demand Aggregation

From the beginning of the twenty-first century, the electrical power industry has moved from traditional power systems toward smart grids. However, with the increasing amount of renewable energy resources integrated into the grid, there is a significant challenge in power system operation due to the intermittency and variability of the renewables. Therefore, the utilization of flexible and controllable demand-side resources to maintain power system efficiency and stability has become a fundamental goal of smart grid initiatives. Meanwhile, due to the development of communication and sensing technologies, intelligent demand-side management with automatic controls enables residential loads to participate in demand response programs. Therefore, the aggregate control of residential appliances is anticipated to be feasible technique in the near future, which will bring considerable benefits to both residential consumers and load-serving entities. Hence, this dissertation proposes a comprehensive optimal framework for incentive based residential demand aggregation. The contents of this dissertation include: 1) a hardware design of smart home energy management system, 2) a new model to assess the responsive residential demand to financial incentives, and 3) an online algorithm for scheduling residential appliances. The proposed framework is expected to generate optimal control strategies over residential appliances enrolled in incentive based DR programs in real time. To residential consumers, this framework will 1) provide easy-to-use smart energy management solution, 2) distribute financial rewards by their quantified contribution in DR events, and 3) maintain residents’ comfort-level expectations based on their energy usage preferences. To LSEs, this framework can 1) aggregate residential demand to enhance system reliability, stability and efficiency, and 2) minimize the total reward costs for executing incentive based DR programs. Since this framework benefits both load serving entities and residents, it can stimulate the potential capability of residential appliances enrolled in incentive based DR programs. Eventually, with the growing number of DR participants, this framework has the potential to be one of the most vital parts in providing effective demand-side ancillary services for the entire power system

A Consensus-based Distributed Temperature Priority Control of Air Conditioner Clusters for Voltage Regulation in Distribution Networks

High penetration of Photovoltaic (PV) to the distribution network may bring under-voltage and over-voltage issues, limiting the PV hosting capacity. Air conditioners (AC) in grid-interactive buildings can support voltage regulation by manipulating flexible energy consumption. This paper developed a novel voltage control strategy to regulate the AC clusters’ on/off states for distribution network voltage regulation under high PV penetrations. The novelty lies in the distributed formulation of temperature priority-based on/off control (TPC) of AC clusters and the strategic selection and permutation of demand response technologies, including the real-time optimal demand response resources dispatch, distributed sensing of ACs based on average consensus algorithm, and the local implementation of TPC strategy and trial calculation scheme for flexibility capacity estimation. Finally, the distributed TPC is validated to be effective for system rebalancing with no comfort violations and an acceptable ON/OFF switching frequency. The theoretical and numerical analysis also proves its scalability and robustness to communication delays and link failures. It is then incorporated into a novel hierarchical control framework for smart grid voltage control in a four-bus three-phase test grid, considering the voltage sensitivities to power injections in different locations and phases

Dynamic electricity pricing for electric vehicles using stochastic programming

Electric Vehicles (EVs) are an important source of uncertainty, due to their variable demand, departure time and location. In smart grids, the electricity demand can be controlled via Demand Response (DR) programs. Smart charging and vehicle-to-grid seem highly promising methods for EVs control. However, high capital costs remain a barrier to implementation. Meanwhile, incentive and price-based schemes that do not require high level of control can be implemented to influence the EVs’ demand. Having effective tools to deal with the increasing level of uncertainty is increasingly important for players, such as energy aggregators. This paper formulates a stochastic model for day-ahead energy resource scheduling, integrated with the dynamic electricity pricing for EVs, to address the challenges brought by the demand and renewable sources uncertainty. The two-stage stochastic programming approach is used to obtain the optimal electricity pricing for EVs. A realistic case study projected for 2030 is presented based on Zaragoza network. The results demonstrate that it is more effective than the deterministic model and that the optimal pricing is preferable. This study indicates that adequate DR schemes like the proposed one are promising to increase the customers’ satisfaction in addition to improve the profitability of the energy aggregation business.info:eu-repo/semantics/acceptedVersio

Optimization models and algorithms for demand response in smart grid.

For demand response in smart grid, a utility company wants to minimize total electricity cost and end users want to maximize their own utility. The latter is considered to consist of two parts in this research: electricity cost and convenience/comfort. We first develop a system optimal (SO) model and a user equilibrium (UE) model for the utility company and end users, respectively and compare the difference of the two. We consider users\u27 possible preference on convenience over cost-saving under the real-time pricing in smart grid, and each user is assumed to have a preferred time window for using a particular appliance. As a result, each user in the proposed energy consumption game wishes to maximize a payoff or utility consisting of two parts: the negative of electricity cost and the convenience of using appliances during their preferred time windows. Numerical results show that users with less flexibility on their preferred usage times have larger impact on the system performance at equilibrium. Second, we found that instead of minimizing total cost, if utility company is regulated to maximize the social welfare, the user equilibrium model can achieve identical optimal solution as the system optimal model. We then design a demand response pricing frame work to accomplish this goal under alternative secondary objectives. We also investigate the non-uniqueness of the user equilibrium solution and prove that there exist alternative user equilibrium solutions. In this case, robust pricing is considered using multi-level optimization for the user equilibrium. Third, we study empirical data from a demand response pilot program in Kentucky in an attempt to understand consumer behavior under demand response and to characterize the thermo dynamics when set point for heat, ventilation and air conditioning (HVAC) is adjusted for demand response. Although sample size is limited, it helps to reveal the great variability in consumers\u27 response to demand response event. Using the real data collected, we consider to minimize the peak demand for a system consisting of smart thermostats, advanced hot water heaters and battery systems for storage. We propose a mixed integer program model as well as a heuristic algorithm for an optimal consumption schedule so that the system peak during a designated period is minimized. Therefore, we propose a consumption scheduling model to optimally control these loads and storage in maximizing efficiency without impacting thermal comfort. The model allows pre-cooling and pre-heating of homes to be performed for variable loads in low-demand times. We propose several future works. First, we introduce the concept of elastic demand to our SO model and UE model. The system problem maximizes net benefit to the energy consumers and the user problem is the usual one of finding equilibrium with elastic demand. The Karush-Kuhn-Tucker (KKT) conditions can be applied to solve the elastic demand problems. We also propose to develop algorithms for multi-level pricing models and further collect and analyze more field data in order to better understand energy users\u27 consumption behavior