Aggregation and Control of Populations of Thermostatically Controlled Loads by Formal Abstractions

This work discusses a two-step procedure, based on formal abstractions, to generate a finite-space stochastic dynamical model as an aggregation of the continuous temperature dynamics of a homogeneous population of Thermostatically Controlled Loads (TCL). The temperature of a single TCL is described by a stochastic difference equation and the TCL status (ON, OFF) by a deterministic switching mechanism. The procedure is formal as it allows the exact quantification of the error introduced by the abstraction -- as such it builds and improves on a known, earlier approximation technique in the literature. Further, the contribution discusses the extension to the case of a heterogeneous population of TCL by means of two approaches resulting in the notion of approximate abstractions. It moreover investigates the problem of global (population-level) regulation and load balancing for the case of TCL that are dependent on a control input. The procedure is tested on a case study and benchmarked against the mentioned alternative approach in the literature.Comment: 40 pages, 21 figures; the paper generalizes the result of conference publication: S. Esmaeil Zadeh Soudjani and A. Abate, "Aggregation of Thermostatically Controlled Loads by Formal Abstractions," Proceedings of the European Control Conference 2013, pp. 4232-4237. version 2: added references for section

Sequential set-point control of thermostatic loads using extended Markov chain abstraction to improve future renewable energy integration

Additional flexible resources are required to achieve resilience and sustainable power systems. Challenges emerged due to the increasing amounts of renewable generation penetrations at both the bulk power system and the distribution sides. System operators are required to deal with higher levels of variable and uncertain power outputs for various time-scales. Moreover, replacing existing thermal units with other inertial-less technologies, make the system sensitive to even small contingencies. Demand-side control is becoming an ingredient part of our future power system operation. Effective utilization of demand-side resources can make the system more elastic to integrate the future renewable plans. To help in resolving these challenges, this work develops a demand-side control framework on the Thermostatically Controlled Loads (TCLs) to support the grid with minimal impacts on customers\u27 comfort and devices\u27 integrity. The Markov chain abstraction method is used to aggregate the TCLs and describe their collective dynamics. Statistical learning techniques of hidden Markov chain analysis is used to identify the parameters of the resulting Markov chains at fixed temperature set-points. Various sensitivities are conducted to reveal the optimal Markov chain representation. To allow extracting or storing additional thermal energy, this thesis develops an Extended Markov Model(EMM) which describes devices\u27 transition when a new set-point is instructed. The results have shown that the EMM is able to capture both devices\u27 transient and steady-state behaviors under small and large set-point adjustments. Parameters heterogeneity affects the accuracy of the EMM model. In contrast to what proposed in the literature, more comprehensive heterogeneous parameters are defined and considered. The K-mean clustering approach is proposed in our analysis to minimize the heterogeneity error. Devices are divided into multiple clusters based on the power ratings and cycling characteristics. The results have shown that clustering highly improves the EMM performance and minimize the heterogeneity errors. Under temperature set-point control the TCLs\u27 aggregated power experience two main challenges before it converges to the new steady-state value, the abrupt load change, and the power oscillations. This is due to devices\u27 synchronous operations once a new operating set-point is ordered. Such power profiles may cause serious stability issues. Therefore, Model Predictive Control (MPC) with direct ON/OFF switching capability is proposed to apply the set-point control sequentially and prevent any possible power oscillations. The MPC can determine the optimal devices\u27 flow toward the new operating set-point. The results have shown that the proposed modeling and control approaches highly minimize the required switching actions. Control actions are required only during the transition between the set-points and finally converges to zero when all devices reach the new set-point setting. In contrast, the models proposed in the literature require very high switching rates which can cause damage or reducing devices\u27 life expectancy. The last part of this thesis proposes a dispatching framework to utilize the TCLs\u27 flexibility. The developed modeling and control techniques are used to support the grid with three demand response ancillary services. Namely, spinning reserves, load reduction, and load shifting. The three ancillary services are designed as demand response programs and integrated into the Security Constrained Unit Commitment (SCUC) Problem. Three participation scenarios are considered to evaluate the benefits of aggregating the TCLs in the day-ahead markets

Modeling and Control for Packetized Energy Management

The overarching goal in power systems operations is to deliver energy in an efficient, reliable, and economical manner. To achieve this objective, the traditional power system operating paradigm is for generation to follow variable demand. As electrification and decarbonization policies are pursued, the levels of variable, renewable generation will increase, which will require that power system operator think beyond supply follows demand. This means that one needs to consider the potential flexibility provided by, for instance, internet-enabled, connected, and responsive loads, which are part of the broad class of behind-the-meter distributed energy resources (DERs). The research work presented in this dissertation is concerned with coordinating large populations of distributed energy resources (DERs) for providing services to the electric grid. DERs are flexible in the sense that their power consumption can be deferred in time, because DERs store energy in some form while serving the end-use customer. For example, electric water heaters store thermal energy in the form of hot-water in the tank. Therefore, aggregate fleets of DERs are an inexpensive source of virtual energy storage that the utilities can tap into for the purpose of balancing the variability in distributed renewable generation such as solar PV, wind etc. In this work, a novel, asynchronous and randomized load coordination scheme called packetized energy management (PEM) is considered. Packetized energy management is a device-driven scheme that uses a unique request-response mechanism for coordinating diverse fleets of DERs. The aggregate dynamics of PEM are captured using state-bin transition models. Parameter heterogeneity is incorporated by grouping together relatively similar DERs. Furthermore, a notion of state of charge can be attached to the aggregate that is representative of the energy content in the fleet by means of a low order model. This low order model is of interest to the utilities and grid operators since it allows them to design control trajectories for DER aggregations depending upon grid requirements and load forecasts. Furthermore, a cyber-physical platform is developed for the validation of aggregate models and control schemes. However, PEM modifies the normal behavior of DERs and for accurate prediction of load dynamics, the underlying customer driven end-use process must be modeled to sufficient accuracy. Moreover, the modeled end-use process must be identifiable from the available data. In this work, the focus is on the uncontrollable hot-water extraction from the tank of an electric water heater. It is relevant and of interest to independent system operators (ISO) since water extraction is not usually measured and only metered interval consumption data (kWh) is collected. This is achieved by designing an estimation strategy based on a stochastic model of the end-use consumption

Online monitoring and control of voltage stability margin via machine learning-based adaptive approaches

Voltage instability or voltage collapse, observed in many blackout events, poses a significant threat to power system reliability. To prevent voltage collapse, the countermeasures suggested by the post analyses of the blackouts usually include the adoption of better online voltage stability monitoring and control tools. Recently, the variability and uncertainty imposed by the increasing penetration of renewable energy further magnifies this need. This work investigates the methodologies for online voltage stability margin (VSM) monitoring and control in the new era of smart grid and big data. It unleashes the value of online measurements and leverages the fruitful results in machine learning and demand response. An online VSM monitoring approach based on local regression and adaptive database is proposed. Considering the increasing variability and uncertainty of power system operation, this approach utilizes the locality of underlying pattern between VSM and reactive power reserve (RPR), and can adapt to the changing condition of system. LASSO (Least Absolute Shrinkage and Selection Operator) is tailored to solve the local regression problem so as to mitigate the curse of dimensionality for large-scale system. Along with the VSM prediction, its prediction interval is also estimated simultaneously in a simple but effective way, and utilized as an evidence to trigger the database updating. IEEE 30-bus system and a 60,000-bus large system are used to test and demonstrate the proposed approach. The results show that the proposed approach can be successfully employed in online voltage stability monitoring for real size systems, and the adaptivity of model and data endows the proposed approach with the advantage in the circumstances where large and unforeseen changes of system condition are inevitable. In case degenerative system conditions are identified, a control strategy is needed to steer the system back to security. A model predictive control (MPC) based framework is proposed to maintain VSM in near-real-time while minimizing the control cost. VSM is locally modeled as a linear function of RPRs based on the VSM monitoring tool, which convexifies the intricate VSM-constrained optimization problem. Thermostatically controlled loads (TCLs) are utilized through a demand response (DR) aggregator as the efficient measure to enhance voltage stability. For such an advanced application of the energy management system (EMS), plug-and-play is a necessary feature that makes the new controller really applicable in a cooperative operating environment. In this work, the cooperation is realized by a predictive interface strategy, which predicts the behaviors of relevant controllers using the simple models declared and updated by those controllers. In particular, the customer dissatisfaction, defined as the cumulative discomfort caused by DR, is explicitly constrained in respect of customers\u27 interests. This constraint maintains the applicability of the control. IEEE 30-bus system is used to demonstrate the proposed control strategy. Adaptivity and proactivity lie at the heart of the proposed approach. By making full use of real-time information, the proposed approach is competent at the task of VSM monitoring and control in a non-stationary and uncertain operating environment

Modeling and Communicating Flexibility in Smart Grids Using Artificial Neural Networks as Surrogate Models

Increasing shares of renewable energies and the transition towards electric vehicles pose major challenges to the energy system. In order to tackle these in an economically sensible way, the flexibility of distributed energy resources (DERs), such as battery energy storage systems, combined heat and power plants, and heat pumps, needs to be exploited. Modeling and communicating this flexibility is a fundamental step when trying to achieve control over DERs. The literature proposes and makes use of many different approaches, not only for the exploitation itself, but also in terms of models. In the first step, this thesis presents an extensive literature review and a general framework for classifying exploitation approaches and the communicated models. Often, the employed models only apply to specific types of DERs, or the models are so abstract that they neglect constraints and only roughly outline the true flexibility. Surrogate models, which are learned from data, can pose as generic DER models and may potentially be trained in a fully automated process. In this thesis, the idea of encoding the flexibility of DERs into ANNs is systematically investigated. Based on the presented framework, a set of ANN-based surrogate modeling approaches is derived and outlined, of which some are only applicable for specific use cases. In order to establish a baseline for the approximation quality, one of the most versatile identified approaches is evaluated in order to assess how well a set of reference models is approximated. If this versatile model is able to capture the flexibility well, a more specific model can be expected to do so even better. The results show that simple DERs are very closely approximated, and for more complex DERs and combinations of multiple DERs, a high approximation quality can be achieved by introducing buffers. Additionally, the investigated approach has been tested in scheduling tasks for multiple different DERs, showing that it is indeed possible to use ANN-based surrogates for the flexibility of DERs to derive load schedules. Finally, the computational complexity of utilizing the different approaches for controlling DERs is compared

Régulation dynamique hybride d'une grande population de systèmes thermostatiques au sein des micro-réseaux intelligents

RÉSUMÉ Dans le contexte actuel des réseaux intelligents et des micro-réseaux, les sources d’énergie renouvelables ont un rôle grandissant dans la production d’une électricité plus respectueuse de l’environnement. Cependant, comme l’utilisation de ces sources dépend fortement des conditions climatiques, la puissance produite peut présenter des fluctuations imprévisibles et difficiles à compenser. Ceci encourage le développement et l’application de la régulation dynamique de la charge (RDC) à des systèmes consommateurs d’électricité pour réduire les écarts de puissance entre l’o˙re et la demande. Dans le cadre de ce projet de recherche, l’accent est mis sur le contrôle de systèmes thermostatiques (TCLs) comme les chau˙ages ou les climatiseurs. Ceux-ci sont largement présents dans les réseaux et représentent une part importante de la consommation électrique. Ainsi, les TCLs ont le potentiel d’apporter de la flexibilité dans le réseau et font l’objet de nombreuses recherches. Ils peuvent être utilisés par exemple comme solution pour réduire les pics de consommation, réguler la fréquence du réseau ou encore minimiser le coût de l’électricité, grâce à un contrôle de leur consommation. Pour cela, on modélise une grande population hétérogène de TCLs grâce à un couple d’équations de Fokker-Planck à laquelle on applique une stratégie de contrôle hybride et sans prévision. D’une part, le contrôle mis en place permet de modifier la consigne de température de la population en réponse aux variations globales de puissance dans le réseau et ce grâce à un modèle d’équations aux dérivées partielles (EDP). D’autre part, des changements d’état forcés basés sur un processus stochastique sont appliqués à une portion de TCLs pour compenser les variations rapides et imprévisibles liées à l’utilisation de sources d’énergie renouvelables. Le contrôle hybride développé permet ainsi une régulation à deux niveaux pour minimiser les écarts de puissance entre la production et la consommation. Ce mémoire vise également à clarifier et justifier l’utilisation d’un tel contrôle à travers la simulation de divers scénarios. Les résultats obtenus montrent que la stratégie de contrôle permet à la population de TCLs de suivre à la fois les fluctuations lentes et rapides dans le réseau et réduit ainsi les écarts de puissance. Enfin, cette approche est validée par la simulation d’un micro-réseau réaliste où la production électrique est assurée par un ensemble de systèmes photovoltaïques (PV). Le contrôle hybride développé peut donc permettre aux systèmes thermostatiques de participer à la régulation dynamique de la charge et faire partie des solutions envisageables pour une meilleure gestion de la consommation électrique dans les micro-réseaux intelligents.----------ABSTRACT In the context of today’s smart grids and microgrids, renewable energy sources (RES) play an important role in producing environmentally friendly and low cost energy. However, as these sources rely significantly on weather conditions, the power produced may be subject to some unpredictable fluctuations that are hard to compensate. This motivates the development and application of dynamic demand control (DDC) to energy-consuming systems to reduce the power gap between supply and demand. The focus of this research project is put on the control of thermostatically controlled loads (TCLs) such as heaters or air conditioners. These systems are widely spread all over the electrical grids and represent a large portion of power consumption. Thus, TCLs have the potential to provide flexibility in the grid and are the subject of numerous studies. They can be used for example as a solution for peak power reduction, frequency regulation or electricity cost minimization through the control of their power consumption. To this end, a large heterogeneous population of TCLs is modeled with two Fokker-Planck equations and a non-predictive hybrid control strategy is applied. The designed control is based on a partial differential equations (PDE) model and is used to change the temperature set-point of the population in response to global power variations in the grid. Moreover, forced state switches based on a stochastic process are applied to a portion of TCLs to counter the sudden and unpredictable variations related to the use of renewables. Consequently, the hybrid control developed in this work provides a two-level regulation to minimize the power gap between production and consumption. This thesis aims also at clarifying and justifying the use of such control strategies through various simulation scenarios. The results obtained show that the control allows the TCLs to follow both slow and fast fluctuations and hence, it reduces the power discrepancies. Finally, the simulation of a realistic microgrid where the power is produced by photovoltaic (PV) cells confirms the validity of the proposed approach. The hybrid control developed in this work allows TCLs to participate in dynamic demand control and hence to be part of a viable solution for a better power consumption management in smart microgrids

Multi-agent reinforcement learning for the coordination of residential energy flexibility

This thesis investigates whether residential energy flexibility can be coordinated without sharing personal data at scale to achieve a positive impact on energy users and the grid. To tackle climate change, energy uses are being electrified at pace, just as electricity is increasingly provided by non-dispatchable renewable energy sources. These shifts increase the requirements for demand-side flexibility. Despite the potential of residential energy to provide such flexibility, it has remained largely untapped due to cost, social acceptance, and technical barriers. This thesis investigates the use of multi-agent reinforcement learning to overcome these challenges. This thesis presents a novel testing environment, which models electric vehicles, space heating, and flexible household loads in a distribution network. Additionally, a generative adversarial network-based data generator is developed to obtain realistic training and testing data. Experiments conducted in this environment showed that standard independent learners fail to coordinate in the partially observable stochastic environment. To address this, additional coordination mechanisms are proposed for agents to practise coordination in a centralised simulated rehearsal, ahead of fully decentralised implementation. Two such coordination mechanisms are proposed: optimisation-informed independent learning, and a centralised but factored critic network. In the former, agents lean from omniscient convex optimisation results ahead of fully decentralised coordination. This enables cooperation at scale where standard independent learners under partial observability could not be coordinated. In the latter, agents employ a deep neural factorisation network to learn to assess their impact on global rewards. This approach delivers comparable performance for four agents and more, with a 34-fold speed improvement for 30 agents and only first-order computational time growth. Finally, the impacts of implementing implicit coordination using these multi- agent reinforcement learning methodologies are modelled. It is observed that even without explicit grid constraint management, cooperating energy users reduce the likelihood of voltage deviations. The cooperative management of voltage constraints could be further promoted by the MARL policies, whereby their likelihood could be reduced by 43.08% relative to an uncoordinated baseline, albeit with trade-offs in other costs. However, while this thesis demonstrates the technical feasibility of MARL-based cooperation, further market mechanisms are required to reward all participants for their cooperation