Designing a generalised reward for Building Energy Management Reinforcement Learning agents

The reduction of the carbon footprint of buildings is a challenging task, partly due to the conflicting goals of maximising occupant comfort and minimising energy consumption. An intelligent management of Heating, Ventilation and Air Conditioning (HVAC) systems is creating a promising research line in which the creation of suitable algorithms could reduce energy consumption maintaining occupants' comfort. In this regard, Reinforcement Learning (RL) approaches are giving a good balance between data requirements and intelligent operations to control building systems. However, there is a gap concerning how to create a generalised reward signal that can train RL agents without delimiting the problem to a specific or controlled scenario. To tackle it, an analysis and discussion is presented about the necessary requirements for the creation of generalist rewards, with the objective of laying the foundations that allow the creation of generalist intelligent agents for building energy management.The work described in this paper was partially supported by the Basque Government under ELKARTEK project (LANTEGI4.0 KK-2020/00072)

B2RL: An open-source Dataset for Building Batch Reinforcement Learning

Batch reinforcement learning (BRL) is an emerging research area in the RL community. It learns exclusively from static datasets (i.e. replay buffers) without interaction with the environment. In the offline settings, existing replay experiences are used as prior knowledge for BRL models to find the optimal policy. Thus, generating replay buffers is crucial for BRL model benchmark. In our B2RL (Building Batch RL) dataset, we collected real-world data from our building management systems, as well as buffers generated by several behavioral policies in simulation environments. We believe it could help building experts on BRL research. To the best of our knowledge, we are the first to open-source building datasets for the purpose of BRL learning

State-of-the-art on research and applications of machine learning in the building life cycle

Fueled by big data, powerful and affordable computing resources, and advanced algorithms, machine learning has been explored and applied to buildings research for the past decades and has demonstrated its potential to enhance building performance. This study systematically surveyed how machine learning has been applied at different stages of building life cycle. By conducting a literature search on the Web of Knowledge platform, we found 9579 papers in this field and selected 153 papers for an in-depth review. The number of published papers is increasing year by year, with a focus on building design, operation, and control. However, no study was found using machine learning in building commissioning. There are successful pilot studies on fault detection and diagnosis of HVAC equipment and systems, load prediction, energy baseline estimate, load shape clustering, occupancy prediction, and learning occupant behaviors and energy use patterns. None of the existing studies were adopted broadly by the building industry, due to common challenges including (1) lack of large scale labeled data to train and validate the model, (2) lack of model transferability, which limits a model trained with one data-rich building to be used in another building with limited data, (3) lack of strong justification of costs and benefits of deploying machine learning, and (4) the performance might not be reliable and robust for the stated goals, as the method might work for some buildings but could not be generalized to others. Findings from the study can inform future machine learning research to improve occupant comfort, energy efficiency, demand flexibility, and resilience of buildings, as well as to inspire young researchers in the field to explore multidisciplinary approaches that integrate building science, computing science, data science, and social science

BEAR: Physics-Principled Building Environment for Control and Reinforcement Learning

Recent advancements in reinforcement learning algorithms have opened doors for researchers to operate and optimize building energy management systems autonomously. However, the lack of an easily configurable building dynamical model and energy management task simulation and evaluation platform has arguably slowed the progress in developing advanced and dedicated reinforcement learning (RL) and control algorithms for building operation tasks. Here we propose "BEAR", a physics-principled Building Environment for Control And Reinforcement Learning. The platform allows researchers to benchmark both model-based and model-free controllers using a broad collection of standard building models in Python without co-simulation using external building simulators. In this paper, we discuss the design of this platform and compare it with other existing building simulation frameworks. We demonstrate the compatibility and performance of BEAR with different controllers, including both model predictive control (MPC) and several state-of-the-art RL methods with two case studies.Comment: Accepted at ACM e-Energy 2023; Code available at https://github.com/chz056/BEA

An investigation into the energy and control implications of adaptive comfort in a modern office building

PhD ThesisAn investigation into the potentials of adaptive comfort in an office building is carried out using fine grained primary data and computer modelling. A comprehensive literature review and background study into energy and comfort aspects of building management provides the backdrop against which a target building is subjected to energy and comfort audit, virtual simulation and impact assessment of adaptive comfort standard (BS EN 15251: 2007). Building fabric design is also brought into focus by examining 2006 and 2010 Approved Document part L potentials against Passive House design. This is to reflect the general direction of regulatory development which tends toward zero carbon design by the end of this decade. In finishing a study of modern controls in buildings is carried out to assess the strongest contenders that next generation heating, ventilation and air-conditioning technologies will come to rely on in future buildings. An actual target building constitutes the vehicle for the work described above. A virtual model of this building was calibrated against an extensive set of actual data using version control method. The results were improved to surpass ASHRAE Guide 14. A set of different scenarios were constructed to account for improved fabric design as well as historical weather files and future weather predictions. These scenarios enabled a comparative study to investigate the effect of BS EN 15251:2007 when compared to conventional space controls. The main finding is that modern commercial buildings built to the latest UK statutory regulations can achieve considerable carbon savings through adaptive comfort standard. However these savings are only modestly improved if fabric design is enhanced to passive house levels. Adaptive comfort can also be readily deployed using current web-enabled control applications. However an actual field study is necessary to provide invaluable insight into occupants’ acceptance of this standard since winter-time space temperature results derived from BS EN 15251:2007 constitute a notable departure from CIBSE environmental guidelines

Neural network based predictive control of personalized heating systems

The aim of a personalized heating system is to provide a desirable microclimate for each individual when heating is needed. In this paper, we present a method based on machine learning algorithms for generation of predictive models for use in control of personalized heating systems. Data was collected from two individual test subjects in an experiment that consisted of 14 sessions per test subject with each session lasting 4 h. A dynamic recurrent nonlinear autoregressive neural network with exogenous inputs (NARX) was used for developing the models for the prediction of personalized heating settings. The models for subjects A and B were tested with the data that was not used in creating the neural network (unseen data) to evaluate the accuracy of the prediction. Trained NARX showed good performance when tested with the unseen data, with no sign of overfitting. For model A, the optimal network was with 12 hidden neurons with root mean square error equal to 0.043 and Pearson correlation coefficient equal to 0.994. The best result for model B was obtained with a neural network with 16 hidden neurons with root mean square error equal to 0.049 and Pearson correlation coefficient equal to 0.966. In addition to the neural network models, several other machine learning algorithms were tested. Furthermore, the models were on-line tested and the results showed that the test subjects were satisfied with the heating settings that were automatically controlled using the models. Tests with automatic control showed that both test subjects felt comfortable throughout the tests and test subjects expressed their satisfaction with the automatic control