Data-driven control of micro-climate in buildings: an event-triggered reinforcement learning approach

Smart buildings have great potential for shaping an energy-efficient, sustainable, and more economic future for our planet as buildings account for approximately 40% of the global energy consumption. Future of the smart buildings lies in using sensory data for adaptive decision making and control that is currently gloomed by the key challenge of learning a good control policy in a short period of time in an online and continuing fashion. To tackle this challenge, an event-triggered -- as opposed to classic time-triggered -- paradigm, is proposed in which learning and control decisions are made when events occur and enough information is collected. Events are characterized by certain design conditions and they occur when the conditions are met, for instance, when a certain state threshold is reached. By systematically adjusting the time of learning and control decisions, the proposed framework can potentially reduce the variance in learning, and consequently, improve the control process. We formulate the micro-climate control problem based on semi-Markov decision processes that allow for variable-time state transitions and decision making. Using extended policy gradient theorems and temporal difference methods in a reinforcement learning set-up, we propose two learning algorithms for event-triggered control of micro-climate in buildings. We show the efficacy of our proposed approach via designing a smart learning thermostat that simultaneously optimizes energy consumption and occupants' comfort in a test building

State-of-the-art on research and applications of machine learning in the building life cycle

Fueled by big data, powerful and affordable computing resources, and advanced algorithms, machine learning has been explored and applied to buildings research for the past decades and has demonstrated its potential to enhance building performance. This study systematically surveyed how machine learning has been applied at different stages of building life cycle. By conducting a literature search on the Web of Knowledge platform, we found 9579 papers in this field and selected 153 papers for an in-depth review. The number of published papers is increasing year by year, with a focus on building design, operation, and control. However, no study was found using machine learning in building commissioning. There are successful pilot studies on fault detection and diagnosis of HVAC equipment and systems, load prediction, energy baseline estimate, load shape clustering, occupancy prediction, and learning occupant behaviors and energy use patterns. None of the existing studies were adopted broadly by the building industry, due to common challenges including (1) lack of large scale labeled data to train and validate the model, (2) lack of model transferability, which limits a model trained with one data-rich building to be used in another building with limited data, (3) lack of strong justification of costs and benefits of deploying machine learning, and (4) the performance might not be reliable and robust for the stated goals, as the method might work for some buildings but could not be generalized to others. Findings from the study can inform future machine learning research to improve occupant comfort, energy efficiency, demand flexibility, and resilience of buildings, as well as to inspire young researchers in the field to explore multidisciplinary approaches that integrate building science, computing science, data science, and social science

Development of a Soft Actor Critic Deep Reinforcement Learning Approach for Harnessing Energy Flexibility in a Large Office Building

This research is concerned with the novel application and investigation of `Soft Actor Critic' (SAC) based Deep Reinforcement Learning (DRL) to control the cooling setpoint (and hence cooling loads) of a large commercial building to harness energy flexibility. The research is motivated by the challenge associated with the development and application of conventional model-based control approaches at scale to the wider building stock. SAC is a model-free DRL technique that is able to handle continuous action spaces and which has seen limited application to real-life or high-fidelity simulation implementations in the context of automated and intelligent control of building energy systems. Such control techniques are seen as one possible solution to supporting the operation of a smart, sustainable and future electrical grid. This research tests the suitability of the SAC DRL technique through training and deployment of the agent on an EnergyPlus based environment of the office building. The SAC DRL was found to learn an optimal control policy that was able to minimise energy costs by 9.7% compared to the default rule-based control (RBC) scheme and was able to improve or maintain thermal comfort limits over a test period of one week. The algorithm was shown to be robust to the different hyperparameters and this optimal control policy was learnt through the use of a minimal state space consisting of readily available variables. The robustness of the algorithm was tested through investigation of the speed of learning and ability to deploy to different seasons and climates. It was found that the SAC DRL requires minimal training sample points and outperforms the RBC after three months of operation and also without disruption to thermal comfort during this period. The agent is transferable to other climates and seasons although further retraining or hyperparameter tuning is recommended.Comment: submitted to Energy and A

Hill of Banchory Geothermal Energy Project Feasibility Study Report

This feasibility study explored the potential for a deep geothermal heat project at Hill of Banchory, Aberdeenshire. The geology of the Hill of Fare, to the north of Banchory, gives cause to believe it has good geothermal potential, while the Hill of Banchory heat network, situated on the northern side of the town, offers a ready-made heat customer. The partners in the consortium consisted of academics and developers with relevant expertise in deep geothermal energy, heat networks, and financial analysis, together with representatives of local Government. They conducted geological fieldwork around the Hill of Fare, engaged with local residents to establish their attitudes to geothermal energy, and built business models to predict the conditions under which the heat network at Hill of Banchory would be commercial if it utilised heat from the proposed geothermal well. They also estimated the potential carbon emission reductions that could be achieved by using deep geothermal energy, both at Hill of Banchory and more widely

Geothermal Energy: Delivering on the Global Potential

After decades of being largely the preserve of countries in volcanic regions, the use of geothermal energy—for both heat and power applications—is now expanding worldwide. This reflects its excellent low-carbon credentials and its ability to offer baseload and dispatchable output - rare amongst the mainstream renewables. Yet uptake of geothermal still lags behind that of solar and wind, principally because of (i) uncertainties over resource availability in poorly-explored reservoirs and (ii) the concentration of full-lifetime costs into early-stage capital expenditure (capex). Recent advances in reservoir characterization techniques are beginning to narrow the bounds of exploration uncertainty, both by improving estimates of reservoir geometry and properties, and by providing pre-drilling estimates of temperature at depth. Advances in drilling technologies and management have potential to significantly lower initial capex, while operating expenditure is being further reduced by more effective reservoir management — supported by robust mathematical models — and increasingly efficient energy conversion systems (flash, binary and combined-heat-and-power). Advances in characterization and modelling are also improving management of shallow low-enthalpy resources that can only be exploited using heat-pump technology. Taken together with increased public appreciation of the benefits of geothermal, the technology is finally ready to take its place as a mainstream renewable technology, This book draws together some of the latest developments in concepts and technology that are enabling the growing realisation of the global potential of geothermal energy in all its manifestations.After decades of being largely the preserve of countries in volcanic regions, the use of geothermal energy—for both heat and power applications—is now expanding worldwide. This reflects its excellent low-carbon credentials and its ability to offer baseload and dispatchable output - rare amongst the mainstream renewables. Yet uptake of geothermal still lags behind that of solar and wind, principally because of (i) uncertainties over resource availability in poorly-explored reservoirs and (ii) the concentration of full-lifetime costs into early-stage capital expenditure (capex). Recent advances in reservoir characterization techniques are beginning to narrow the bounds of exploration uncertainty, both by improving estimates of reservoir geometry and properties, and by providing pre-drilling estimates of temperature at depth. Advances in drilling technologies and management have potential to significantly lower initial capex, while operating expenditure is being further reduced by more effective reservoir management — supported by robust mathematical models — and increasingly efficient energy conversion systems (flash, binary and combined-heat-and-power). Advances in characterization and modelling are also improving management of shallow low-enthalpy resources that can only be exploited using heat-pump technology. Taken together with increased public appreciation of the benefits of geothermal, the technology is finally ready to take its place as a mainstream renewable technology