Given the expected high penetration of renewable energy production in future electricity systems, it is common to consider buildings as a valuable source for the provisioning of flexibility to support the power grids.
						Motivated by this concept, a wide variety of control strategies for building energy management has been proposed throughout the last decades and especially for the previously mentioned components. 
						However, these algorithms are usually implemented and evaluated for very specific settings and considerations.
						Thus, a neutral comparison, especially of performance measures, is nearly impossible.
					Inspired by recent developments in reinforcement learning research, we suggest the use of common environments (i.e. benchmarks) for filling this gap and finally propose a general concept for standardized benchmarks for the evaluation of control strategies for building energy management

Förderer, Kevin

Schmeck, Hartmut

Wölfle, David

English

KITopen

David Wölfle et al.POSTER ABSTRACT P3A Concept for Standardized Benchmarks for theEvaluation of Control Strategies for BuildingEnergy ManagementDavid Wölfle1*, Kevin Förderer1 and Hartmut Schmeck1,2*Correspondence: woelfle@fzi.de1FZI Research Center forInformation Technology,Haid-und-Neu-Str. 10-14,Karlsruhe, 76131, GermanyFull list of author information isavailable at the end of the articleAbstractGiven the expected high penetration of renewable energy production in futureelectricity systems, it is common to consider buildings as a valuable source for theprovisioning of flexibility to support the power grids. Motivated by this concept, awide variety of control strategies for building energy management has beenproposed throughout the last decades. However, these algorithms are usuallyimplemented and evaluated for very specific settings and considerations. Thus, aneutral comparison, especially of performance measures, is nearly impossible.Inspired by recent developments in reinforcement learning research, we suggestthe use of common environments (i.e. benchmarks) for filling this gap and finallypropose a general concept for standardized benchmarks for the evaluation ofcontrol strategies for building energy management.Keywords: building energy management; building control; environment;benchmark; evaluation; reinforcement learningIntroductionIn order to limit the unpredictable and potentially devastating effects of globalwarming, the governments of most countries have committed themselves to drasti-cally cut CO2 emissions by 2050 [1]. Consequently, the federal government of Ger-many has enacted the ”Klimaschutzplan” [2] (literally ”climate protection plan”)defining concrete measures to achieve this ambitious target, including a completedecarbonisation of the German electricity production and an extensive electrifica-tion of the heat and mobility sector (”Sektorenkopplung”). While it seems to beconsensus that high shares of renewable energy production will lead to an increaseddemand for energy flexibility [3, 4, 5], it also appears that buildings, in particu-lar the heating, ventilation and air conditioning (HVAC) components of these, areespecially suited to provide such energy flexibility [2, 6].Thus, it is not very surprising that the optimization of energy consumption pat-terns of buildings is a common research topic. A review on control systems forbuilding energy management [7] lists 121 publications, of which the vast majorityoptimize HVAC components. Albeit these control strategies appear unrelated to theprovisioning of energy flexibility at first glance, both concepts can be connected bydynamic pricing strategies, as suggested, for example, in [8, 9].Given the need for energy flexibility and the vast and diverse set of potentialcontrol strategies to provide such with buildings, the question arises how to evalu-ate which approach is best suited for a particular task. Following up we will thusPage 9 of 96Energy Informatics 2019, 2(Suppl 2):P3David Wölfle et al.investigate the current state of the art regarding the comparison and evaluation ofcontrol approaches for building energy management.State of the ArtWhile reviewing the existing literature, it was possible to identify qualitative andquantitative approaches for the comparison of control strategies for building energymanagement. General reviews like [7, 10, 11] use rather qualitative measures andthus focus on a meta level. The contrasting of control strategies is carried out usinggeneral topics, such as the implemented algorithms, the applied control schemas, theutilized simulation tools, and/or the considered devices. Albeit these publicationsare very useful in general, they provide no objective comparison of the achievedperformance of the reviewed control strategies.On the other hand, in publications following the second approach, it is commonthat the authors evaluate one or more control strategies in a very specific settingusing rather quantitative metrics. For example, Salpakari & Lund [12] developeda complex model consisting of a heat pump with storage, photovoltaic panels, abattery and smart appliances inspired by an existing low energy house. Using thatmodel, the authors compared a rule based based control (RBC) strategy to costoptimal behavior with respect to energy costs and self-consumption rates. BothLösch et al. [13] and Faßnacht et al. [14] developed an algorithm for the schedulingof heat pumps based on a dynamic electricity tariff. The algorithms have beencompared against the default hysteresis control strategy in a simulation with energycost as performance measure. Oldewurtel et al. [15] compared RBC with two MPCapproaches utilizing a custom developed simulation model on 1280 test cases withrespect to energy usage, thermal comfort and temperature dynamics.We were able to identify numerous publications following a similar schema to theones introduced above. However we refrain from giving a more extensive review atthis point, as it is out of this paper’s scope. Concerning the main topic of this work,the evaluation of control strategies for building energy management, we identifytwo major issues. First and although many publications use similar metrics likeenergy usage or energy costs, it is not feasible to compare different approaches ina quantitative manner. This is due to the usual procedure of not reusing the sim-ulation models of others, which may also be caused by the uncommonness of opensource publications of these. The second issue arises if one considers the provision-ing of electric flexibility through buildings on larger scales, which obviously leads tothe utilization of numerous, potentially very diverse, buildings. However, the cur-rent procedure of evaluating control approaches against one simulation model aloneleaves no indication how well the evaluated strategies generalize to other buildings.Standardized Benchmarks in Reinforcement LearningIn reinforcement learning, a field closely related to control theory [16], several bench-marks have been published for the development, evaluation and comparison of al-gorithms. These benchmarks, commonly referred to as environments, have beenwidely adopted, which allows the direct performance comparison of newly devel-oped algorithms with existing approaches, e.g. in [17, 18, 19]. Popular examplesare often based on computer games, like the Arcade Learning Environment [20]Page 10 of 96David Wölfle et al.while other environments focus on the control of robots [21] or implement ”classiccontrol” problems like balancing a pole on a cart [22]. It appears especially notewor-thy that the Arcade Learning Environment in fact consists of 57 games and thatreinforcement learning algorithms, e.g. [17, 18, 19], are often routinely evaluatedagainst all of those. The distribution of scores is thereby considered a measure forthe generalization abilities of the proposed algorithms, that is, an indication of theexpected performance on related tasks.The interaction with the environments follows the schema of observations, actionsand rewards, which is generally well suited for control problems [16] and has beenapplied successfully to challenging domains, like e.g. aerobatic helicopter flight [23].Consider the Atari game of space invaders as an example, in which the player cancontrol a space ship at the lower end of the screen and receives points for shootingalien space ships approaching from the top, which is actually part of the ArcadeLearning Environment. As common in reinforcement learning, the environment isprocessed in discrete time steps. In every time step, the environment emits a frameof the gameplay, called observation, as well as the current score referred to as reward.Observation and reward are then passed to the evaluated algorithm, which is usuallynamed agent in reinforcement learning. The agent is queried for the next action,i.e. the control input to the environment, like e.g. move left, move right, shoot ordo nothing. Once the action is passed to the environment, it will advance one timestep, emitting a new observation and reward. These steps are usually run as a loopuntil a terminal state is reached, i.e. game over, while the accumulated reward iscommonly used as metric to evaluate the performance of the algorithm.ProposalIn order to overcome the issues identified in the current State of the Art, we pro-pose the establishment of standardized and shared benchmarks for the evaluationof control strategies for building energy management following the example of en-vironments used in reinforcement learning. We propose the following:• To allow the development of building energy management approaches thatcan be applied to large number of diverse buildings, the benchmarks shouldbe a collection of distinct building simulation models. These should focus ondifferent optimization targets, e.g. heating, cooling or appliances, as well asvarying objectives like e.g. own consumption or energy costs.• All benchmarks should be published as open source projects to allowwidespread usage and verification.• The communication between algorithm and benchmarks should be standard-ized in order to limit the effort to execute a benchmark to the necessary min-imum. The interface should follow the schema of observations, rewards andactions, as it is well established and allows the usage of available reinforcementlearning algorithms for comparison.AcknowledgementsThe authors would like to thank the anonymous reviewers for their valuable feedback.FundingThis research has been partially founded by the Helmholtz Association’s Supercomputing and Big Data program.Page 11 of 96David Wölfle et al.Availability of data and materialsNot applicable.Author’s contributionsDW developed the initial concept and wrote this article. KF and HS contributed discussion, feedback and revision.Competing interestsThe authors declare that they have no competing interests.Author details1FZI Research Center for Information Technology, Haid-und-Neu-Str. 10-14, Karlsruhe, 76131, Germany.2Karlsruhe Institute of Technology, Institute AIFB, Kaiserstr. 12, Karlsruhe, 76131, Germany.References1. United Nations Framework Convention on Climate Change: Report of the Conference of the Parties on itstwenty-first session, held in Paris from 30 November to 13 December 2015 Addendum Part two: Action takenby the Conference of the Parties at its twenty-first session. Technical report, United Nations FrameworkConvention on Climate Change (2016). http://unfccc.int/resource/docs/2015/cop21/eng/10a01.pdf2. Bundesministerium für Umwelt, Naturschutz, B.u.R.B.: Klimaschutzplan 2050 – KlimaschutzpolitischeGrundsätze und Ziele der Bundesregierung. Technical report (2017).https://www.bmu.de/fileadmin/Daten_BMU/Download_PDF/Klimaschutz/klimaschutzplan_2050_bf.pdf3. Kondziella, H., Bruckner, T.: Flexibility requirements of renewable energy based electricity systems – a reviewof research results and methodologies. Renewable and Sustainable Energy Reviews 53, 10–22 (2016)4. Papaefthymiou, G., Dragoon, K.: Towards 100% renewable energy systems: Uncapping power system flexibility.Energy Policy 92, 69–82 (2016)5. Alizadeh, M.I., Parsa Moghaddam, M., Amjady, N., Siano, P., Sheikh-El-Eslami, M.K.: Flexibility in futurepower systems with high renewable penetration: A review. Renewable and Sustainable Energy Reviews 57,1186–1193 (2016)6. Jensen, S.Ø., Marszal-Pomianowska, A., Lollini, R., Pasut, W., Knotzer, A., Engelmann, P., Stafford, A.,Reynders, G.: IEA EBC Annex 67 Energy Flexible Buildings. Energy and Buildings 155, 25–34 (2017)7. Shaikh, P.H., Nor, N.B.M., Nallagownden, P., Elamvazuthi, I., Ibrahim, T.: A review on optimized controlsystems for building energy and comfort management of smart sustainable buildings. Renewable andSustainable Energy Reviews 34, 409–429 (2014)8. Samadi, P., Mohsenian-Rad, H., Wong, V.W.S., Schober, R.: Real-Time Pricing for Demand Response Basedon Stochastic Approximation. IEEE Transactions on Smart Grid 5(2), 789–798 (2014)9. Lübkert, T., Venzke, M., Turau, V.: Calculating retail prices from demand response target schedules to operatedomestic electric water heaters. Energy Informatics 1(S1), 31 (2018)10. Wang, S., Ma, Z.: Supervisory and Optimal Control of Building HVAC Systems: A Review. HVAC&R Research14(1), 3–32 (2008)11. Dounis, A.I., Caraiscos, C.: Advanced control systems engineering for energy and comfort management in abuilding environment—A review. Renewable and Sustainable Energy Reviews 13(6-7), 1246–1261 (2009)12. Salpakari, J., Lund, P.: Optimal and rule-based control strategies for energy flexibility in buildings with PV.Applied Energy 161 (2016)13. Loesch, M., Hufnagel, D., Steuer, S., Fabnacht, T., Schmeck, H.: Demand side management in smart buildingsby intelligent scheduling of heat pumps. In: 2014 IEEE International Conference on Intelligent Energy andPower Systems (IEPS), pp. 1–6 (2014). http://ieeexplore.ieee.org/document/6874181/14. Faßnacht, T., Loesch, M., Wagner, A.: Simulation Study of a Heuristic Predictive Optimization Scheme forGrid-Reactive Heat Pump Operation. In: Proceedings of REHVA Annual Conference ”Advanced HVAC andNatural Gas Technologies”, vol. 0, p. 132. Riga Technical University, Riga (2015).https://journals.rtu.lv/index.php/AHNGT/article/view/rehvaconf.2015.01915. Oldewurtel, F., Parisio, A., Jones, C.N., Gyalistras, D., Gwerder, M., Stauch, V., Lehmann, B., Morari, M.: Useof model predictive control and weather forecasts for energy efficient building climate control. Energy andBuildings 45, 15–27 (2012)16. Sutton, R.S., Barto, A.: Reinforcement Learning - An Introduction, 2nd edn., p. 526 (2018).http://incompleteideas.net/book/the-book.html17. Gruslys, A., Dabney, W., Azar, M.G., Piot, B., Bellemare, M., Munos, R.: The Reactor: A fast andsample-efficient Actor-Critic agent for Reinforcement Learning (2017). 1704.0465118. Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T.P., Harley, T., Silver, D., Kavukcuoglu, K.:Asynchronous Methods for Deep Reinforcement Learning (2016). 1602.0178319. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M.,Fidjeland, A.K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D.,Wierstra, D., Legg, S., Hassabis, D.: Human-level control through deep reinforcement learning. Nature518(7540), 529–533 (2015)20. G. Bellemare, M., Naddaf, Y., Veness, J., Bowling, M.: The Arcade Learning Environment: An EvaluationPlatform for General Agents. Journal of Artificial Intelligence Research 47 (2012)21. Plappert, M., Andrychowicz, M., Ray, A., McGrew, B., Baker, B., Powell, G., Schneider, J., Tobin, J., Chociej,M., Welinder, P., Kumar, V., Zaremba, W.: Multi-Goal Reinforcement Learning: Challenging RoboticsEnvironments and Request for Research (2018). 1802.0946422. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W.: OpenAI Gym(2016)23. Abbeel, P., Coates, A., Quigley, M., Ng, A.Y.: An Application of Reinforcement Learning to AerobaticHelicopter Flight. Advances in Neural Information Processing Systems 19 (NIPS 2006) (2006)Page 12 of 96

A Concept for Standardized Benchmarks for the Evaluation of Control Strategies for Building Energy Management

https://publikationen.bibliothek.kit.edu/1000104350/63576760

A Concept for Standardized Benchmarks for the Evaluation of Control Strategies for Building Energy Management

Abstract

Similar works

Full text

Available Versions

KITopen