A Multi-Agent Reinforcement Learning Approach to Price and Comfort Optimization in HVAC-Systems

This paper addresses the challenge of minimizing training time for the control of Heating, Ventilation, and Air-conditioning (HVAC) systems with online Reinforcement Learning (RL). This is done by developing a novel approach to Multi-Agent Reinforcement Learning (MARL) to HVAC systems. In this paper, the environment formed by the HVAC system is formulated as a Markov Game (MG) in a general sum setting. The MARL algorithm is designed in a decentralized structure, where only relevant states are shared between agents, and actions are shared in a sequence, which are sensible from a system’s point of view. The simulation environment is a domestic house located in Denmark and designed to resemble an average house. The heat source in the house is an air-to-water heat pump, and the HVAC system is an Underfloor Heating system (UFH). The house is subjected to weather changes from a data set collected in Copenhagen in 2006, spanning the entire year except for June, July, and August, where heat is not required. It is shown that: (1) When comparing Single Agent Reinforcement Learning (SARL) and MARL, training time can be reduced by 70% for a four temperature-zone UFH system, (2) the agent can learn and generalize over seasons, (3) the cost of heating can be reduced by 19% or the equivalent to 750 kWh of electric energy per year for an average Danish domestic house compared to a traditional control method, and (4) oscillations in the room temperature can be reduced by 40% when comparing the RL control methods with a traditional control method

A learning-based approach towards the data-driven predictive control of combined wastewater networks - An experimental study

Smart control in water systems aims to reduce the cost of infrastructure expansion by better utilizing the available capacity through real-time control. The recent availability of sensors and advanced data processing is expected to transform the view of water system operators, increasing the need for deploying a new generation of data-driven control solutions. To that end, this paper proposes a data-driven control framework for combined wastewater and stormwater networks. We propose to learn the effect of wet- and dry-weather flows through the variation of water levels by deploying a number of level sensors in the network. To tackle the challenges associated with combining hydraulic and hydrologic modelling, we adopt a Gaussian process-based predictive control tool to capture the dynamic effect of rain and wastewater inflows, while applying domain knowledge to preserve the balance of water volumes. To show the practical feasibility of the approach, we test the control performance on a laboratory setup, inspired by the topology of a real-world wastewater network. We compare our method to a rule-based controller currently used by the water utility operating the proposed network. Overall, the controller learns the wastewater load and the temporal dynamics of the network, and therefore significantly outperforms the baseline controller, especially during high-intensity rain periods. Finally, we discuss the benefits and drawbacks of the approach for practical real-time control implementations.Peer ReviewedPostprint (published version

Luxury and legacy effects on urban biodiversity, vegetation cover and ecosystem services

Unidad de excelencia María de Maeztu CEX2019-000940-MSocio-economic and historical drivers shape urban nature distribution and characteristics, as luxury (wealth-related) and legacy (historical management) effects. Using remote sensing and census data on biodiversity and socio-economic indicators, we examined these effects on urban biodiversity and vegetation cover in Vitoria-Gasteiz (Basque Country). We also tested the luxury and legacy hypotheses on regulating ecosystem services (ES) and explored predictor interactions. Higher educational attainment positively correlated with urban biodiversity, confirming the luxury effect, but had no effect on vegetation cover or ES. Older areas had higher vegetation cover and ES evidencing a legacy effect with an inverse response on biodiversity, attributable to more recent management strategies promoting biodiversity in green spaces. Habitat quality amplified the luxury effect, while population density strengthened the legacy effect. Our results suggest that urban biodiversity is mainly driven by socio-economic factors, while vegetation cover and ES are influenced by management legacies in interaction with population density