Integrating Inverse Reinforcement Learning and Direct Policy Search for Modeling Multipurpose Water Reservoir Systems

System identification and optimal control have always contributed to water resources systems planning and management. Although water control problems are commonly formulated as multi-objective Markov Decision Processes, accurately modeling reservoir systems controlled by human operators remains challenging due to the absence of a formal definition of the objective function guiding their behavior. In this letter, we introduce a mixed Reinforcement Learning approach to model the dynamics of multipurpose reservoir systems. Specifically, our method first uses Inverse Reinforcement Learning to extract the tradeoff among competing objectives from historical observations of the reservoir system dynamics. The identified objective function is then used in the formulation of an optimal control problem returning a closed-loop policy which allows the simulation of the observed dynamics of the reservoir system. We demonstrate the potential of the proposed method in a real-world application involving the multipurpose regulation of Lake Como in northern Italy. Results show that our approach effectively infers the tradeoff between flood control and water supply adopted in the observed system's operation, and yields a control policy that closely approximates the observed system dynamics

Exploring the Climate Puzzle: A Surprising Twist in Fighting Climate Change

Sometimes, when scientists try to help people, they can end up with a surprise ending in which things do not work out as expected. Their “help” might even accidentally make the situation worse for some people. We wanted to know if this could be true for a strategy to slow down climate change: charging countries a fee when they cut down forests to create farmland. We used computers to predict what might happen if countries were charged different fees, to keep things fair. Specifically, countries with less money would only have to pay low (or no) fees, while rich countries would pay higher fees. However, our computer model showed that this plan could have unexpected negative consequences for water availability in some places that pay low fees, like certain regions in Africa. This tells us that, as we fight climate change, we must keep our eyes open for unintended consequences that could result from our attempts to help the planet

Partitioning the impacts of streamflow and evaporation uncertainty on the operations of multipurpose reservoirs in arid regions

Ongoing changes in global climate are expected to alter the hydrologic regime of many river basins worldwide, expanding historically observed variability as well as increasing the frequency and intensity of extreme events. Understanding the vulnerabilities of water systems under such uncertain and variable hydrologic conditions is key to supporting strategic planning and design adaptation options. In this paper, we contribute a multiobjective assessment of the impacts of hydrologic uncertainty on the operations of multipurpose water reservoirs systems in arid climates. We focus our analysis on the Dez and Karoun river system in Iran, which is responsible for the production of more than 20% of the total hydropower generation of the country. A system of dams controls most of the water flowing to the lower part of the basin, where irrigation and domestic supply are strategic objectives, along with flood protection.We first design the optimal operations of the system using observed inflows and evaporation rates. Then, we simulate the resulting solutions over different ensembles of stochastic hydrology to partition the impacts of streamflow and evaporation uncertainty. Numerical results show that system operations are extremely sensitive to alterations of both uncertainty sources. In particular, we show that in this arid river basin, long-term objectives are mainly vulnerable to inflow uncertainty, whereas evaporation rate uncertainty mostly affects short-term objectives. Our results suggest that local water authorities should properly characterize hydrologic uncertainty in the design of future operations of the expanded network of reservoirs, possibly also investing in the improvement of the existing monitoring network to obtain more reliable data for modeling streamflow and evaporation processes

Improving the Protection of Aquatic Ecosystems by Dynamically Constraining Reservoir Operation Via Direct Policy Conditioning

Water management problems generally involve conflicting and non-commensurable objectives. Assuming a centralized perspective at the system-level, the set of Pareto-optimal alternatives represents the ideal solution of most of the problems. Yet, in typical real-world applications, only a few primary objectives are explicitly considered, taking precedence over all other concerns. These remaining concerns are then internalized as static constraints within the problem's formulation. This approach yields to solutions that fail to explore the full set of objectives tradeoffs. In this paper, we propose a novel method, called direct policy conditioning (DPC), that combines direct policy search, multi-objective evolutionary algorithms, and input variable selection to design dynamic constraints that change according to the current system conditions. The method is demonstrated for the management problem of the Conowingo Dam, located within the Lower Susquehanna River, USA. The DPC method is used to identify environmental protection mechanisms and is contrasted with traditional static constraints de fining minimum environmental flow requirements. Results show that the DPC method identifies a set of dynamically constrained control policies that overcome the current alternatives based on the minimum environmental flow constraint, in terms of environmental protection but also of the primary objectives

Effects of olive and pomegranate by-products on human microbiota : a study using the SHIME (R) in vitro simulator

Two by-products containing phenols and polysaccharides, a "pate" (OP) from the extra virgin olive oil milling process and a decoction of pomegranate mesocarp (PM), were investigated for their effects on human microbiota using the SHIME (R) system. The ability of these products to modulate the microbial community was studied simulating a daily intake for nine days. Microbial functionality, investigated in terms of short chain fatty acids (SCFA) and NH4+, was stable during the treatment. A significant increase in Lactobacillaceae and Bifidobacteriaceae at nine days was induced by OP mainly in the proximal tract. Polyphenol metabolism indicated the formation of tyrosol from OP mainly in the distal tract, while urolithins C and A were produced from PM, identifying the human donor as a metabotype A. The results confirm the SHIME (R) system as a suitable in vitro tool to preliminarily investigate interactions between complex botanicals and human microbiota before undertaking more challenging human studies

Using crowdsourced web content for informing water systems operations in snow-dominated catchments

Snow is a key component of the hydrologic cycle in many regions of the world. Despite recent advances in environmental monitoring that are making a wide range of data available, continuous snow monitoring systems that can collect data at high spatial and temporal resolution are not well established yet, especially in inaccessible high-latitude or mountainous regions. The unprecedented availability of user-generated data on the web is opening new opportunities for enhancing real-time monitoring and modeling of environmental systems based on data that are public, low-cost, and spatiotemporally dense. In this paper, we contribute a novel crowdsourcing procedure for extracting snow-related information from public web images, either produced by users or generated by touristic webcams. A fully automated process fetches mountain images from multiple sources, identifies the peaks present therein, and estimates virtual snow indexes representing a proxy of the snow-covered area. Our procedure has the potential for complementing traditional snow-related information, minimizing costs and efforts for obtaining the virtual snow indexes and, at the same time, maximizing the portability of the procedure to several locations where such public images are available. The operational value of the obtained virtual snow indexes is assessed for a real-world water-management problem, the regulation of Lake Como, where we use these indexes for informing the daily operations of the lake. Numerical results show that such information is effective in extending the anticipation capacity of the lake operations, ultimately improving the system performance

Learning-based hierarchical control of water reservoir systems

The optimal control of a water reservoir systems represents a challenging problem, due to uncertain hydrologic inputs and the need to adapt to changing environment and varying control objectives. In this work, we propose a real-time learning-based control strategy based on a hierarchical predictive control architecture. Two control loops are implemented: the inner loop is aimed to make the overall dynamics similar to an assigned linear through data-driven control design, then the outer economic model-predictive controller compensates for model mismatches, enforces suitable constraints, and boosts the tracking performance. The effectiveness of the proposed approach as compared to traditional dynamic programming strategies is illustrated on an accurate simulator of the Hoa Binh reservoir in Vietnam. Results show that the proposed approach performs better than the one based on stochastic dynamic programming

A FOSS Based Web Geo- Service Architecture For Data Management In Complex Water Resources Contexts

Advances in environmental monitoring systems from remote sensing to pervasive real and virtual sensor networks are enlarging the amount and types of data available at local and global scale at increasingly higher temporal and spatial resolution. However, accessing and integrating these data for modeling and operational purposes can be challenging and highly time consuming, particularly in complex physical and institutional contexts, where data are from different sources. This research focuses on the design of a web geo- service architecture, based on Free and Open Source Software (FOSS), to enable collection and sharing of data coming from complex water resources domains and managed by multiple institutions. The heterogeneous nature of these data requires the combination of different geospatial data servers (Catalog Service for the Web, Web Map Service, Web Feature service, Web Coverage Service, Sensor Observations Service,) and interface technologies that enable interoperability of all complex resources data types. This is a key feature of web geo- service tools in multidata and multiowners environment. Besides the storage of the available hydrological data according to the Open Geospatial Consortium standards, the architecture provides a platform for comparatively analyzing alternative water supply and demand management strategies. The architecture is developed for the Lake Como system (Italy), a regulated lake serving multiple and often competing water uses (irrigation, hydropower, flood control) in northern Italy. . This research gives important insights on currently operating GEOSS (Global Earth Observation System of Systems) architectures, demonstrating that Spatial Data Infrastructures using FOSS are a feasible and effective alternative to data and metadata collection, storage, sharing and visualization in complex water resources management contexts, using open international standards