NeuralHydrology -- Interpreting LSTMs in Hydrology

Despite the huge success of Long Short-Term Memory networks, their applications in environmental sciences are scarce. We argue that one reason is the difficulty to interpret the internals of trained networks. In this study, we look at the application of LSTMs for rainfall-runoff forecasting, one of the central tasks in the field of hydrology, in which the river discharge has to be predicted from meteorological observations. LSTMs are particularly well-suited for this problem since memory cells can represent dynamic reservoirs and storages, which are essential components in state-space modelling approaches of the hydrological system. On basis of two different catchments, one with snow influence and one without, we demonstrate how the trained model can be analyzed and interpreted. In the process, we show that the network internally learns to represent patterns that are consistent with our qualitative understanding of the hydrological system.Comment: Pre-print of published book chapter. See journal reference and DOI for more inf

Long short-term memory networks enhance rainfall-runoff modelling at the national scale of Denmark

This study explores the application of long short-term memory (LSTM) networks to simulate runoff at the national scale of Denmark using data from 301 catchments. This is the first LSTM application on Danish data. The results were benchmarked against the Danish national water resources model (DK-model), a physically based hydrological model. The median Kling-Gupta Efficiency (KGE), a common metric to assess performance of runoff predictions (optimum of 1), increased from 0.7 (DK-model) to 0.8 (LSTM) when trained against all catchments. Overall, the LSTM outperformed the DK-model in 80% of catchments. Despite the compelling KGE evaluation, the water balance closure was modelled less accurately by the LSTM. The applicability of LSTM networks for modelling ungauged catchments was assessed via a spatial split-sample experiment. A 20% spatial hold-out showed poorer performance of the LSTM with respect to the DK model. However, after pre-training, that is, weight initialisation obtained from training against simulated data from the DK-model, the performance of the LSTM was effectively improved. This formed a convincing argument supporting the knowledge-guided machine learning (ML) paradigm to integrate physically based models and ML to train robust models that generalise well

Deep learning for hydrological modelling: from benchmarking to concept formation

Hydrological modelling seeks to address the question: what happens to water once it falls on the land surface? Water can flow into river systems, it can pass through soils into the subsurface, it can be absorbed by the biosphere, or it can be released back into the atmosphere as evaporation. The ultimate purpose of hydrological modelling is twofold, to improve our predictions about the system of interest, and to understand how the system works. In recent decades, advances in science and technology have been made by using techniques from the field of Deep Learning, whereby flexible models are calibrated on large datasets to deduce relationships and make predictions. These techniques have begun to be applied across the environmental sciences. In this thesis I will explore a particular model architecture for deriving relationships between inputs and outputs from data, to provide accurate simulations of hydrological systems as well as to improve our understanding of the hydrological processes themselves. The Long Short-Term Memory (LSTM) is a neural network architecture from the field of Deep Learning which has shown promise for time-series modelling. This model architecture was chosen for its correspondence with our perceptual model of hydrology, whereby we consider the hydrological system to be characterised by a description of its state, and processes that govern the transfer of energy and materials from that state. This input-state-output architecture is similar in many ways to traditional process-based and conceptual models. However, unlike these models the LSTM is capable of searching a much wider range of possible functions that map inputs to outputs, capable of learning any process that can be deduced from the data, as opposed to being limited by the encoding in the traditional models. The chapters that make up this thesis first demonstrate that the LSTM is an appropriate architecture for rainfall-runoff modelling on the island of Great Britain. I trained the model using meteorological and catchment averaged attributes as input, and river discharge as outputs over a large sample of catchments. In comparison with often used conceptual model architectures, I show that the LSTM demonstrates state-of-the-art performance and justify further interrogation of what the model has learned. In a follow up study, I explore what the LSTM has learned about the hydrological system by taking the trained model weights and interpreting them with reference to intermediate stores of water that relate the meteorological inputs to the outputs of discharge. Despite the complexity of translating rainfall to discharge, hydrology is not limited to rainfall-runoff modelling. The final chapter in this thesis turns to the problem of forecasting a satellite-derived vegetation health metric which is used operationally as a proxy for drought conditions. Like rainfall-runoff modelling, the system being simulated is driven by the complex interaction of meteorological and land surface attributes, however, the target variable is now a store of water (vegetation) rather than a flux (discharge). This dissertation provides the hydrological community with three important outcomes. Firstly, the LSTM model results are provided as a benchmark for future work looking to develop a national rainfall runoff model for Great Britain. Secondly, this dissertation demonstrates a method used elsewhere in machine learning research that allows a scientist to diagnose what the LSTM has learned about the hydrological system. Finally, this dissertation demonstrates the utility of the LSTM in a drought monitoring context, forecasting a satellite derived vegetation health metric with the potential to improve the ability of national agencies to respond to drought events. Ultimately, this dissertation offers a demonstration of the power of Deep Learning models in hydrology, and calls on the community to interrogate these tools further to not only advance our predictive goals, but also our scientific ones

What Role Does Hydrological Science Play in the Age of Machine Learning?

ABSTRACT: This paper is derived from a keynote talk given at the Google's 2020 Flood Forecasting Meets Machine Learning Workshop. Recent experiments applying deep learning to rainfall‐runoff simulation indicate that there is significantly more information in large‐scale hydrological data sets than hydrologists have been able to translate into theory or models. While there is a growing interest in machine learning in the hydrological sciences community, in many ways, our community still holds deeply subjective and nonevidence‐based preferences for models based on a certain type of “process understanding” that has historically not translated into accurate theory, models, or predictions. This commentary is a call to action for the hydrology community to focus on developing a quantitative understanding of where and when hydrological process understanding is valuable in a modeling discipline increasingly dominated by machine learning. We offer some potential perspectives and preliminary examples about how this might be accomplished

Groundwater level forecasting with artificial neural networks: A comparison of long short-term memory (LSTM), convolutional neural networks (CNNs), and non-linear autoregressive networks with exogenous input (NARX)

It is now well established to use shallow artificial neural networks (ANNs) to obtain accurate and reliable groundwater level forecasts, which are an important tool for sustainable groundwater management. However, we observe an increasing shift from conventional shallow ANNs to state-of-the-art deep-learning (DL) techniques, but a direct comparison of the performance is often lacking. Although they have already clearly proven their suitability, shallow recurrent networks frequently seem to be excluded from the study design due to the euphoria about new DL techniques and its successes in various disciplines. Therefore, we aim to provide an overview on the predictive ability in terms of groundwater levels of shallow conventional recurrent ANNs, namely non-linear autoregressive networks with exogenous input (NARX) and popular state-of-the-art DL techniques such as long short-term memory (LSTM) and convolutional neural networks (CNNs). We compare the performance on both sequence-to-value (seq2val) and sequence-to-sequence (seq2seq) forecasting on a 4-year period while using only few, widely available and easy to measure meteorological input parameters, which makes our approach widely applicable. Further, we also investigate the data dependency in terms of time series length of the different ANN architectures. For seq2val forecasts, NARX models on average perform best; however, CNNs are much faster and only slightly worse in terms of accuracy. For seq2seq forecasts, mostly NARX outperform both DL models and even almost reach the speed of CNNs. However, NARX are the least robust against initialization effects, which nevertheless can be handled easily using ensemble forecasting. We showed that shallow neural networks, such as NARX, should not be neglected in comparison to DL techniques especially when only small amounts of training data are available, where they can clearly outperform LSTMs and CNNs; however, LSTMs and CNNs might perform substantially better with a larger dataset, where DL really can demonstrate its strengths, which is rarely available in the groundwater domain though

Evaluation of Snow and Streamflow in the National Water Model with Analysis using Machine Learning

Snow has great influence on land-atmosphere interactions and snowmelt from the mountains is a vital water source for downstream communities dependent on snow fed lakes, rivers and streams. This study explored the snow and streamflow prediction capabilities of process-based numerical prediction and data-driven machine learning models. The overall goal of this study was to understand the deficiencies in the NOAA’s National Water Model (NWM) to represent snow, subsequently streamflow, and recognize the areas where it could be improved for future model developments. The goal was also to evaluate if the recent advancements in machine learning techniques is useful for predicting snow in mountainous terrain where numerical prediction models such as the NWM have been known to face difficulties. The first chapter of this document will give an overview of the history of process-based hydrological models from their inception to a highly complex numerical expression of fully distributed models. It also describes the most recent evolution of more data-driven machine learning (ML) approaches such as artificial recurrent neural networks that are able to learn the non-linear input-output hydrological relationships in a catchment without explicit physical representation of the processes. This will help in understanding the vices and virtues of each method. The second chapter of this document will evaluate snow representation in the NWM through a single-column experiment and subsequently streamflow across the Aroostook River Basin in northeastern Maine, United States. It will analyze the uncertainties in the meteorological forcing data and model biases that lead to biases in the snow representation in the model simulations. The simulated streamflow estimates will be compared against the observed values from the United States Geological Survey and highlight the factors that lead to the discrepancies. The insights from the single-column and the basin study will help in the future development of the NWM or other process-based physical numerical prediction hydrological models. The third chapter of this document will evaluate the long short-term memory network machine learning technique to predict the daily snow water equivalent SWE in the Sierra Nevada basins in the western United States. This chapter takes advantage of a newly developed unique SWE reanalysis dataset, and compares SWE predicted with the ML technique to the SWE estimates from the physically based National Water Model. Snow in the Sierra Nevada range in the Western United States is critical for providing water to most parts of the south western United States, the regions that are facing severe drought since the past decade. The results of this study will be valuable towards the ongoing research on ways to more accurately estimate and predict snow stored in the mountains as SWE every season

Deep learning for vegetation health forecasting: a case study in Kenya

East Africa has experienced a number of devastating droughts in recent decades, including the 2010/2011 drought. The National Drought Management Authority in Kenya relies on real-time information from MODIS satellites to monitor and respond to emerging drought conditions in the arid and semi-arid lands of Kenya. Providing accurate and timely information on vegetation conditions and health—and its probable near-term future evolution—is essential for minimising the risk of drought conditions evolving into disasters as the country’s herders directly rely on the conditions of grasslands. Methods from the field of machine learning are increasingly being used in hydrology, meteorology, and climatology. One particular method that has shown promise for rainfall-runoff modelling is the Long Short Term Memory (LSTM) network. In this study, we seek to test two LSTM architectures for vegetation health forecasting. We find that these models provide sufficiently accurate forecasts to be useful for drought monitoring and forecasting purposes, showing competitive performances with lower resolution ensemble methods and improved performances over a shallow neural network and a persistence baseline