A Method for Upscaling In Situ Soil Moisture Measurements to Satellite Footprint Scale Using Random Forests

Geophysical products generated from remotely sensed data require validation to evaluate their accuracy. Typically in situ measurements are used for validation, as is the case for satellite-derived soil moisture products. However, a large disparity in scales often exists between in situ measurements (covering meters to 10 s of meters) and satellite footprints (often hundreds of meters to several kilometers), making direct comparison difficult. Before using in situ measurements for validation, they must be “upscaled” to provide the mean soil moisture within the satellite footprint. There are a number of existing upscaling methods previously applied to soil moisture measurements, but many place strict requirements on the number and spatial distribution of soil moisture sensors difficult to achieve with permanent/semipermanent ground networks necessary for long-term validation efforts. A new method for upscaling is presented here, using Random Forests to fit a model between in situ measurements and a number of landscape parameters and variables impacting the spatial and temporal distributions of soil moisture. The method is specifically intended for validation of the NASA soil moisture active passive (SMAP) products at 36-, 9-, and 3-km scales. The method was applied to in situ data from the SoilSCAPE network in California, validated with data from the SMAPVEX12 campaign in Manitoba, Canada with additional verification from the TxSON network in Texas. For the SMAPVEX12 site, the proposed method was compared to extensive field measurements and was able to predict mean soil moisture over a large area more accurately than other upscaling approaches

Mapping Soil Moisture from Remotely Sensed and In-situ Data with Statistical Methods

Soil moisture is an important factor for accurate prediction of agricultural productivity and rainfall runoff with hydrological models. Remote sensing satellites such as Soil Moisture Active Passive (SMAP) offer synoptic views of soil moisture distribution at a regional-to-global scale. To use the soil moisture product from these satellites, however, requires a downscaling of the data from an usually large instantaneous field of view (i.e. 36 km) to the watershed analysis scales ranging from 30 m to 1 km. In addition, validation of the soil moisture products using the ground station observations without an upscaling treatment would lead to cross-level fallacy. In the literature of geographical analysis, scale is one of the top research concens because of the needs for multi-source geospatial data fusion. This dissertation research introduced a multi-level soil moisture data assimilation and processing methodology framework based on spatial information theories. The research contains three sections: downscaling using machine learning and geographically weighted regression, upscaling ground network observation to calibrate satellite data, and spatial and temporal multi-scale data assimilation using spatio-temporal interpolation. (1) Soil moisture downscaling In the first section, a downscaling method is designed using 1-km geospatial data to obtain subpixel soil moisture from the 9-km soil moisture product of the SMAP satellite. The geospatial data includes normalized difference vegetation index (NDVI), land surface temperature (LST), gross primary productivity (GPP), topographical moisture index (TMI), with all resampled to 1-km resolution. The machine learning algorithm – random forest was used to create a prediction model of the soil moisture at a 1-km resolution. The 1-km soil moisture product was compared with the ground samples from the West Texas Mesonet (WTM) station data. The residual was then interpolated to compensate the unpredicted variability of the model. The entire process was based on the concept of regression kriging- where the regression was done by the random forest model. Results show that the downscaling approach was able to achieve better accuracy than the current statistical downscaling methods. (2) Station network data upscaling The Texas Soil Observation Network (TxSON) network was designed to test the feasibility of upscaling the in-situ data to match the scale of the SMAP data. I advanced the upscaling method by using the Voronoi polygons and block kriging with a Gaussian kernel aggregation. The upscaling algorithm was calibrated using different spatial aggregation parameters, such as the fishnet cell size and Gaussian kernel standard deviation. The use of the kriging can significantly reduce the spatial autocorrelation among the TxSON stations because of its declustering ability. The result proved the new upscaling method was better than the traditional ones. (3) Multi-scale data fusion in a spatio-temporal framework None of the current works for soil moisture statistical downscaling honors time and space equally. It is important, however, that the soil moisture products are consistent in both domains. In this section, the space-time kriging model for soil moisture downscaling and upscaling computation framework designed in the last two sections is implemented to create a spatio-temporal integrated solution to soil moisture multi-scale mapping. The present work has its novelty in using spatial statistics to reconcile the scale difference from satellite data and ground observations, and therefore proposes new theories and solutions for dealing with the modifiable areal unit problem (MAUP) incurred in soil moisture mapping from satellite and ground stations

A review of spatial downscaling of satellite remotely sensed soil moisture

Satellite remote sensing technology has been widely used to estimate surface soil moisture. Numerous efforts have been devoted to develop global soil moisture products. However, these global soil moisture products, normally retrieved from microwave remote sensing data, are typically not suitable for regional hydrological and agricultural applications such as irrigation management and flood predictions, due to their coarse spatial resolution. Therefore, various downscaling methods have been proposed to improve the coarse resolution soil moisture products. The purpose of this paper is to review existing methods for downscaling satellite remotely sensed soil moisture. These methods are assessed and compared in terms of their advantages and limitations. This review also provides the accuracy level of these methods based on published validation studies. In the final part, problems and future trends associated with these methods are analyzed

Application of observation operators for field scale soil moisture averages and variances in agricultural landscapes

Scale difference between in situ and remotely sensed soil moisture observations and model grid size has been an issue for validation of remote sensing data, soil moisture data assimilation and calibration of hydrologic models. This study aims to link two different scales of soil moisture estimates by upscaling single point measurements to field averages for representing field-scale agricultural areas (∼2 ha) located within the Upper Cedar Creek Watershed in northeastern Indiana. Several statistical methods, mainly focusing on cumulative distribution function (CDF) matching, are tested to upscale point measurements to spatially representative soil moisture. These transforming equations are termed observation operators. The CDF matching is found to be the most successful upscaling method followed by the mean relative difference method using temporally stable point measurements. This study also tests the temporal and spatial (horizontal and vertical) transferability of the different observation operators. Results indicate that the two observation operators from the CDF matching approach and the mean relative difference method using a temporally stable location are transferable in space, but not in time. Rainfall characteristic is most likely the dominant factor affecting the success of observation operator transferability. In addition, the CDF matching approach is found to be an effective method to deduce spatial variability (standard deviation) of soil moisture from single point measurements

Ground, Proximal, and Satellite Remote Sensing of Soil Moisture

Soil moisture (SM) is a key hydrologic state variable that is of significant importance for numerous Earth and environmental science applications that directly impact the global environment and human society. Potential applications include, but are not limited to, forecasting of weather and climate variability; prediction and monitoring of drought conditions; management and allocation of water resources; agricultural plant production and alleviation of famine; prevention of natural disasters such as wild fires, landslides, floods, and dust storms; or monitoring of ecosystem response to climate change. Because of the importance and wide‐ranging applicability of highly variable spatial and temporal SM information that links the water, energy, and carbon cycles, significant efforts and resources have been devoted in recent years to advance SM measurement and monitoring capabilities from the point to the global scales. This review encompasses recent advances and the state‐of‐the‐art of ground, proximal, and novel SM remote sensing techniques at various spatial and temporal scales and identifies critical future research needs and directions to further advance and optimize technology, analysis and retrieval methods, and the application of SM information to improve the understanding of critical zone moisture dynamics. Despite the impressive progress over the last decade, there are still many opportunities and needs to, for example, improve SM retrieval from remotely sensed optical, thermal, and microwave data and opportunities for novel applications of SM information for water resources management, sustainable environmental development, and food security

High-Resolution SMAP-Derived Root-Zone Soil Moisture Using an Exponential Filter Model Calibrated per Land Cover Type

Root-zone soil moisture (RZSM) plays a key role for most water and energy budgets, as it is particularly relevant in controlling plant transpiration and hydraulic redistribution. RZSM data is needed for a variety of different applications, such as forecasting crop yields, improving flood predictions and monitoring agricultural drought, among others. Remote sensing provides surface soil moisture (SSM) retrievals, whose key advantage is the large spatial coverage on a systematic basis. This study tests a simple method to retrieve RZSM estimates from high-resolution SSM derived from SMAP (Soil Moisture Active Passive). A recursive exponential filter using a time constant τ is calibrated per land cover type, which uses as an intermediate step a long-term ISBA-DIF (Interaction Soil Biosphere Atmosphere—Diffusion scheme) dataset over an area located in Catalonia, NE of Spain. The τ values thus obtained are then used as an input to the same recursive exponential filter, to derive 1 km resolution RZSM estimates from 1 km SMAP SSM, which are obtained from the original data by downscaling to a 1 km resolution, through the DISPATCH (DISaggregation based on a Physical and Theoretical scale CHange) methodology. The results are then validated with scaled in situ observations at different depths, over two different areas, one representative of rainfed crops, and the other of irrigated crops. In general, the estimates agree well with the observations over the rainfed crops, especially at a 10 cm and 25 cm depth. Nash–Sutcliffe (NS) scores ranging between 0.33 and 0.58, and between 0.37 and 0.56 have been found, respectively. Correlation coefficients for these depths are high, between 0.76 and 0.91 (10 cm), and between 0.71 and 0.90 (25 cm). For the irrigated sites, results are poorer (partly due to the extremely high heterogeneity present), with NS scores ranging between −2.57 and 0.16, and correlations ranging between −0.56 and 0.48 at 25 cm. Given the strong correlations and NS scores found in the surface, the sensitivity of the filter to different τ values was investigated. For the rainfed site, it was found, as expected, with increasing τ, increasing NS and correlations with the deeper layers, suggesting a better coupling. Nevertheless, a strong correlation with the surface (5 cm) or shallower depths (10 cm) observed over certain sites indicates a certain lack of skill of the filter to represent processes which occur at lower levels in the SM column. All in all, a calibration accounting for the vegetation was shown to be an adequate methodology in applying the recursive exponential filter to derive the RZSM estimates over large areas. Nevertheless, the relative shallow surface at which the estimates correlate in some cases seem to indicate that an effect of evapotranspiration in the profile is not well captured by the filter.This research was funded by the Torrres Quevedo program of the Spanish Science Ministry, MICINN (grant number PTQ-16-08766)

Evaluation of SMAP Core Validation Site Representativeness Errors Using Dense Networks of In Situ Sensors and Random Forests

In order to validate its soil moisture products, the NASA Soil Moisture Active Passive (SMAP) mission utilises sites with permanent networks of in situ soil moisture sensors maintained by independent calibration and validation partners in a variety of ecosystems around the world. Measurements from each core validation site (CVS) are combined in a weighted average to produce an estimate of soil moisture at a 33-km scale that represents the SMAP’s radiometer-based retrievals. Since upscaled estimates produced in this manner are dependent on the weighting scheme applied, an independent method of quantifying their biases is needed.Here,we present one such method that uses soil moisture measurements taken from a dense, but temporary, network of soil moisture sensors deployed at each CVS to train a random forests regression expressing soil moisture in terms of a set of spatial variables. The regression then serves as an independent source of upscaled estimates against which permanent network upscaled estimates can be compared in order to calculate bias statistics.This method,which offers a systematic and unified approach to estimate bias across a variety of validation sites, was applied to estimate biases at four CVSs. The results showed that the magnitude of the uncertainty in the permanent network upscaling bias can sometimes exceed 80% of the upper limit on SMAP’s entire allowable unbiased root-mean-square error(ubRMSE).Such large CVS bias uncertainties could make it more difficult to assess biases in soil moisture estimates from SMAP

Soil Moisture Active Passive Mission L4_SM Data Product Assessment (Version 2 Validated Release)

During the post-launch SMAP calibration and validation (Cal/Val) phase there are two objectives for each science data product team: 1) calibrate, verify, and improve the performance of the science algorithm, and 2) validate the accuracy of the science data product as specified in the science requirements and according to the Cal/Val schedule. This report provides an assessment of the SMAP Level 4 Surface and Root Zone Soil Moisture Passive (L4_SM) product specifically for the product's public Version 2 validated release scheduled for 29 April 2016. The assessment of the Version 2 L4_SM data product includes comparisons of SMAP L4_SM soil moisture estimates with in situ soil moisture observations from core validation sites and sparse networks. The assessment further includes a global evaluation of the internal diagnostics from the ensemble-based data assimilation system that is used to generate the L4_SM product. This evaluation focuses on the statistics of the observation-minus-forecast (O-F) residuals and the analysis increments. Together, the core validation site comparisons and the statistics of the assimilation diagnostics are considered primary validation methodologies for the L4_SM product. Comparisons against in situ measurements from regional-scale sparse networks are considered a secondary validation methodology because such in situ measurements are subject to up-scaling errors from the point-scale to the grid cell scale of the data product. Based on the limited set of core validation sites, the wide geographic range of the sparse network sites, and the global assessment of the assimilation diagnostics, the assessment presented here meets the criteria established by the Committee on Earth Observing Satellites for Stage 2 validation and supports the validated release of the data. An analysis of the time average surface and root zone soil moisture shows that the global pattern of arid and humid regions are captured by the L4_SM estimates. Results from the core validation site comparisons indicate that "Version 2" of the L4_SM data product meets the self-imposed L4_SM accuracy requirement, which is formulated in terms of the ubRMSE: the RMSE (Root Mean Square Error) after removal of the long-term mean difference. The overall ubRMSE of the 3-hourly L4_SM surface soil moisture at the 9 km scale is 0.035 cubic meters per cubic meter requirement. The corresponding ubRMSE for L4_SM root zone soil moisture is 0.024 cubic meters per cubic meter requirement. Both of these metrics are comfortably below the 0.04 cubic meters per cubic meter requirement. The L4_SM estimates are an improvement over estimates from a model-only SMAP Nature Run version 4 (NRv4), which demonstrates the beneficial impact of the SMAP brightness temperature data. L4_SM surface soil moisture estimates are consistently more skillful than NRv4 estimates, although not by a statistically significant margin. The lack of statistical significance is not surprising given the limited data record available to date. Root zone soil moisture estimates from L4_SM and NRv4 have similar skill. Results from comparisons of the L4_SM product to in situ measurements from nearly 400 sparse network sites corroborate the core validation site results. The instantaneous soil moisture and soil temperature analysis increments are within a reasonable range and result in spatially smooth soil moisture analyses. The O-F residuals exhibit only small biases on the order of 1-3 degrees Kelvin between the (re-scaled) SMAP brightness temperature observations and the L4_SM model forecast, which indicates that the assimilation system is largely unbiased. The spatially averaged time series standard deviation of the O-F residuals is 5.9 degrees Kelvin, which reduces to 4.0 degrees Kelvin for the observation-minus-analysis (O-A) residuals, reflecting the impact of the SMAP observations on the L4_SM system. Averaged globally, the time series standard deviation of the normalized O-F residuals is close to unity, which would suggest that the magnitude of the modeled errors approximately reflects that of the actual errors. The assessment report also notes several limitations of the "Version 2" L4_SM data product and science algorithm calibration that will be addressed in future releases. Regionally, the time series standard deviation of the normalized O-F residuals deviates considerably from unity, which indicates that the L4_SM assimilation algorithm either over- or under-estimates the actual errors that are present in the system. Planned improvements include revised land model parameters, revised error parameters for the land model and the assimilated SMAP observations, and revised surface meteorological forcing data for the operational period and underlying climatological data. Moreover, a refined analysis of the impact of SMAP observations will be facilitated by the construction of additional variants of the model-only reference data. Nevertheless, the Version 2 validated release of the L4_SM product is sufficiently mature and of adequate quality for distribution to and use by the larger science and application communities

Upscaling soil moisture measurements from in situ sensors

Accurate, field-scale soil moisture information is needed to match the spatial scale of land and water management decisions related to agricultural production and environmental protection. Soil moisture measurements at the field-scale are limited because the resolution of most satellite-based soil moisture products is too coarse, while most in situ monitoring networks can provide only point-scale, not field-scale, data. This research attempts to develop a broadly applicable upscaling approach for observations from in situ soil moisture sensors using data from the Marena, Oklahoma, In Situ Sensor Testbed (MOISST) and a cosmic-ray neutron rover. The landscape at the MOISST site is predominantly grassland with some deciduous trees and eastern redcedar intermixed. Cosmic-ray neutron rover survey data were used to measure average soil moisture for the ~64 ha site on 12 dates in 2019-2020. The relationships between the point-scale in situ data and the field-scale rover data were examined using data from six in situ stations. Statistical modeling was used to identify the soil, terrain, and vegetation properties that influence these relationships. Site-specific linear upscaling models estimated the field average soil moisture with root mean squared error (RMSE) values ranging from 0.014 - 0.022 cm3 cm-3, but these models are not transferable to other sites. A general upscaling model using soil texture data was developed and achieved RMSE values ranging from 0.017 - 0.038 cm3 cm-3 for four calibration sites and values ranging from 0.015 - 0.021 cm3 cm-3 for two validation sites. The general upscaling model demonstrated accuracy better than the commonly used threshold of 0.04 cm3 cm-3 and should be further tested to evaluate its suitability as a broadly applicable upscaling approach for point-scale in situ monitoring stations