Improving satellite measurements of clouds and precipitation using machine learning

Observing and measuring clouds and precipitation is essential for climate science, meteorology, and an increasing range of societal and economic activities. This importance is due to the role of clouds and precipitation in the hydrological cycle and the weather and climate of the Earth. Furthermore, patterns of cloudiness and precipitation interact across continental scales and are highly variable in both space and time. Therefore their study and monitoring require observations with global coverage and high temporal resolution, which currently can only be provided by satellite observations.Inferring properties of clouds or precipitation from satellite observations is a non-trivial task. Due to the limited information content of the observations and the complex physics of the atmosphere, such retrievals are endowed with significant uncertainties. Traditional methods to perform these retrievals trade-off processing speed against accuracy and the ability to characterize the uncertainties in their predictions.This thesis develops and evaluates two neural-network-based methods for performing retrievals of hydrometeors, i.e., clouds and precipitation, that are capable of providing accurate predictions of the retrieval uncertainty. The practicality and benefits of the proposed methods are demonstrated using three real-world retrieval applications of cloud properties and precipitation. The demonstrated benefits of these methods over traditional retrieval methods led to the adoption of one of the algorithms for operational use at the European Organisation for the Exploitation of Meteorological Satellites. The two other algorithms are planned to be integrated into the operational processing at the Brazilian National Institute for Space Research, as well as the processing of observations from the Global Precipitation Measurement, a joint satellite mission by NASA and the Japanese Aerospace Exploration Agency.The principal advantage of the proposed methods is their simplicity and computational efficiency. A minor modification of the architecture and training of conventional neural networks is sufficient to capture the dominant source of uncertainty for remote sensing retrievals. As shown in this thesis, deep neural networks can significantly improve the accuracy of satellite retrievals of hydrometeors. With the proposed methods, the benefits of modern neural network architectures can be combined with reliable uncertainty estimates, which are required to improve the characterization of the observed hydrometeors

Remote sensing of clouds and precipitation using active and passive microwave observations

Global observations of clouds and precipitation are of great importance for weather prediction and the monitoring of the climate. Nonetheless, the currently available record of global observations does not constrain the properties of clouds very well owing to the inherent limitations of the observation systems used to produce them. The upcoming Ice Cloud Imager (ICI) microwave radiometer, which will be launched on the next generation of European weather satellites, will improve this situation by providing observations of clouds at sub-millimeter wavelengths. ICI will be the first sensor of its kind to deliver these observations, which will significantly improve the sensitivity to small ice particles and low mass concentrations compared to currently available microwave observations.This thesis presents research aimed at developing knowledge and methodology required for the modeling and interpretation of the observations that will be provided by ICI. Two studies are presented which propose a method for measuring distributions of ice hydrometeors from ICI-type sub-millimeter observations combined with radar observations.The first study uses simulated observations to assess the potential benefits of combin- ing sub-millimeter radiometer observations with radar observations for the retrieval of ice hydrometeors. It is found that the combined observations improve the sensitivity to microphysical properties of clouds, which can help to reduce the error in the retrieved mass concentrations for specific hydrometeor types. Furthermore, improved sensitivity to supercooled liquid cloud is found as an additional synergy between the active and passive observations.The second study aims to validate the results from the first by applying the synergistic retrieval algorithm to observations from a flight campaign. The retrieval algorithm is found to show overall good agreement with in-situ measured ice concentrations taking into account the sensitivity limits of the sensors. In addition to that, indications of a signal from mixed-phase particles are found in observations of convective updrafts. In the two presented studies, a synergistic retrieval algorithm for ice hydrometeors from radar and passive sub-millimeters has been developed, characterized and vali- dated. The method can be applied in a future satellite mission to reduce uncertainties in global observations of clouds or used to study cloud microphysical properties in field campaigns. In addition to that, the presented application to field campaign data provides one of the rare validation cases for the radiative transfer modeling involving clouds at sub-millimeter wavelengths

Study to support the definition of Arctic Weather Satellite (AWS) high frequency channels

This study compares the options of having a channel at 229 GHz or having some around\ua0325 GHz from a single perspective, cloud filtering/correction of 183 GHz data.Final report of EUMETSAT study under contract :\ua0 EUM/CO/20/4600002417/CJ

An experimental 2D-Var retrieval using AMSR2

A two-dimensional variational retrieval (2D-Var) is presented for a passive microwave imager. The overlapping antenna patterns of all frequencies from the Advanced Microwave Scanning Radiometer 2 (AMSR2) are explicitly simulated to attempt retrieval of near-surface wind speed and surface skin temperature at finer spatial scales than individual antenna beams. This is achieved, with the effective spatial resolution of retrieved parameters judged by analysis of 2D-Var averaging kernels. Sea surface temperature retrievals achieve about 30 km resolution, with wind speed retrievals at about 10 km resolution. It is argued that multi-dimensional optimal estimation permits greater use of total information content from microwave sensors than other methods, with no compromises on target resolution needed; instead, various targets are retrieved at the highest possible spatial resolution, driven by the channels\u27 sensitivities. All AMSR2 channels can be simulated within near their published noise characteristics for observed clear-sky scenes, though calibration and emissivity model errors are key challenges. This experimental retrieval shows the feasibility of 2D-Var for cloud-free retrievals and opens the possibility of stand-alone 3D-Var retrievals of water vapour and hydrometeor fields from microwave imagers in the future. The results have implications for future satellite missions and sensor design, as spatial oversampling can somewhat mitigate the need for larger antennas in the push for higher spatial resolution

Using passive and active observations at microwave and sub-millimetre wavelengths to constrain ice particle models

Satellite microwave remote sensing is an important tool for determining the distribution of atmospheric ice globally. The upcoming Ice Cloud Imager (ICI) will provide unprecedented measurements at sub-millimetre frequencies, employing channels up to 664 GHz. However, the utilization of such measurements requires detailed data on how individual ice particles scatter and absorb radiation, i.e. single scattering data. Several single scattering databases are currently available, with the one by Eriksson et al. (2018) specifically tailored to ICI. This study attempts to validate and constrain the large set of particle models available in this database to a smaller and more manageable set. A combined active and passive model framework is developed and employed, which converts CloudSat observations to simulated brightness temperatures (TBs) measured by the Global Precipitation Measurement (GPM) Microwave Imager (GMI) and ICI. Simulations covering about 1 month in the tropical Pacific Ocean are performed, assuming different microphysical settings realized as combinations of the particle model and particle size distribution (PSD). Firstly, it is found that when the CloudSat inversions and the passive forward model are considered separately, the assumed particle model and PSD have a considerable impact on both radar-retrieved ice water content (IWC) and simulated TBs. Conversely, when the combined active and passive framework is employed instead, the uncertainty due to the assumed particle model is significantly reduced. Furthermore, simulated TBs for almost all the tested microphysical combinations, from a statistical point of view, agree well with GMI measurements (166, 186.31, and 190.31 GHz), indicating the robustness of the simulations. However, it is difficult to identify a particle model that outperforms any other. One aggregate particle model, composed of columns, yields marginally better agreement with GMI compared to the other particles, mainly for the most severe cases of deep convection. Of the tested PSDs, the one by McFarquhar and Heymsfield (1997) is found to give the best overall agreement with GMI and also yields radar dBZ–IWC relationships closely matching measurements by Protat et al. (2016). Only one particle, modelled as an air–ice mixture spheroid, performs poorly overall. On the other hand, simulations at the higher ICI frequencies (328.65, 334.65, and 668.2 GHz) show significantly higher sensitivity to the assumed particle model. This study thus points to the potential use of combined ICI and 94 GHz radar measurements to constrain ice hydrometeor properties in radiative transfer (RT) using the method demonstrated in this paper

Ice water path retrievals from Meteosat-9 using quantile regression neural networks

The relationship between geostationary radiances and ice water path (IWP) is complex, and traditional retrieval approaches are not optimal. This work applies machine learning to improve the IWP retrieval from Meteosat-9 observations, with a focus on low latitudes, training the models against retrievals based on CloudSat. Advantages of machine learning include avoiding explicit physical assumptions on the data, an efficient use of information from all channels, and easily leveraging spatial information. Thermal infrared (IR) retrievals are used as input to achieve a performance independent of the solar angle. They are compared with retrievals including solar reflectances as well as a subset of IR channels for compatibility with historical sensors. The retrievals are accomplished with quantile regression neural networks. This network type provides case-specific uncertainty estimates, compatible with non-Gaussian errors, and is flexible enough to be applied to different network architectures. Spatial information is incorporated into the network through a convolutional neural network (CNN) architecture. This choice outperforms architectures that only work pixelwise. In fact, the CNN shows a good retrieval performance by using only IR channels. This makes it possible to compute diurnal cycles, a problem that CloudSat cannot resolve due to its limited temporal and spatial sampling. These retrievals compare favourably with IWP retrievals in CLAAS, a dataset based on a traditional approach. These results highlight the possibilities to overcome limitations from physics-based approaches using machine learning while providing efficient, probabilistic IWP retrieval methods. Moreover, they suggest this first work can be extended to higher latitudes as well as that geostationary data can be considered as a complement to the upcoming Ice Cloud Imager mission, for example, to bridge the gap in temporal sampling with respect to space-based radars

Can machine learning correct microwave humidity radiances for the influence of clouds?

A methodology based on quantile regression neural networks (QRNNs) is presented that identifies and corrects the cloud impact on microwave humidity sounder radiances at 183 GHz. This approach estimates the posterior distributions of noise-free clear-sky (NFCS) radiances, providing nearly bias-free estimates of clear-sky radiances with a full posterior error distribution. It is first demonstrated by application to a present sensor, the MicroWave Humidity Sounder 2 (MWHS-2); then the applicability to sub-millimetre (sub-mm) sensors is also analysed. The QRNN results improve upon what operational cloud filtering techniques like a scattering index can achieve but are ultimately imperfect due to limited information content on cirrus impact from traditional microwave channels - the negative departures associated with high cloud impact are successfully corrected, but thin cirrus clouds cannot be fully corrected. In contrast, when sub-mm observations are used, QRNN successfully corrects most cases with cloud impact, with only 2 %-6 % of the cases left partially corrected. The methodology works well even if only one sub-mm channel (325 GHz) is available. When using sub-mm observations, cloud correction usually results in error distributions with a standard deviation less than typical channel noise values. Furthermore, QRNN outputs predicted quantiles for case-specific uncertainty estimates, successfully representing the uncertainty of cloud correction for each observation individually. In comparison to deterministic correction or filtering approaches, the corrected radiances and attendant uncertainty estimates have great potential to be used efficiently in assimilation systems due to being largely unbiased and adding little further uncertainty to the measurements

Fast Radiative Transfer Approximating Ice Hydrometeor Orientation and its Implication on IWP Retrievals

The accurate simulation of microwave observations of clouds and precipitation are com-putationally challenging. A common simplification is the assumption of totally random orientation (TRO); however, studies have revealed that TRO occurs relatively infrequently in reality. A more appropriate assumption is that of azimuthally random orientation (ARO), but so far it has been a com-putationally expensive task. Recently a fast approximate approach was introduced that incorporates hydrometeor orientation into the assimilation of data from microwave conically scanning instruments. The approach scales the extinction in vertical (V) and horizontal (H) polarised channels to approximate ARO. In this study, the application of the approach was extended to a more basic radiative transfer perspective using the Atmospheric Radiative Transfer Simulator and the high-frequency channels of the Global Precipitation Measurement Microwave Imager (GMI). The comparison of forward simulations and GMI observations showed that with a random selection of scaling factors from a uniform distribution between 1 and 1.4–1.5, it is possible to mimic the full distribution of observed polarisation differences at 166 GHz over land and water. The applicability of this model at 660 GHz was also successfully demonstrated by means of existing airborne data. As a complement, a statistical model for polarised snow emissivity between 160 and 190 GHz was also developed. Combining the two models made it possible to reproduce the polarisation signals that were observed over all surface types, including snow and sea ice. Further, we also investigated the impact of orientation on the ice water path (IWP) retrievals. It has been shown that ignoring hydrometeor orientation has a significant negative impact (∼20% in the tropics) on retrieval accuracy. The retrieval with GMI observations produced highly realistic IWP distributions. A significant highlight was the retrieval over snow covered regions, which have been neglected in previous retrieval studies. These results provide increased confidence in the performance of passive microwave observation simulations and mark an essential step towards developing the retrievals of ice hydrometeor properties based on data from GMI, the Ice Cloud Imager (ICI) and other conically scanning instruments

GPROF-NN: a neural-network-based implementation of the Goddard Profiling Algorithm

The Global Precipitation Measurement (GPM) mission measures global precipitation at a temporal resolution of a few hours to enable close monitoring of the global hydrological cycle. GPM achieves this by combining observations from a spaceborne precipitation radar, a constellation of passive microwave (PMW) sensors, and geostationary satellites. The Goddard Profiling Algorithm (GPROF) is used operationally to retrieve precipitation from all PMW sensors of the GPM constellation. Since the resulting precipitation rates serve as input for many of the level 3 retrieval products, GPROF constitutes an essential component of the GPM processing pipeline. This study investigates ways to improve GPROF using modern machine learning methods. We present two neuralnetwork-based, probabilistic implementations of GPROF: GPROF-NN 1D, which (just like the current GPROF implementation) processes pixels individually, and GPROF-NN 3D, which employs a convolutional neural network to incorporate structural information into the retrieval. The accuracy of the retrievals is evaluated using a test dataset consistent with the data used in the development of the GPROF and GPROF-NN retrievals. This allows for assessing the accuracy of the retrieval method isolated from the representativeness of the training data, which remains a major source of uncertainty in the development of precipitation retrievals. Despite using the same input information as GPROF, the GPROF-NN 1D retrieval improves the accuracy of the retrieved surface precipitation for the GPM Microwave Imager (GMI) from 0.079 to 0.059mmh 1 in terms of mean abso- lute error (MAE), from 76.1% to 69.5% in terms of symmetric mean absolute percentage error (SMAPE) and from 0 :797 to 0 :847 in terms of correlation. The improvements for the Microwave Humidity Sounder (MHS) are from 0.085 to 0.061mmh 1 in terms of MAE, from 81% to 70.1% for SMAPE, and from 0 :724 to 0 :804 in terms of correlation. Comparable improvements are found for the retrieved hydrometeor profiles and their column integrals, as well as the detection of precipitation. Moreover, the ability of the retrievals to resolve small-scale variability is improved by more than 40% for GMI and 29% for MHS. The GPROFNN 3D retrieval further improves the MAE to 0.043mmh 1; the SMAPE to 48.67 %; and the correlation to 0:897 for GMI and 0.043mmh 1, 63.42 %, and 0:83 for MHS. Application of the retrievals to GMI observations of Hurricane Harvey shows moderate improvements when compared to co-located GPM-combined and ground-based radar measurements indicating that the improvements at least partially carry over to assessment against independent measurements. Similar retrievals for MHS do not show equally clear improvements, leaving the validation against independent measurements for future investigation. Both GPROF-NN algorithms make use of the same input and output data as the original GPROF algorithm and thus may replace the current implementation in a future update of the GPM processing pipeline. Despite their superior accuracy, the single-core runtime required for the operational processing of an orbit of observations is lower than that of GPROF. The GPROF-NN algorithms promise to be a simple and cost-efficient way to increase the accuracy of the PMW precipitation retrievals of the GPM constellation and thus improve the monitoring of the global hydrological cycle

An improved near-real-Time precipitation retrieval for Brazil

Observations from geostationary satellites can provide spatially continuous coverage at continental scales with high spatial and temporal resolution. Because of this, they are commonly used to complement ground-based precipitation measurements, whose coverage is often more limited. We present Hydronn, a neural-network-based, near-real-Time precipitation retrieval for Brazil based on visible and infrared (Vis-IR) observations from the Advanced Baseline Imager (ABI) on the Geostationary Operational Environmental Satellite 16 (GOES-16). The retrieval, which employs a convolutional neural network to perform Bayesian precipitation retrievals, was developed with the aims of (1) leveraging the full potential of latest-generation geostationary observations and (2) providing probabilistic precipitation estimates with well-calibrated uncertainties. The retrieval is trained using more than 3 years of collocations with combined radar and radiometer retrievals from the Global Precipitation Measurement (GPM) core observatory over South America. The accuracy of instantaneous precipitation estimates is assessed using a separate year of GPM combined retrievals and compared to retrievals from passive microwave (PMW) sensors and HYDRO, the Vis-IR retrieval that is currently in operational use at the Brazilian Institute for Space Research. Using all available channels of the ABI, Hydronn achieves accuracy close to that of state-of-The-Art PMW precipitation retrievals in both precipitation estimation and detection despite the lower information content of the Vis-IR observations. Hourly, daily, and monthly precipitation accumulations are evaluated against gauge measurements for June and December 2020 and compared to HYDRO, the Precipitation Estimation from Remotely Sensed Information using Artificial Neural Networks (PERSIANN) Cloud Classification System (CCS), and the Integrated Multi-satellitE Retrievals for GPM (IMERG). Compared to HYDRO, Hydronn reduces the mean absolute error for hourly accumulations by 21% (22%) compared to HYDRO by 44% (41%) for the mean squared error (MSE) and increases the correlation by 138% (312%) for June (December) 2020. Compared to IMERG, the improvements correspond to 16% (14%), 12% (12%), and 20% (56%), respectively. Furthermore, we show that the probabilistic retrieval is well calibrated against gauge measurements when differences in the distributions of the training data and the gauge measurements are accounted for. Hydronn has the potential to significantly improve near-real-Time precipitation retrievals over Brazil. Furthermore, our results show that precipitation retrievals based on convolutional neural networks (CNNs) that leverage the full range of available observations from latest-generation geostationary satellites can provide instantaneous precipitation estimates with accuracy close to that of state-of-The-Art PMW retrievals. The high temporal resolution of the geostationary observation allows Hydronn to provide more accurate precipitation accumulations than any of the tested conventional precipitation retrievals. Hydronn thus clearly shows the potential of deep-learning-based precipitation retrievals to improve precipitation estimates from currently available satellite imagery