Improving satellite measurements of clouds and precipitation using machine learning

Observing and measuring clouds and precipitation is essential for climate science, meteorology, and an increasing range of societal and economic activities. This importance is due to the role of clouds and precipitation in the hydrological cycle and the weather and climate of the Earth. Furthermore, patterns of cloudiness and precipitation interact across continental scales and are highly variable in both space and time. Therefore their study and monitoring require observations with global coverage and high temporal resolution, which currently can only be provided by satellite observations.Inferring properties of clouds or precipitation from satellite observations is a non-trivial task. Due to the limited information content of the observations and the complex physics of the atmosphere, such retrievals are endowed with significant uncertainties. Traditional methods to perform these retrievals trade-off processing speed against accuracy and the ability to characterize the uncertainties in their predictions.This thesis develops and evaluates two neural-network-based methods for performing retrievals of hydrometeors, i.e., clouds and precipitation, that are capable of providing accurate predictions of the retrieval uncertainty. The practicality and benefits of the proposed methods are demonstrated using three real-world retrieval applications of cloud properties and precipitation. The demonstrated benefits of these methods over traditional retrieval methods led to the adoption of one of the algorithms for operational use at the European Organisation for the Exploitation of Meteorological Satellites. The two other algorithms are planned to be integrated into the operational processing at the Brazilian National Institute for Space Research, as well as the processing of observations from the Global Precipitation Measurement, a joint satellite mission by NASA and the Japanese Aerospace Exploration Agency.The principal advantage of the proposed methods is their simplicity and computational efficiency. A minor modification of the architecture and training of conventional neural networks is sufficient to capture the dominant source of uncertainty for remote sensing retrievals. As shown in this thesis, deep neural networks can significantly improve the accuracy of satellite retrievals of hydrometeors. With the proposed methods, the benefits of modern neural network architectures can be combined with reliable uncertainty estimates, which are required to improve the characterization of the observed hydrometeors

Remote sensing of clouds and precipitation using active and passive microwave observations

Global observations of clouds and precipitation are of great importance for weather prediction and the monitoring of the climate. Nonetheless, the currently available record of global observations does not constrain the properties of clouds very well owing to the inherent limitations of the observation systems used to produce them. The upcoming Ice Cloud Imager (ICI) microwave radiometer, which will be launched on the next generation of European weather satellites, will improve this situation by providing observations of clouds at sub-millimeter wavelengths. ICI will be the first sensor of its kind to deliver these observations, which will significantly improve the sensitivity to small ice particles and low mass concentrations compared to currently available microwave observations.This thesis presents research aimed at developing knowledge and methodology required for the modeling and interpretation of the observations that will be provided by ICI. Two studies are presented which propose a method for measuring distributions of ice hydrometeors from ICI-type sub-millimeter observations combined with radar observations.The first study uses simulated observations to assess the potential benefits of combin- ing sub-millimeter radiometer observations with radar observations for the retrieval of ice hydrometeors. It is found that the combined observations improve the sensitivity to microphysical properties of clouds, which can help to reduce the error in the retrieved mass concentrations for specific hydrometeor types. Furthermore, improved sensitivity to supercooled liquid cloud is found as an additional synergy between the active and passive observations.The second study aims to validate the results from the first by applying the synergistic retrieval algorithm to observations from a flight campaign. The retrieval algorithm is found to show overall good agreement with in-situ measured ice concentrations taking into account the sensitivity limits of the sensors. In addition to that, indications of a signal from mixed-phase particles are found in observations of convective updrafts. In the two presented studies, a synergistic retrieval algorithm for ice hydrometeors from radar and passive sub-millimeters has been developed, characterized and vali- dated. The method can be applied in a future satellite mission to reduce uncertainties in global observations of clouds or used to study cloud microphysical properties in field campaigns. In addition to that, the presented application to field campaign data provides one of the rare validation cases for the radiative transfer modeling involving clouds at sub-millimeter wavelengths

Study to support the definition of Arctic Weather Satellite (AWS) high frequency channels

This study compares the options of having a channel at 229 GHz or having some around\ua0325 GHz from a single perspective, cloud filtering/correction of 183 GHz data.Final report of EUMETSAT study under contract :\ua0 EUM/CO/20/4600002417/CJ

An experimental 2D-Var retrieval using AMSR2

A two-dimensional variational retrieval (2D-Var) is presented for a passive microwave imager. The overlapping antenna patterns of all frequencies from the Advanced Microwave Scanning Radiometer 2 (AMSR2) are explicitly simulated to attempt retrieval of near-surface wind speed and surface skin temperature at finer spatial scales than individual antenna beams. This is achieved, with the effective spatial resolution of retrieved parameters judged by analysis of 2D-Var averaging kernels. Sea surface temperature retrievals achieve about 30 km resolution, with wind speed retrievals at about 10 km resolution. It is argued that multi-dimensional optimal estimation permits greater use of total information content from microwave sensors than other methods, with no compromises on target resolution needed; instead, various targets are retrieved at the highest possible spatial resolution, driven by the channels\u27 sensitivities. All AMSR2 channels can be simulated within near their published noise characteristics for observed clear-sky scenes, though calibration and emissivity model errors are key challenges. This experimental retrieval shows the feasibility of 2D-Var for cloud-free retrievals and opens the possibility of stand-alone 3D-Var retrievals of water vapour and hydrometeor fields from microwave imagers in the future. The results have implications for future satellite missions and sensor design, as spatial oversampling can somewhat mitigate the need for larger antennas in the push for higher spatial resolution

Using passive and active observations at microwave and sub-millimetre wavelengths to constrain ice particle models

Satellite microwave remote sensing is an important tool for determining the distribution of atmospheric ice globally. The upcoming Ice Cloud Imager (ICI) will provide unprecedented measurements at sub-millimetre frequencies, employing channels up to 664 GHz. However, the utilization of such measurements requires detailed data on how individual ice particles scatter and absorb radiation, i.e. single scattering data. Several single scattering databases are currently available, with the one by Eriksson et al. (2018) specifically tailored to ICI. This study attempts to validate and constrain the large set of particle models available in this database to a smaller and more manageable set. A combined active and passive model framework is developed and employed, which converts CloudSat observations to simulated brightness temperatures (TBs) measured by the Global Precipitation Measurement (GPM) Microwave Imager (GMI) and ICI. Simulations covering about 1 month in the tropical Pacific Ocean are performed, assuming different microphysical settings realized as combinations of the particle model and particle size distribution (PSD). Firstly, it is found that when the CloudSat inversions and the passive forward model are considered separately, the assumed particle model and PSD have a considerable impact on both radar-retrieved ice water content (IWC) and simulated TBs. Conversely, when the combined active and passive framework is employed instead, the uncertainty due to the assumed particle model is significantly reduced. Furthermore, simulated TBs for almost all the tested microphysical combinations, from a statistical point of view, agree well with GMI measurements (166, 186.31, and 190.31 GHz), indicating the robustness of the simulations. However, it is difficult to identify a particle model that outperforms any other. One aggregate particle model, composed of columns, yields marginally better agreement with GMI compared to the other particles, mainly for the most severe cases of deep convection. Of the tested PSDs, the one by McFarquhar and Heymsfield (1997) is found to give the best overall agreement with GMI and also yields radar dBZ–IWC relationships closely matching measurements by Protat et al. (2016). Only one particle, modelled as an air–ice mixture spheroid, performs poorly overall. On the other hand, simulations at the higher ICI frequencies (328.65, 334.65, and 668.2 GHz) show significantly higher sensitivity to the assumed particle model. This study thus points to the potential use of combined ICI and 94 GHz radar measurements to constrain ice hydrometeor properties in radiative transfer (RT) using the method demonstrated in this paper

Ice water path retrievals from Meteosat-9 using quantile regression neural networks

The relationship between geostationary radiances and ice water path (IWP) is complex, and traditional retrieval approaches are not optimal. This work applies machine learning to improve the IWP retrieval from Meteosat-9 observations, with a focus on low latitudes, training the models against retrievals based on CloudSat. Advantages of machine learning include avoiding explicit physical assumptions on the data, an efficient use of information from all channels, and easily leveraging spatial information. Thermal infrared (IR) retrievals are used as input to achieve a performance independent of the solar angle. They are compared with retrievals including solar reflectances as well as a subset of IR channels for compatibility with historical sensors. The retrievals are accomplished with quantile regression neural networks. This network type provides case-specific uncertainty estimates, compatible with non-Gaussian errors, and is flexible enough to be applied to different network architectures. Spatial information is incorporated into the network through a convolutional neural network (CNN) architecture. This choice outperforms architectures that only work pixelwise. In fact, the CNN shows a good retrieval performance by using only IR channels. This makes it possible to compute diurnal cycles, a problem that CloudSat cannot resolve due to its limited temporal and spatial sampling. These retrievals compare favourably with IWP retrievals in CLAAS, a dataset based on a traditional approach. These results highlight the possibilities to overcome limitations from physics-based approaches using machine learning while providing efficient, probabilistic IWP retrieval methods. Moreover, they suggest this first work can be extended to higher latitudes as well as that geostationary data can be considered as a complement to the upcoming Ice Cloud Imager mission, for example, to bridge the gap in temporal sampling with respect to space-based radars

KMWin – A Convenient Tool for Graphical Presentation of Results from Kaplan-Meier Survival Time Analysis

BACKGROUND: Analysis of clinical studies often necessitates multiple graphical representations of the results. Many professional software packages are available for this purpose. Most packages are either only commercially available or hard to use especially if one aims to generate or customize a huge number of similar graphical outputs. We developed a new, freely available software tool called KMWin (Kaplan-Meier for Windows) facilitating Kaplan-Meier survival time analysis. KMWin is based on the statistical software environment R and provides an easy to use graphical interface. Survival time data can be supplied as SPSS (sav), SAS export (xpt) or text file (dat), which is also a common export format of other applications such as Excel. Figures can directly be exported in any graphical file format supported by R. RESULTS: On the basis of a working example, we demonstrate how to use KMWin and present its main functions. We show how to control the interface, customize the graphical output, and analyse survival time data. A number of comparisons are performed between KMWin and SPSS regarding graphical output, statistical output, data management and development. Although the general functionality of SPSS is larger, KMWin comprises a number of features useful for survival time analysis in clinical trials and other applications. These are for example number of cases and number of cases under risk within the figure or provision of a queue system for repetitive analyses of updated data sets. Moreover, major adjustments of graphical settings can be performed easily on a single window. CONCLUSIONS: We conclude that our tool is well suited and convenient for repetitive analyses of survival time data. It can be used by non-statisticians and provides often used functions as well as functions which are not supplied by standard software packages. The software is routinely applied in several clinical study groups

Can machine learning correct microwave humidity radiances for the influence of clouds?

A methodology based on quantile regression neural networks (QRNNs) is presented that identifies and corrects the cloud impact on microwave humidity sounder radiances at 183 GHz. This approach estimates the posterior distributions of noise-free clear-sky (NFCS) radiances, providing nearly bias-free estimates of clear-sky radiances with a full posterior error distribution. It is first demonstrated by application to a present sensor, the MicroWave Humidity Sounder 2 (MWHS-2); then the applicability to sub-millimetre (sub-mm) sensors is also analysed. The QRNN results improve upon what operational cloud filtering techniques like a scattering index can achieve but are ultimately imperfect due to limited information content on cirrus impact from traditional microwave channels - the negative departures associated with high cloud impact are successfully corrected, but thin cirrus clouds cannot be fully corrected. In contrast, when sub-mm observations are used, QRNN successfully corrects most cases with cloud impact, with only 2 %-6 % of the cases left partially corrected. The methodology works well even if only one sub-mm channel (325 GHz) is available. When using sub-mm observations, cloud correction usually results in error distributions with a standard deviation less than typical channel noise values. Furthermore, QRNN outputs predicted quantiles for case-specific uncertainty estimates, successfully representing the uncertainty of cloud correction for each observation individually. In comparison to deterministic correction or filtering approaches, the corrected radiances and attendant uncertainty estimates have great potential to be used efficiently in assimilation systems due to being largely unbiased and adding little further uncertainty to the measurements