Assessment of a Variable Projection Algorithm for Trace Gas Retrieval in the Short-Wave Infrared

An important part of atmospheric remote sensing is the monitoring of its composition, which can be retrieved from radiance measurements, e.g. in the short-wave infrared (SWIR). For deriving trace gas concentrations in the SWIR spectral region a radiative transfer model is fitted to observations by least squares optimization. The aim of this thesis is to present the well-established variable projection method for solving separable nonlinear least squares problems and to examine and configure it for trace gas retrieval. For this, a Python implementation of the algorithm, called varpro.py, will be outlined and later utilized in retrievals with real satellite observations. These are meant to assess the efficiency, accuracy and robustness of three iterative algorithms for nonlinear least squares problems which have been built into varpro.py. Furthermore, a new feature - applying bounds to the non-linear fit parameters - will be included in the implementation and evaluated for its quality and usefulness. As a result of these tests, a new 'default' configuration will be suggested based on the algorithm with the best performance for trace gas retrieval. Also, ideas for analysing and testing strategies which could lead to even more insights will be proposed. Finally, possible future applications for trace gas retrieval will be motivated and suggestions for further research and modifications of varpro.py will be made

A Generalized Variable Projection Algorithm for Least Squares Problems in Atmospheric Remote Sensing

The paper presents a solution for efficiently and accurately solving separable least squares problems with multiple datasets. These problems involve determining linear parameters that are specific to each dataset while ensuring that the nonlinear parameters remain consistent across all datasets. A well-established approach for solving such problems is the variable projection algorithm introduced by Golub and LeVeque, which effectively reduces a separable problem to its nonlinear component. However, this algorithm assumes that the datasets have equal sizes and identical auxiliary model parameters. This article is motivated by a real-world remote sensing application where these assumptions do not apply. Consequently, we propose a generalized algorithm that extends the original theory to overcome these limitations. The new algorithm has been implemented and tested using both synthetic and real satellite data for atmospheric carbon dioxide retrievals. It has also been compared to conventional state-of-the-art solvers, and its advantages are thoroughly discussed. The experimental results demonstrate that the proposed algorithm significantly outperforms all other methods in terms of computation time, while maintaining comparable accuracy and stability. Hence, this novel method can have a positive impact on future applications in remote sensing and could be valuable for other scientific fitting problems with similar properties