Deep learning in remote sensing: a review

Standing at the paradigm shift towards data-intensive science, machine learning techniques are becoming increasingly important. In particular, as a major breakthrough in the field, deep learning has proven as an extremely powerful tool in many fields. Shall we embrace deep learning as the key to all? Or, should we resist a 'black-box' solution? There are controversial opinions in the remote sensing community. In this article, we analyze the challenges of using deep learning for remote sensing data analysis, review the recent advances, and provide resources to make deep learning in remote sensing ridiculously simple to start with. More importantly, we advocate remote sensing scientists to bring their expertise into deep learning, and use it as an implicit general model to tackle unprecedented large-scale influential challenges, such as climate change and urbanization.Comment: Accepted for publication IEEE Geoscience and Remote Sensing Magazin

A novel fusion framework embedded with zero-shot super-resolution and multivariate autoregression for precipitable water vapor across the continental Europe

Precipitable water vapor (PWV), as the most abundant greenhouse gas, significantly impacts the evapotranspiration process and thus the global climate. However, the applicability of mainstream satellite PWV products is limited by the tradeoff between spatial and temporal resolutions, as well as some external factors such as cloud contamination. In this study, we proposed a novel PWV spatio-temporal fusion framework based on the zero-shot super-resolution and the multivariate autoregression models (ZSSR-ARF) to improve the accuracy and continuity of PWV. The framework is implemented in a way that the satellite-derived observations (MOD05) are fused with the reanalysis data (ERA5) to generate accurate and seamless PWV of high spatio-temporal resolution (0.01°, daily) across the European continent from 2001 to 2021. Firstly, the ZSSR approach is used to enhance the spatial resolution of ERA5 PWV based on the internal recurrence of image information. Secondly, the optimal ERA5-MOD05 image pairs are selected based on the image similarity as inputs to improve the fusion accuracy. Thirdly, the framework develops a multivariate autoregressive fusion approach to allocate weights adaptively for the high-resolution image prediction, which primely addresses the non-stationarity and autocorrelation of PWV. The results reveal that the accuracies of fused PWV are consistent with those of the GPS retrievals (r = 0.82–0.95 and RMSE = 2.21–4.01 mm), showing an enhancement in the accuracy and continuity compared to the original MODIS PWV. The ZSSR-ARF fusion framework outperforms the other methods with R$^2$ improved by over 24% and RMSE reduced by over 0.61 mm. Furthermore, the fused PWV exhibits similar temporal consistency (mean difference of 0.40 mm and DSTD of 3.22 mm) to the reliable ERA5 products, and substantial increasing trends (mean of 0.057 mm/year and over 0.1 mm/year near the southern and western coasts) are observed over the European continent. As the accuracy and continuity of PWV are improved, the outcome of this paper has potential for climatic analyses during the land-atmosphere cycle process

Target-adaptive CNN-based pansharpening

We recently proposed a convolutional neural network (CNN) for remote sensing image pansharpening obtaining a significant performance gain over the state of the art. In this paper, we explore a number of architectural and training variations to this baseline, achieving further performance gains with a lightweight network which trains very fast. Leveraging on this latter property, we propose a target-adaptive usage modality which ensures a very good performance also in the presence of a mismatch w.r.t. the training set, and even across different sensors. The proposed method, published online as an off-the-shelf software tool, allows users to perform fast and high-quality CNN-based pansharpening of their own target images on general-purpose hardware

A data-fusion approach to motion-stereo

This paper introduces a novel method for performing motion--stereo, based on dynamic integration of depth (or its proxy) measures obtained by pairwise stereo matching of video frames. The focus is on the data fusion issue raised by the motion--stereo approach, which is solved within a Kalman filtering framework. Integration occurs along the temporal and spatial dimension, so that the final measure for a pixel results from the combination of measures of the same pixel in time and whose of its neighbors. The method has been validated on both synthetic and natural images, using the simplest stereo matching strategy and a range of different confidence measures, and has been compared to baseline and optimal strategies

A Simple Fusion Mehtod for Image Time Series Based on the Estimation of Image Temporal Validity

High-spatial-resolution satellites usually have the constraint of a low temporal frequency, which leads to long periods without information in cloudy areas. Furthermore, low-spatial-resolution satellites have higher revisit cycles. Combining information from high- and low- spatial-resolution satellites is thought a key factor for studies that require dense time series of high-resolution images, e.g., crop monitoring. There are several fusion methods in the bibliography, but they are time-consuming and complicated to implement. Moreover, the local evaluation of the fused images is rarely analyzed. In this paper, we present a simple and fast fusion method based on a weighted average of two input images (H and L ), which are weighted by their temporal validity to the image to be fused. The method was applied to two years (2009-2010) of Landsat and MODIS (MODerate Imaging Spectroradiometer) images that were acquired over a cropped area in Brazil. The fusion method was evaluated at global and local scales. The results show that the fused images reproduced reliable crop temporal profiles and correctly delineated th e boundaries between two neighboring fields. The great est advantages of the proposed method are the execution time and ease of use, which allow us to obtain a fused image in less than five minutes