A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community

In recent years, deep learning (DL), a re-branding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, natural language processing, etc. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV; e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should be aware of, if not at the leading edge of, of advancements like DL. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as it relates to (i) inadequate data sets, (ii) human-understandable solutions for modelling physical phenomena, (iii) Big Data, (iv) non-traditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.Comment: 64 pages, 411 references. To appear in Journal of Applied Remote Sensin

Recent Advances in Image Restoration with Applications to Real World Problems

In the past few decades, imaging hardware has improved tremendously in terms of resolution, making widespread usage of images in many diverse applications on Earth and planetary missions. However, practical issues associated with image acquisition are still affecting image quality. Some of these issues such as blurring, measurement noise, mosaicing artifacts, low spatial or spectral resolution, etc. can seriously affect the accuracy of the aforementioned applications. This book intends to provide the reader with a glimpse of the latest developments and recent advances in image restoration, which includes image super-resolution, image fusion to enhance spatial, spectral resolution, and temporal resolutions, and the generation of synthetic images using deep learning techniques. Some practical applications are also included

A Multi-temporal fusion-based approach for land cover mapping in support of nuclear incident response

An increasingly important application of remote sensing is to provide decision support during emergency response and disaster management efforts. Land cover maps constitute one such useful application product during disaster events; if generated rapidly after any disaster, such map products can contribute to the efficacy of the response effort. In light of recent nuclear incidents, e.g., after the earthquake/tsunami in Japan (2011), our research focuses on constructing rapid and accurate land cover maps of the impacted area in case of an accidental nuclear release. The methodology involves integration of results from two different approaches, namely coarse spatial resolution multi-temporal and fine spatial resolution imagery, to increase classification accuracy. Although advanced methods have been developed for classification using high spatial or temporal resolution imagery, only a limited amount of work has been done on fusion of these two remote sensing approaches. The presented methodology thus involves integration of classification results from two different remote sensing modalities in order to improve classification accuracy. The data used included RapidEye and MODIS scenes over the Nine Mile Point Nuclear Power Station in Oswego (New York, USA). The first step in the process was the construction of land cover maps from freely available, high temporal resolution, low spatial resolution MODIS imagery using a time-series approach. We used the variability in the temporal signatures among different land cover classes for classification. The time series-specific features were defined by various physical properties of a pixel, such as variation in vegetation cover and water content over time. The pixels were classified into four land cover classes - forest, urban, water, and vegetation - using Euclidean and Mahalanobis distance metrics. On the other hand, a high spatial resolution commercial satellite, such as RapidEye, can be tasked to capture images over the affected area in the case of a nuclear event. This imagery served as a second source of data to augment results from the time series approach. The classifications from the two approaches were integrated using an a posteriori probability-based fusion approach. This was done by establishing a relationship between the classes, obtained after classification of the two data sources. Despite the coarse spatial resolution of MODIS pixels, acceptable accuracies were obtained using time series features. The overall accuracies using the fusion-based approach were in the neighborhood of 80%, when compared with GIS data sets from New York State. This fusion thus contributed to classification accuracy refinement, with a few additional advantages, such as correction for cloud cover and providing for an approach that is robust against point-in-time seasonal anomalies, due to the inclusion of multi-temporal data

A Simple Fusion Mehtod for Image Time Series Based on the Estimation of Image Temporal Validity

High-spatial-resolution satellites usually have the constraint of a low temporal frequency, which leads to long periods without information in cloudy areas. Furthermore, low-spatial-resolution satellites have higher revisit cycles. Combining information from high- and low- spatial-resolution satellites is thought a key factor for studies that require dense time series of high-resolution images, e.g., crop monitoring. There are several fusion methods in the bibliography, but they are time-consuming and complicated to implement. Moreover, the local evaluation of the fused images is rarely analyzed. In this paper, we present a simple and fast fusion method based on a weighted average of two input images (H and L ), which are weighted by their temporal validity to the image to be fused. The method was applied to two years (2009-2010) of Landsat and MODIS (MODerate Imaging Spectroradiometer) images that were acquired over a cropped area in Brazil. The fusion method was evaluated at global and local scales. The results show that the fused images reproduced reliable crop temporal profiles and correctly delineated th e boundaries between two neighboring fields. The great est advantages of the proposed method are the execution time and ease of use, which allow us to obtain a fused image in less than five minutes

A novel fusion framework embedded with zero-shot super-resolution and multivariate autoregression for precipitable water vapor across the continental Europe

Precipitable water vapor (PWV), as the most abundant greenhouse gas, significantly impacts the evapotranspiration process and thus the global climate. However, the applicability of mainstream satellite PWV products is limited by the tradeoff between spatial and temporal resolutions, as well as some external factors such as cloud contamination. In this study, we proposed a novel PWV spatio-temporal fusion framework based on the zero-shot super-resolution and the multivariate autoregression models (ZSSR-ARF) to improve the accuracy and continuity of PWV. The framework is implemented in a way that the satellite-derived observations (MOD05) are fused with the reanalysis data (ERA5) to generate accurate and seamless PWV of high spatio-temporal resolution (0.01°, daily) across the European continent from 2001 to 2021. Firstly, the ZSSR approach is used to enhance the spatial resolution of ERA5 PWV based on the internal recurrence of image information. Secondly, the optimal ERA5-MOD05 image pairs are selected based on the image similarity as inputs to improve the fusion accuracy. Thirdly, the framework develops a multivariate autoregressive fusion approach to allocate weights adaptively for the high-resolution image prediction, which primely addresses the non-stationarity and autocorrelation of PWV. The results reveal that the accuracies of fused PWV are consistent with those of the GPS retrievals (r = 0.82–0.95 and RMSE = 2.21–4.01 mm), showing an enhancement in the accuracy and continuity compared to the original MODIS PWV. The ZSSR-ARF fusion framework outperforms the other methods with R$^2$ improved by over 24% and RMSE reduced by over 0.61 mm. Furthermore, the fused PWV exhibits similar temporal consistency (mean difference of 0.40 mm and DSTD of 3.22 mm) to the reliable ERA5 products, and substantial increasing trends (mean of 0.057 mm/year and over 0.1 mm/year near the southern and western coasts) are observed over the European continent. As the accuracy and continuity of PWV are improved, the outcome of this paper has potential for climatic analyses during the land-atmosphere cycle process

Joint-SRVDNet: Joint Super Resolution and Vehicle Detection Network

In many domestic and military applications, aerial vehicle detection and super-resolutionalgorithms are frequently developed and applied independently. However, aerial vehicle detection on super-resolved images remains a challenging task due to the lack of discriminative information in the super-resolved images. To address this problem, we propose a Joint Super-Resolution and Vehicle DetectionNetwork (Joint-SRVDNet) that tries to generate discriminative, high-resolution images of vehicles fromlow-resolution aerial images. First, aerial images are up-scaled by a factor of 4x using a Multi-scaleGenerative Adversarial Network (MsGAN), which has multiple intermediate outputs with increasingresolutions. Second, a detector is trained on super-resolved images that are upscaled by factor 4x usingMsGAN architecture and finally, the detection loss is minimized jointly with the super-resolution loss toencourage the target detector to be sensitive to the subsequent super-resolution training. The network jointlylearns hierarchical and discriminative features of targets and produces optimal super-resolution results. Weperform both quantitative and qualitative evaluation of our proposed network on VEDAI, xView and DOTAdatasets. The experimental results show that our proposed framework achieves better visual quality than thestate-of-the-art methods for aerial super-resolution with 4x up-scaling factor and improves the accuracy ofaerial vehicle detection

Monitoring wetlands and water bodies in semi-arid Sub-Saharan regions

Surface water in wetlands is a critical resource in semi-arid West-African regions that are frequently exposed to droughts. Wetlands are of utmost importance for the population as well as the environment, and are subject to rapidly changing seasonal fluctuations. Dynamics of wetlands in the study area are still poorly understood, and the potential of remote sensing-derived information as a large-scale, multi-temporal, comparable and independent measurement source is not exploited. This work shows successful wetland monitoring with remote sensing in savannah and Sahel regions in Burkina Faso, focusing on the main study site Lac Bam (Lake Bam). Long-term optical time series from MODIS with medium spatial resolution (MR), and short-term synthetic aperture radar (SAR) time series from TerraSAR-X and RADARSAT-2 with high spatial resolution (HR) successfully demonstrate the classification and dynamic monitoring of relevant wetland features, e.g. open water, flooded vegetation and irrigated cultivation. Methodological highlights are time series analysis, e.g. spatio-temporal dynamics or multitemporal-classification, as well as polarimetric SAR (polSAR) processing, i.e. the Kennaugh elements, enabling physical interpretation of SAR scattering mechanisms for dual-polarized data. A multi-sensor and multi-frequency SAR data combination provides added value, and reveals that dual-co-pol SAR data is most recommended for monitoring wetlands of this type. The interpretation of environmental or man-made processes such as water areas spreading out further but retreating or evaporating faster, co-occurrence of droughts with surface water and vegetation anomalies, expansion of irrigated agriculture or new dam building, can be detected with MR optical and HR SAR time series. To capture long-term impacts of water extraction, sedimentation and climate change on wetlands, remote sensing solutions are available, and would have great potential to contribute to water management in Africa