A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community

In recent years, deep learning (DL), a re-branding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, natural language processing, etc. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV; e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should be aware of, if not at the leading edge of, of advancements like DL. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as it relates to (i) inadequate data sets, (ii) human-understandable solutions for modelling physical phenomena, (iii) Big Data, (iv) non-traditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.Comment: 64 pages, 411 references. To appear in Journal of Applied Remote Sensin

DeepAqua: Self-Supervised Semantic Segmentation of Wetland Surface Water Extent with SAR Images using Knowledge Distillation

Deep learning and remote sensing techniques have significantly advanced water monitoring abilities; however, the need for annotated data remains a challenge. This is particularly problematic in wetland detection, where water extent varies over time and space, demanding multiple annotations for the same area. In this paper, we present DeepAqua, a self-supervised deep learning model that leverages knowledge distillation (a.k.a. teacher-student model) to eliminate the need for manual annotations during the training phase. We utilize the Normalized Difference Water Index (NDWI) as a teacher model to train a Convolutional Neural Network (CNN) for segmenting water from Synthetic Aperture Radar (SAR) images, and to train the student model, we exploit cases where optical- and radar-based water masks coincide, enabling the detection of both open and vegetated water surfaces. DeepAqua represents a significant advancement in computer vision techniques by effectively training semantic segmentation models without any manually annotated data. Experimental results show that DeepAqua outperforms other unsupervised methods by improving accuracy by 7%, Intersection Over Union by 27%, and F1 score by 14%. This approach offers a practical solution for monitoring wetland water extent changes without needing ground truth data, making it highly adaptable and scalable for wetland conservation efforts.Comment: 29 pages, 8 figures, 1 tabl

Habitat classification using convolutional neural networks and multitemporal multispectral aerial imagery

The monitoring of threatened habitats is a key objective of European environmental policies. Due to the high cost of current field-based habitat mapping techniques, there is keen interest in proposing solutions that can reduce cost through increased levels of automation. Our study aims to propose a habitat mapping solution that benefits both from the merits of convolutional neural networks (CNNs) for image classification tasks, as well as from the high spatial, spectral, and multitemporal unmanned aerial vehicle image data, which shows great potential for accurate vegetation classification. The proposed CNN-based method uses multitemporal multispectral aerial imagery for the classification of threatened coastal habitats in the Maharees (Ireland) and shows a high level of classification accuracy.This project has received funding from the European Union’s Horizon 2020 Research and Innovation program under the Marie Skłodowska-Curie Grant Agreement No. 847402. The authors would like to thank the EPA-funded iHabiMap project for providing the data used in this work. We thank the anonymous reviewers whose comments and suggestions helped improve and clarify this manuscript. The authors declare no conflicts of interes

Large Area Land Cover Mapping Using Deep Neural Networks and Landsat Time-Series Observations

This dissertation focuses on analysis and implementation of deep learning methodologies in the field of remote sensing to enhance land cover classification accuracy, which has important applications in many areas of environmental planning and natural resources management. The first manuscript conducted a land cover analysis on 26 Landsat scenes in the United States by considering six classifier variants. An extensive grid search was conducted to optimize classifier parameters using only the spectral components of each pixel. Results showed no gain in using deep networks by using only spectral components over conventional classifiers, possibly due to the small reference sample size and richness of features. The effect of changing training data size, class distribution, or scene heterogeneity were also studied and we found all of them having significant effect on classifier accuracy. The second manuscript reviewed 103 research papers on the application of deep learning methodologies in remote sensing, with emphasis on per-pixel classification of mono-temporal data and utilizing spectral and spatial data dimensions. A meta-analysis quantified deep network architecture improvement over selected convolutional classifiers. The effect of network size, learning methodology, input data dimensionality and training data size were also studied, with deep models providing enhanced performance over conventional one using spectral and spatial data. The analysis found that input dataset was a major limitation and available datasets have already been utilized to their maximum capacity. The third manuscript described the steps to build the full environment for dataset generation based on Landsat time-series data using spectral, spatial, and temporal information available for each pixel. A large dataset containing one sample block from each of 84 ecoregions in the conterminous United States (CONUS) was created and then processed by a hybrid convolutional+recurrent deep network, and the network structure was optimized with thousands of simulations. The developed model achieved an overall accuracy of 98% on the test dataset. Also, the model was evaluated for its overall and per-class performance under different conditions, including individual blocks, individual or combined Landsat sensors, and different sequence lengths. The analysis found that although the deep model performance per each block is superior to other candidates, the per block performance still varies considerably from block to block. This suggests extending the work by model fine-tuning for local areas. The analysis also found that including more time stamps or combining different Landsat sensor observations in the model input significantly enhances the model performance

A Review of Landcover Classification with Very-High Resolution Remotely Sensed Optical Images—Analysis Unit, Model Scalability and Transferability

As an important application in remote sensing, landcover classification remains one of the most challenging tasks in very-high-resolution (VHR) image analysis. As the rapidly increasing number of Deep Learning (DL) based landcover methods and training strategies are claimed to be the state-of-the-art, the already fragmented technical landscape of landcover mapping methods has been further complicated. Although there exists a plethora of literature review work attempting to guide researchers in making an informed choice of landcover mapping methods, the articles either focus on the review of applications in a specific area or revolve around general deep learning models, which lack a systematic view of the ever advancing landcover mapping methods. In addition, issues related to training samples and model transferability have become more critical than ever in an era dominated by data-driven approaches, but these issues were addressed to a lesser extent in previous review articles regarding remote sensing classification. Therefore, in this paper, we present a systematic overview of existing methods by starting from learning methods and varying basic analysis units for landcover mapping tasks, to challenges and solutions on three aspects of scalability and transferability with a remote sensing classification focus including (1) sparsity and imbalance of data; (2) domain gaps across different geographical regions; and (3) multi-source and multi-view fusion. We discuss in detail each of these categorical methods and draw concluding remarks in these developments and recommend potential directions for the continued endeavor

A Review of Landcover Classification with Very-High Resolution Remotely Sensed Optical Images—Analysis Unit, Model Scalability and Transferability

As an important application in remote sensing, landcover classification remains one of the most challenging tasks in very-high-resolution (VHR) image analysis. As the rapidly increasing number of Deep Learning (DL) based landcover methods and training strategies are claimed to be the state-of-the-art, the already fragmented technical landscape of landcover mapping methods has been further complicated. Although there exists a plethora of literature review work attempting to guide researchers in making an informed choice of landcover mapping methods, the articles either focus on the review of applications in a specific area or revolve around general deep learning models, which lack a systematic view of the ever advancing landcover mapping methods. In addition, issues related to training samples and model transferability have become more critical than ever in an era dominated by data-driven approaches, but these issues were addressed to a lesser extent in previous review articles regarding remote sensing classification. Therefore, in this paper, we present a systematic overview of existing methods by starting from learning methods and varying basic analysis units for landcover mapping tasks, to challenges and solutions on three aspects of scalability and transferability with a remote sensing classification focus including (1) sparsity and imbalance of data; (2) domain gaps across different geographical regions; and (3) multi-source and multi-view fusion. We discuss in detail each of these categorical methods and draw concluding remarks in these developments and recommend potential directions for the continued endeavor

Wetland mapping and monitoring using polarimetric and interferometric synthetic aperture radar (SAR) data and tools

Wetlands are home to a great variety of flora and fauna species and provide several unique environmental functions, such as controlling floods, improving water-quality, supporting wildlife habitat, and shoreline stabilization. Detailed information on spatial distribution of wetland classes is crucial for sustainable management and resource assessment. Furthermore, hydrological monitoring of wetlands is also important for maintaining and preserving the habitat of various plant and animal species. This thesis investigates the existing knowledge and technological challenges associated with wetland mapping and monitoring and evaluates the limitations of the methodologies that have been developed to date. The study also proposes new methods to improve the characterization of these productive ecosystems using advanced remote sensing (RS) tools and data. Specifically, a comprehensive literature review on wetland monitoring using Synthetic Aperture Radar (SAR) and Interferometric SAR (InSAR) techniques is provided. The application of the InSAR technique for wetland mapping provides the following advantages: (i) the high sensitivity of interferometric coherence to land cover changes is taken into account and (ii) the exploitation of interferometric coherence for wetland classification further enhances the discrimination between similar wetland classes. A statistical analysis of the interferometric coherence and SAR backscattering variation of Canadian wetlands, which are ignored in the literature, is carried out using multi-temporal, multi-frequency, and multi-polarization SAR data. The study also examines the capability of compact polarimetry (CP) SAR data, which will be collected by the upcoming RADARSAT Constellation Mission (RCM) and will constitute the main source of SAR observation in Canada, for wetland mapping. The research in this dissertation proposes a methodology for wetland classification using the synergistic use of intensity, polarimetry, and interferometry features using a novel classification framework. Finally, this work introduces a novel model based on the deep convolutional neural network (CNN) for wetland classification that can be trained in an end-to-end scheme and is specifically designed for the classification of wetland complexes using polarimetric SAR (PolSAR) imagery. The results of the proposed methods are promising and will significantly contribute to the ongoing efforts of conservation strategies for wetlands and monitoring changes. The approaches presented in this thesis serve as frameworks, progressing towards an operational methodology for mapping wetland complexes in Canada, as well as other wetlands worldwide with similar ecological characteristics