A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community

In recent years, deep learning (DL), a re-branding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, natural language processing, etc. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV; e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should be aware of, if not at the leading edge of, of advancements like DL. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as it relates to (i) inadequate data sets, (ii) human-understandable solutions for modelling physical phenomena, (iii) Big Data, (iv) non-traditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.Comment: 64 pages, 411 references. To appear in Journal of Applied Remote Sensin

Advanced machine learning algorithms for Canadian wetland mapping using polarimetric synthetic aperture radar (PolSAR) and optical imagery

Wetlands are complex land cover ecosystems that represent a wide range of biophysical conditions. They are one of the most productive ecosystems and provide several important environmental functionalities. As such, wetland mapping and monitoring using cost- and time-efficient approaches are of great interest for sustainable management and resource assessment. In this regard, satellite remote sensing data are greatly beneficial, as they capture a synoptic and multi-temporal view of landscapes. The ability to extract useful information from satellite imagery greatly affects the accuracy and reliability of the final products. This is of particular concern for mapping complex land cover ecosystems, such as wetlands, where complex, heterogeneous, and fragmented landscape results in similar backscatter/spectral signatures of land cover classes in satellite images. Accordingly, the overarching purpose of this thesis is to contribute to existing methodologies of wetland classification by proposing and developing several new techniques based on advanced remote sensing tools and optical and Synthetic Aperture Radar (SAR) imagery. Specifically, the importance of employing an efficient speckle reduction method for polarimetric SAR (PolSAR) image processing is discussed and a new speckle reduction technique is proposed. Two novel techniques are also introduced for improving the accuracy of wetland classification. In particular, a new hierarchical classification algorithm using multi-frequency SAR data is proposed that discriminates wetland classes in three steps depending on their complexity and similarity. The experimental results reveal that the proposed method is advantageous for mapping complex land cover ecosystems compared to single stream classification approaches, which have been extensively used in the literature. Furthermore, a new feature weighting approach is proposed based on the statistical and physical characteristics of PolSAR data to improve the discrimination capability of input features prior to incorporating them into the classification scheme. This study also demonstrates the transferability of existing classification algorithms, which have been developed based on RADARSAT-2 imagery, to compact polarimetry SAR data that will be collected by the upcoming RADARSAT Constellation Mission (RCM). The capability of several well-known deep Convolutional Neural Network (CNN) architectures currently employed in computer vision is first introduced in this thesis for classification of wetland complexes using multispectral remote sensing data. Finally, this research results in the first provincial-scale wetland inventory maps of Newfoundland and Labrador using the Google Earth Engine (GEE) cloud computing resources and open access Earth Observation (EO) collected by the Copernicus Sentinel missions. Overall, the methodologies proposed in this thesis address fundamental limitations/challenges of wetland mapping using remote sensing data, which have been ignored in the literature. These challenges include the backscattering/spectrally similar signature of wetland classes, insufficient classification accuracy of wetland classes, and limitations of wetland mapping on large scales. In addition to the capabilities of the proposed methods for mapping wetland complexes, the use of these developed techniques for classifying other complex land cover types beyond wetlands, such as sea ice and crop ecosystems, offers a potential avenue for further research

Synthetic Aperture Radar (SAR) Meets Deep Learning

This reprint focuses on the application of the combination of synthetic aperture radars and depth learning technology. It aims to further promote the development of SAR image intelligent interpretation technology. A synthetic aperture radar (SAR) is an important active microwave imaging sensor, whose all-day and all-weather working capacity give it an important place in the remote sensing community. Since the United States launched the first SAR satellite, SAR has received much attention in the remote sensing community, e.g., in geological exploration, topographic mapping, disaster forecast, and traffic monitoring. It is valuable and meaningful, therefore, to study SAR-based remote sensing applications. In recent years, deep learning represented by convolution neural networks has promoted significant progress in the computer vision community, e.g., in face recognition, the driverless field and Internet of things (IoT). Deep learning can enable computational models with multiple processing layers to learn data representations with multiple-level abstractions. This can greatly improve the performance of various applications. This reprint provides a platform for researchers to handle the above significant challenges and present their innovative and cutting-edge research results when applying deep learning to SAR in various manuscript types, e.g., articles, letters, reviews and technical reports

Study of the speckle noise effects over the eigen decomposition of polarimetric SAR data: a review

This paper is focused on considering the effects of speckle noise on the eigen decomposition of the co- herency matrix. Based on a perturbation analysis of the matrix, it is possible to obtain an analytical expression for the mean value of the eigenvalues and the eigenvectors, as well as for the Entropy, the Anisotroopy and the dif- ferent a angles. The analytical expressions are compared against simulated polarimetric SAR data, demonstrating the correctness of the different expressions.Peer ReviewedPostprint (published version

A review of technical factors to consider when designing neural networks for semantic segmentation of Earth Observation imagery

Semantic segmentation (classification) of Earth Observation imagery is a crucial task in remote sensing. This paper presents a comprehensive review of technical factors to consider when designing neural networks for this purpose. The review focuses on Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Generative Adversarial Networks (GANs), and transformer models, discussing prominent design patterns for these ANN families and their implications for semantic segmentation. Common pre-processing techniques for ensuring optimal data preparation are also covered. These include methods for image normalization and chipping, as well as strategies for addressing data imbalance in training samples, and techniques for overcoming limited data, including augmentation techniques, transfer learning, and domain adaptation. By encompassing both the technical aspects of neural network design and the data-related considerations, this review provides researchers and practitioners with a comprehensive and up-to-date understanding of the factors involved in designing effective neural networks for semantic segmentation of Earth Observation imagery.Comment: 145 pages with 32 figure

Sea Ice Mapping in Labrador Coast with Sentinel-1 Synthetic Aperture Radar Imagery

Sea ice mapping is crucial to Canadian coast, including marine transportation, environmental protection, resource management, disaster and emergency management, especially under current background of climate change. Canadian RADARSAT-2, like other synthetic aperture radar (SAR) sensors, is an essential source for current sea ice mapping in Canada, However, its limited revisiting makes daily ice chart generation challenging. The RADARSAT Constellation project is expected to be launched in 2018, the gap of data availability is expected to be filled with imagery from multiple sources. Sentinel-1, launched by European Space Agency (ESA) in late 2014, is an alternative source for sea ice mapping with comparable capability of RADARSAT-2 in wide swath mode. The main objective of this study is to examine the performance of Sentinel-1 imagery in sea ice mapping with a semi-automated image segmentation workflow. The methodology consists of two main steps. First, the most significant features in sea ice interpretation were determined using a random forest feature selection method. Second, an unsupervised graph-cut image segmentation is performed. The workflow was tested on 15 dual-polarized Sentinel-1A Extra Wide (EW) scenes in Labrador coast from December, 2015 to June, 2016, and the results were evaluated on the accuracy of water segmentation. The study found that: 1) GLCM features are effective in distinguishing different ice classes and 6 most important features were selected; 2) the proposed semi-automated workflow is able to segment Sentinel-1 imagery into 3 to 8 classes for water identification; and 3) generally Sentinel-1 imagery has similar responses from first-year ice compared with previous sensors, but with a different noise pattern in cross-polarized bands; and the overall accuracy of water identification reached close to 95%

Combination of optical and SAR remote sensing data for wetland mapping and monitoring

Wetlands provide many services to the environment and humans. They play a pivotal role in water quality, climate change, as well as carbon and hydrological cycles. Wetlands are environmental health indicators because of their contributions to plant and animal habitats. While a large portion of Newfoundland and Labrador (NL) is covered by wetlands, no significant efforts had been conducted to identify and monitor these valuable environments when I initiated this project. At that time, there were only two small areas in NL that had been classified using basic Remote Sensing (RS) methods with low accuracies. There was an immediate need to develop new methods for conserving and managing these vital resources using up-to-date maps of wetland distributions. In this thesis, object- and pixel-based classification methods were compared to show the high potential of the former method when medium or high spatial resolution imagery were used to classify wetlands. The maps produced using several classification algorithms were also compared to select the optimum classifier for future experiments. Moreover, a novel Multiple Classifier System (MCS), which combined several algorithms, was proposed to increase the classification accuracy of complex and similar land covers, such as wetlands. Landsat-8 images captured in different months were also investigated to select the time, for which wetlands had the highest separability using the Random Forest (RF) algorithm. Additionally, various spectral, polarimetric, texture, and ratio features extracted from multi-source optical and Synthetic Aperture Radar (SAR) data were assessed to select the most effective features for discriminating wetland classes. The methods developed during this dissertation were validated in five study areas to show their effectiveness. Finally, in collaboration with a team, a website (http://nlwetlands.ca/) and a software package were developed (named the Advanced Remote Sensing Lab (ARSeL)) to automatically preprocess optical/SAR data and classify wetlands using advanced algorithms. In summary, the outputs of this work are promising and can be incorporated into future studies related to wetlands. The province can also benefit from the results in many ways

Deep representation learning: Fundamentals, Perspectives, Applications, and Open Challenges

Machine Learning algorithms have had a profound impact on the field of computer science over the past few decades. These algorithms performance is greatly influenced by the representations that are derived from the data in the learning process. The representations learned in a successful learning process should be concise, discrete, meaningful, and able to be applied across a variety of tasks. A recent effort has been directed toward developing Deep Learning models, which have proven to be particularly effective at capturing high-dimensional, non-linear, and multi-modal characteristics. In this work, we discuss the principles and developments that have been made in the process of learning representations, and converting them into desirable applications. In addition, for each framework or model, the key issues and open challenges, as well as the advantages, are examined

Remote Sensing in Mangroves

The book highlights recent advancements in the mapping and monitoring of mangrove forests using earth observation satellite data. New and historical satellite data and aerial photographs have been used to map the extent, change and bio-physical parameters, such as phenology and biomass. Research was conducted in different parts of the world. Knowledge and understanding gained from this book can be used for the sustainable management of mangrove forests of the worl