Deep learning in remote sensing: a review

Standing at the paradigm shift towards data-intensive science, machine learning techniques are becoming increasingly important. In particular, as a major breakthrough in the field, deep learning has proven as an extremely powerful tool in many fields. Shall we embrace deep learning as the key to all? Or, should we resist a 'black-box' solution? There are controversial opinions in the remote sensing community. In this article, we analyze the challenges of using deep learning for remote sensing data analysis, review the recent advances, and provide resources to make deep learning in remote sensing ridiculously simple to start with. More importantly, we advocate remote sensing scientists to bring their expertise into deep learning, and use it as an implicit general model to tackle unprecedented large-scale influential challenges, such as climate change and urbanization.Comment: Accepted for publication IEEE Geoscience and Remote Sensing Magazin

Multi-frequency PolSAR Image Fusion Classification Based on Semantic Interactive Information and Topological Structure

Compared with the rapid development of single-frequency multi-polarization SAR image classification technology, there is less research on the land cover classification of multifrequency polarimetric SAR (MF-PolSAR) images. In addition, the current deep learning methods for MF-PolSAR classification are mainly based on convolutional neural networks (CNNs), only local spatiality is considered but the nonlocal relationship is ignored. Therefore, based on semantic interaction and nonlocal topological structure, this paper proposes the MF semantics and topology fusion network (MF-STFnet) to improve MF-PolSAR classification performance. In MF-STFnet, two kinds of classification are implemented for each band, semantic information-based (SIC) and topological property-based (TPC). They work collaboratively during MF-STFnet training, which can not only fully leverage the complementarity of bands, but also combine local and nonlocal spatial information to improve the discrimination between different categories. For SIC, the designed crossband interactive feature extraction module (CIFEM) is embedded to explicitly model the deep semantic correlation among bands, thereby leveraging the complementarity of bands to make ground objects more separable. For TPC, the graph sample and aggregate network (GraphSAGE) is employed to dynamically capture the representation of nonlocal topological relations between land cover categories. In this way, the robustness of classification can be further improved by combining nonlocal spatial information. Finally, an adaptive weighting fusion (AWF) strategy is proposed to merge inference from different bands, so as to make the MF joint classification decisions of SIC and TPC. The comparative experiments show that MF-STFnet can achieve more competitive classification performance than some state-of-the-art methods

Optimum graph cuts for pruning binary partition trees of polarimetric SAR images

This paper investigates several optimum graph-cut techniques for pruning binary partition trees (BPTs) and their usefulness for the low-level processing of polarimetric synthetic aperture radar (PolSAR) images. BPTs group pixels to form homogeneous regions, which are hierarchically structured by inclusion in a binary tree. They provide multiple resolutions of description and easy access to subsets of regions. Once constructed, BPTs can be used for a large number of applications. Many of these applications consist in populating the tree with a specific feature and in applying a graph cut called pruning to extract a partition of the space. In this paper, different pruning examples involving the optimization of a global criterion are discussed and analyzed in the context of PolSAR images for segmentation. Through the objective evaluation of the resulting partitions by means of precision-and-recall-for-boundaries curves, the best pruning technique is identified, and the influence of the tree construction on the performances is assessed.Peer ReviewedPostprint (author's final draft

Unsupervised classification of multilook polarimetric SAR data using spatially variant wishart mixture model with double constraints

This paper addresses the unsupervised classification problems for multilook Polarimetric synthetic aperture radar (PolSAR) images by proposing a patch-level spatially variant Wishart mixture model (SVWMM) with double constraints. We construct this model by jointly modeling the pixels in a patch (rather than an individual pixel) so as to effectively capture the local correlation in the PolSAR images. More importantly, a responsibility parameter is introduced to the proposed model, providing not only the possibility to represent the importance of different pixels within a patch but also the additional flexibility for incorporating the spatial information. As such, double constraints are further imposed by simultaneously utilizing the similarities of the neighboring pixels, respectively, defined on two different parameter spaces (i.e., the hyperparameter in the posterior distribution of mixing coefficients and the responsibility parameter). Furthermore, the variational inference algorithm is developed to achieve effective learning of the proposed SVWMM with the closed-form updates, facilitating the automatic determination of the cluster number. Experimental results on several PolSAR data sets from both airborne and spaceborne sensors demonstrate that the proposed method is effective and it enables better performances on unsupervised classification than the conventional methods

Accuracy Analysis Comparison of Supervised Classification Methods for Anomaly Detection on Levees Using SAR Imagery

This paper analyzes the use of a synthetic aperture radar (SAR) imagery to support levee condition assessment by detecting potential slide areas in an efficient and cost-effective manner. Levees are prone to a failure in the form of internal erosion within the earthen structure and landslides (also called slough or slump slides). If not repaired, slough slides may lead to levee failures. In this paper, we compare the accuracy of the supervised classification methods minimum distance (MD) using Euclidean and Mahalanobis distance, support vector machine (SVM), and maximum likelihood (ML), using SAR technology to detect slough slides on earthen levees. In this work, the effectiveness of the algorithms was demonstrated using quad-polarimetric L-band SAR imagery from the NASA Jet Propulsion Laboratory’s (JPL’s) uninhabited aerial vehicle synthetic aperture radar (UAVSAR). The study area is a section of the lower Mississippi River valley in the Southern USA, where earthen flood control levees are maintained by the US Army Corps of Engineers

Analysis of Polarimetric Synthetic Aperture Radar and Passive Visible Light Polarimetric Imaging Data Fusion for Remote Sensing Applications

The recent launch of spaceborne (TerraSAR-X, RADARSAT-2, ALOS-PALSAR, RISAT) and airborne (SIRC, AIRSAR, UAVSAR, PISAR) polarimetric radar sensors, with capability of imaging through day and night in almost all weather conditions, has made polarimetric synthetic aperture radar (PolSAR) image interpretation and analysis an active area of research. PolSAR image classification is sensitive to object orientation and scattering properties. In recent years, significant work has been done in many areas including agriculture, forestry, oceanography, geology, terrain analysis. Visible light passive polarimetric imaging has also emerged as a powerful tool in remote sensing for enhanced information extraction. The intensity image provides information on materials in the scene while polarization measurements capture surface features, roughness, and shading, often uncorrelated with the intensity image. Advantages of visible light polarimetric imaging include high dynamic range of polarimetric signatures and being comparatively straightforward to build and calibrate. This research is about characterization and analysis of the basic scattering mechanisms for information fusion between PolSAR and passive visible light polarimetric imaging. Relationships between these two modes of imaging are established using laboratory measurements and image simulations using the Digital Image and Remote Sensing Image Generation (DIRSIG) tool. A novel low cost laboratory based S-band (2.4GHz) PolSAR instrument is developed that is capable of capturing 4 channel fully polarimetric SAR image data. Simple radar targets are formed and system calibration is performed in terms of radar cross-section. Experimental measurements are done using combination of the PolSAR instrument with visible light polarimetric imager for scenes capturing basic scattering mechanisms for phenomenology studies. The three major scattering mechanisms studied in this research include single, double and multiple bounce. Single bounce occurs from flat surfaces like lakes, rivers, bare soil, and oceans. Double bounce can be observed from two adjacent surfaces where one horizontal flat surface is near a vertical surface such as buildings and other vertical structures. Randomly oriented scatters in homogeneous media produce a multiple bounce scattering effect which occurs in forest canopies and vegetated areas. Relationships between Pauli color components from PolSAR and Degree of Linear Polarization (DOLP) from passive visible light polarimetric imaging are established using real measurements. Results show higher values of the red channel in Pauli color image (|HH-VV|) correspond to high DOLP from double bounce effect. A novel information fusion technique is applied to combine information from the two modes. In this research, it is demonstrated that the Degree of Linear Polarization (DOLP) from passive visible light polarimetric imaging can be used for separation of the classes in terms of scattering mechanisms from the PolSAR data. The separation of these three classes in terms of the scattering mechanisms has its application in the area of land cover classification and anomaly detection. The fusion of information from these particular two modes of imaging, i.e. PolSAR and passive visible light polarimetric imaging, is a largely unexplored area in remote sensing and the main challenge in this research is to identify areas and scenarios where information fusion between the two modes is advantageous for separation of the classes in terms of scattering mechanisms relative to separation achieved with only PolSAR

Support matrix machine: A review

Support vector machine (SVM) is one of the most studied paradigms in the realm of machine learning for classification and regression problems. It relies on vectorized input data. However, a significant portion of the real-world data exists in matrix format, which is given as input to SVM by reshaping the matrices into vectors. The process of reshaping disrupts the spatial correlations inherent in the matrix data. Also, converting matrices into vectors results in input data with a high dimensionality, which introduces significant computational complexity. To overcome these issues in classifying matrix input data, support matrix machine (SMM) is proposed. It represents one of the emerging methodologies tailored for handling matrix input data. The SMM method preserves the structural information of the matrix data by using the spectral elastic net property which is a combination of the nuclear norm and Frobenius norm. This article provides the first in-depth analysis of the development of the SMM model, which can be used as a thorough summary by both novices and experts. We discuss numerous SMM variants, such as robust, sparse, class imbalance, and multi-class classification models. We also analyze the applications of the SMM model and conclude the article by outlining potential future research avenues and possibilities that may motivate academics to advance the SMM algorithm