A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community

In recent years, deep learning (DL), a re-branding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, natural language processing, etc. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV; e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should be aware of, if not at the leading edge of, of advancements like DL. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as it relates to (i) inadequate data sets, (ii) human-understandable solutions for modelling physical phenomena, (iii) Big Data, (iv) non-traditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.Comment: 64 pages, 411 references. To appear in Journal of Applied Remote Sensin

Hyperspectral Unmixing Overview: Geometrical, Statistical, and Sparse Regression-Based Approaches

Imaging spectrometers measure electromagnetic energy scattered in their instantaneous field view in hundreds or thousands of spectral channels with higher spectral resolution than multispectral cameras. Imaging spectrometers are therefore often referred to as hyperspectral cameras (HSCs). Higher spectral resolution enables material identification via spectroscopic analysis, which facilitates countless applications that require identifying materials in scenarios unsuitable for classical spectroscopic analysis. Due to low spatial resolution of HSCs, microscopic material mixing, and multiple scattering, spectra measured by HSCs are mixtures of spectra of materials in a scene. Thus, accurate estimation requires unmixing. Pixels are assumed to be mixtures of a few materials, called endmembers. Unmixing involves estimating all or some of: the number of endmembers, their spectral signatures, and their abundances at each pixel. Unmixing is a challenging, ill-posed inverse problem because of model inaccuracies, observation noise, environmental conditions, endmember variability, and data set size. Researchers have devised and investigated many models searching for robust, stable, tractable, and accurate unmixing algorithms. This paper presents an overview of unmixing methods from the time of Keshava and Mustard's unmixing tutorial [1] to the present. Mixing models are first discussed. Signal-subspace, geometrical, statistical, sparsity-based, and spatial-contextual unmixing algorithms are described. Mathematical problems and potential solutions are described. Algorithm characteristics are illustrated experimentally.Comment: This work has been accepted for publication in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensin

An Automatic Statistical Segmentation Algorithm for Extraction of Fire and Smoke Regions

Estimation of the extent and spread of wildland fires is an important application of high spatial resolution multispectral images. This work addresses a fuzzy segmentation algorithm to map fire extent, active fire front, hot burn scar, and smoke regions based on a statistical model. The fuzzy results are useful data sources for integrated fire behavior and propagation models built using Dynamic Data Driven Applications Systems (DDDAS) concepts that use data assimilation techniques which require error estimates or probabilities for the data parameters. The Hidden Markov Random Field (HMRF) model has been used widely in image segmentation, but it is assumed that each pixel has a particular class label belonging to a prescribed finite set. The mixed pixel problem can be addressed by modeling the fuzzy membership process as a continuous Multivariate Gaussian Markov Random Field. Techniques for estimating the class membership and model parameters are discussed. Experimental results obtained by applying this technique to two Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) images show that the proposed methodology is robust with regard to noise and variation in fire characteristics as well as background. The segmentation results of our algorithm are compared with the results of a K-means algorithm, an Expectation Maximization (EM) algorithm (which is very similar to the Fuzzy C-Means Clustering algorithm with entropy regularization), and an MRF-MAP algorithm. Our fuzzy algorithm achieves more consistent segmentation results than the comparison algorithms for these test images with the added advantage of simultaneously providing a proportion or error map needed for the data assimilation problem

Maximizing the relationship of yield to site-specific management zones with object-oriented segmentation of hyperspectral images

Quick and low cost delineation of site-specific management zones (SSMZ) would improve applications of precision agriculture. In this study, a new method for delineating SSMZ using object-oriented segmentation of airborne imagery was demonstrated. Three remote sensing domains—spectral, spatial, and temporal- are exploited to improve the SSMZ relationship to yield. Common vegetation indices (VI), and first and second derivatives ([Formula: see text], [Formula: see text]) from twelve airborne hyperspectral images of a cotton field for one season [Formula: see text] were used as input layers for object-oriented segmentation. The optimal combination of VI, SSMZ size and crop phenological stage were used as input variables for SSMZ delineation, determined by maximizing the correlation to segmented yield monitor maps. Combining narrow band vegetation indices and object-oriented segmentation provided higher correlation between VI and yield at SSMZ scale than that at pixel scale by reducing multi-resource data noise. VI performance varied during the cotton growing season, providing better SSMZ delineation at the beginning and middle of the season (days after planting (DAP) 66–143).The optimal scale determined for SSMZ delineation was approximately 240 polygons for the study field, but the method also provided flexibility enabling the setting of practical scales for a given field. For a defined scale, the optimal single phenological stage for the study field was near July 11 (DAP 87) early in the growing season. SSMZs determined from multispectral VIs at a single stage were also satisfactory; compared to hyperspectral indices, temporal resolution of multi-spectral data seems more important for SSMZ delineation

Robust Modular Feature-Based Terrain-Aided Visual Navigation and Mapping

The visual feature-based Terrain-Aided Navigation (TAN) system presented in this thesis addresses the problem of constraining inertial drift introduced into the location estimate of Unmanned Aerial Vehicles (UAVs) in GPS-denied environment. The presented TAN system utilises salient visual features representing semantic or human-interpretable objects (roads, forest and water boundaries) from onboard aerial imagery and associates them to a database of reference features created a-priori, through application of the same feature detection algorithms to satellite imagery. Correlation of the detected features with the reference features via a series of the robust data association steps allows a localisation solution to be achieved with a finite absolute bound precision defined by the certainty of the reference dataset. The feature-based Visual Navigation System (VNS) presented in this thesis was originally developed for a navigation application using simulated multi-year satellite image datasets. The extension of the system application into the mapping domain, in turn, has been based on the real (not simulated) flight data and imagery. In the mapping study the full potential of the system, being a versatile tool for enhancing the accuracy of the information derived from the aerial imagery has been demonstrated. Not only have the visual features, such as road networks, shorelines and water bodies, been used to obtain a position ’fix’, they have also been used in reverse for accurate mapping of vehicles detected on the roads into an inertial space with improved precision. Combined correction of the geo-coding errors and improved aircraft localisation formed a robust solution to the defense mapping application. A system of the proposed design will provide a complete independent navigation solution to an autonomous UAV and additionally give it object tracking capability