A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community

In recent years, deep learning (DL), a re-branding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, natural language processing, etc. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV; e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should be aware of, if not at the leading edge of, of advancements like DL. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as it relates to (i) inadequate data sets, (ii) human-understandable solutions for modelling physical phenomena, (iii) Big Data, (iv) non-traditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.Comment: 64 pages, 411 references. To appear in Journal of Applied Remote Sensin

Key Information Retrieval in Hyperspectral Imagery through Spatial-Spectral Data Fusion

Hyperspectral (HS) imaging is measuring the radiance of materials within each pixel area at a large number of contiguous spectral wavelength bands. The key spatial information such as small targets and border lines are hard to be precisely detected from HS data due to the technological constraints. Therefore, the need for image processing techniques is an important field of research in HS remote sensing. A novel semisupervised spatial-spectral data fusion method for resolution enhancement of HS images through maximizing the spatial correlation of the endmembers (signature of pure or purest materials in the scene) using a superresolution mapping (SRM) technique is proposed in this paper. The method adopts a linear mixture model and a fully constrained least squares spectral unmixing algorithm to obtain the endmember abundances (fractional images) of HS images. Then, the extracted endmember distribution maps are fused with the spatial information using a spatial-spectral correlation maximizing model and a learning-based SRM technique to exploit the subpixel level data. The obtained results validate the reliability of the technique for key information retrieval. The proposed method is very efficient and is low in terms of computational cost which makes it favorable for real-time applications

Subpixel Target Enhancement in Hyperspectral Images

Hyperspectral images due to their higher spectral resolution are increasingly being used for various remote sensing applications including information extraction at subpixel level. Typically whenever an object gets spectrally resolved but not spatially, mixed pixels in the images result. Numerous man made and/or natural disparatetar gets may thus occur inside such mixed pixels giving rise to subpixel target detection problem. Various spectral unmixing models such as linear mixture modeling (LMM) are in vogue to recover components of a mixed pixel. Spectral unmixing outputs both the endmember spectrum and their corresponding a bundance fractions inside the pixel. It, however, does not provide spatial distribution of these abundance fractions within a pixel. This limits the applicability of hyperspectral data for subpixel target detection. In this paper, a new inverse Euclidean distance based super-resolution mapping method has been presented. In this method, the subpixel target detection is performed by adjusting spatial distribution of abundance fraction within a pixel of an hyperspectral image. Results obtainedat different resolutions indicate that super-resolution mapping may effectively be utilized in enhancing the target detection at sub-pixel level.Defence Science Journal, 2013, 63(1), pp.63-68, DOI:http://dx.doi.org/10.14429/dsj.63.376

Nonlinear unmixing of hyperspectral images: Models and algorithms

When considering the problem of unmixing hyperspectral images, most of the literature in the geoscience and image processing areas relies on the widely used linear mixing model (LMM). However, the LMM may be not valid, and other nonlinear models need to be considered, for instance, when there are multiscattering effects or intimate interactions. Consequently, over the last few years, several significant contributions have been proposed to overcome the limitations inherent in the LMM. In this article, we present an overview of recent advances in nonlinear unmixing modeling

Reducing the impacts of intra-class spectral variability on the accuracy of soft classification and super-resolution mapping of shoreline

The main objective of this research is to assess the impact of intra-class spectral variation on the accuracy of soft classification and super-resolution mapping. The accuracy of both analyses was negatively related to the degree of intra-class spectral variation, but the effect could be reduced through use of spectral sub-classes. The latter is illustrated in mapping the shoreline at a sub-pixel scale from Landsat ETM+ data. Reducing the degree of intra-class spectral variation increased the accuracy of soft classification, with the correlation between predicted and actual class coverage rising from 0.87 to 0.94, and super-resolution mapping, with the RMSE in shoreline location decreasing from 41.13 m to 35.22 m

Assessing the role of EO in biodiversity monitoring: options for integrating in-situ observations with EO within the context of the EBONE concept

The European Biodiversity Observation Network (EBONE) is a European contribution on terrestrial monitoring to GEO BON, the Group on Earth Observations Biodiversity Observation Network. EBONE’s aims are to develop a system of biodiversity observation at regional, national and European levels by assessing existing approaches in terms of their validity and applicability starting in Europe, then expanding to regions in Africa. The objective of EBONE is to deliver: 1. A sound scientific basis for the production of statistical estimates of stock and change of key indicators; 2. The development of a system for estimating past changes and forecasting and testing policy options and management strategies for threatened ecosystems and species; 3. A proposal for a cost-effective biodiversity monitoring system. There is a consensus that Earth Observation (EO) has a role to play in monitoring biodiversity. With its capacity to observe detailed spatial patterns and variability across large areas at regular intervals, our instinct suggests that EO could deliver the type of spatial and temporal coverage that is beyond reach with in-situ efforts. Furthermore, when considering the emerging networks of in-situ observations, the prospect of enhancing the quality of the information whilst reducing cost through integration is compelling. This report gives a realistic assessment of the role of EO in biodiversity monitoring and the options for integrating in-situ observations with EO within the context of the EBONE concept (cfr. EBONE-ID1.4). The assessment is mainly based on a set of targeted pilot studies. Building on this assessment, the report then presents a series of recommendations on the best options for using EO in an effective, consistent and sustainable biodiversity monitoring scheme. The issues that we faced were many: 1. Integration can be interpreted in different ways. One possible interpretation is: the combined use of independent data sets to deliver a different but improved data set; another is: the use of one data set to complement another dataset. 2. The targeted improvement will vary with stakeholder group: some will seek for more efficiency, others for more reliable estimates (accuracy and/or precision); others for more detail in space and/or time or more of everything. 3. Integration requires a link between the datasets (EO and in-situ). The strength of the link between reflected electromagnetic radiation and the habitats and their biodiversity observed in-situ is function of many variables, for example: the spatial scale of the observations; timing of the observations; the adopted nomenclature for classification; the complexity of the landscape in terms of composition, spatial structure and the physical environment; the habitat and land cover types under consideration. 4. The type of the EO data available varies (function of e.g. budget, size and location of region, cloudiness, national and/or international investment in airborne campaigns or space technology) which determines its capability to deliver the required output. EO and in-situ could be combined in different ways, depending on the type of integration we wanted to achieve and the targeted improvement. We aimed for an improvement in accuracy (i.e. the reduction in error of our indicator estimate calculated for an environmental zone). Furthermore, EO would also provide the spatial patterns for correlated in-situ data. EBONE in its initial development, focused on three main indicators covering: (i) the extent and change of habitats of European interest in the context of a general habitat assessment; (ii) abundance and distribution of selected species (birds, butterflies and plants); and (iii) fragmentation of natural and semi-natural areas. For habitat extent, we decided that it did not matter how in-situ was integrated with EO as long as we could demonstrate that acceptable accuracies could be achieved and the precision could consistently be improved. The nomenclature used to map habitats in-situ was the General Habitat Classification. We considered the following options where the EO and in-situ play different roles: using in-situ samples to re-calibrate a habitat map independently derived from EO; improving the accuracy of in-situ sampled habitat statistics, by post-stratification with correlated EO data; and using in-situ samples to train the classification of EO data into habitat types where the EO data delivers full coverage or a larger number of samples. For some of the above cases we also considered the impact that the sampling strategy employed to deliver the samples would have on the accuracy and precision achieved. Restricted access to European wide species data prevented work on the indicator ‘abundance and distribution of species’. With respect to the indicator ‘fragmentation’, we investigated ways of delivering EO derived measures of habitat patterns that are meaningful to sampled in-situ observations