A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community

In recent years, deep learning (DL), a re-branding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, natural language processing, etc. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV; e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should be aware of, if not at the leading edge of, of advancements like DL. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as it relates to (i) inadequate data sets, (ii) human-understandable solutions for modelling physical phenomena, (iii) Big Data, (iv) non-traditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.Comment: 64 pages, 411 references. To appear in Journal of Applied Remote Sensin

GeoAI-enhanced Techniques to Support Geographical Knowledge Discovery from Big Geospatial Data

abstract: Big data that contain geo-referenced attributes have significantly reformed the way that I process and analyze geospatial data. Compared with the expected benefits received in the data-rich environment, more data have not always contributed to more accurate analysis. “Big but valueless” has becoming a critical concern to the community of GIScience and data-driven geography. As a highly-utilized function of GeoAI technique, deep learning models designed for processing geospatial data integrate powerful computing hardware and deep neural networks into various dimensions of geography to effectively discover the representation of data. However, limitations of these deep learning models have also been reported when People may have to spend much time on preparing training data for implementing a deep learning model. The objective of this dissertation research is to promote state-of-the-art deep learning models in discovering the representation, value and hidden knowledge of GIS and remote sensing data, through three research approaches. The first methodological framework aims to unify varied shadow into limited number of patterns, with the convolutional neural network (CNNs)-powered shape classification, multifarious shadow shapes with a limited number of representative shadow patterns for efficient shadow-based building height estimation. The second research focus integrates semantic analysis into a framework of various state-of-the-art CNNs to support human-level understanding of map content. The final research approach of this dissertation focuses on normalizing geospatial domain knowledge to promote the transferability of a CNN’s model to land-use/land-cover classification. This research reports a method designed to discover detailed land-use/land-cover types that might be challenging for a state-of-the-art CNN’s model that previously performed well on land-cover classification only.Dissertation/ThesisDoctoral Dissertation Geography 201

Spatial Analysis of Post-Hurricane Katrina Thermal Pattern and Intensity in Greater New Orleans: Implications for Urban Heat Island Research

In 2005, Hurricane Katrina’s diverse impacts on the Greater New Orleans area included damaged and destroyed trees, and other despoiled vegetation, which also increased the exposure of artificial and bare surfaces, known factors that contribute to the climatic phenomenon known as the urban heat island (UHI). This is an investigation of UHI in the aftermath of Hurricane Katrina, which entails the analysis of pre and post-hurricane Katrina thermal imagery of the study area, including changes to surface heat patterns and vegetative cover. Imagery from Landsat TM was used to show changes to the pattern and intensity of the UHI effect, caused by an extreme weather event. Using remote sensing visualization methods, field data, and local knowledge, the author found there was a measurable change in the pattern and intensity of the New Orleans UHI effect, as well as concomitant changes to vegetative land cover. This finding may be relevant for urban planners and citizens, especially in the context of recovery from a large-scale disaster of a coastal city, regarding future weather events, and other natural and human impacts

Binary Patterns Encoded Convolutional Neural Networks for Texture Recognition and Remote Sensing Scene Classification

Designing discriminative powerful texture features robust to realistic imaging conditions is a challenging computer vision problem with many applications, including material recognition and analysis of satellite or aerial imagery. In the past, most texture description approaches were based on dense orderless statistical distribution of local features. However, most recent approaches to texture recognition and remote sensing scene classification are based on Convolutional Neural Networks (CNNs). The d facto practice when learning these CNN models is to use RGB patches as input with training performed on large amounts of labeled data (ImageNet). In this paper, we show that Binary Patterns encoded CNN models, codenamed TEX-Nets, trained using mapped coded images with explicit texture information provide complementary information to the standard RGB deep models. Additionally, two deep architectures, namely early and late fusion, are investigated to combine the texture and color information. To the best of our knowledge, we are the first to investigate Binary Patterns encoded CNNs and different deep network fusion architectures for texture recognition and remote sensing scene classification. We perform comprehensive experiments on four texture recognition datasets and four remote sensing scene classification benchmarks: UC-Merced with 21 scene categories, WHU-RS19 with 19 scene classes, RSSCN7 with 7 categories and the recently introduced large scale aerial image dataset (AID) with 30 aerial scene types. We demonstrate that TEX-Nets provide complementary information to standard RGB deep model of the same network architecture. Our late fusion TEX-Net architecture always improves the overall performance compared to the standard RGB network on both recognition problems. Our final combination outperforms the state-of-the-art without employing fine-tuning or ensemble of RGB network architectures.Comment: To appear in ISPRS Journal of Photogrammetry and Remote Sensin

High resolution urban monitoring using neural network and transform algorithms

The advent of new high spatial resolution optical satellite imagery has greatly increased our ability to monitor land cover from space. Satellite observations are carried out regularly and continuously and provide a great deal of information on land cover over large areas. High spatial resolution imagery makes it possible to overcome the “mixed-pixel” problem inherent in more moderate resolution satellite sensors. At the same time, high-resolution images present a new challenge over other satellite systems since a relatively large amount of data must be analyzed, processed, and classified in order to characterize land cover features and to produce classification maps. Actually, in spite of the great potential of remote sensing as a source of information on land cover and the long history of research devoted to the extraction of land cover information from remotely sensed imagery, many problems have been encountered, and the accuracy of land cover maps derived from remotely sensed imagery has often been viewed as too low for operational users. This study focuses on high resolution urban monitoring using Neural Network (NN) analyses for land cover classification and change detection, and Fast Fourier Transform (FFT) evaluations of wavenumber spectra to characterize the spatial scales of land cover features. The contributions of the present work include: classification and change detection for urban areas using NN algorithms and multi-temporal very high resolution multi-spectral images (QuickBird, Digital Globe Co.); development and implementation of neural networks apt to classify a variety of multi-spectral images of cities arbitrarily located in the world; use of different wavenumber spectra produced by two-dimensional FFTs to understand the origin of significant features in the images of different urban environments subject to the subsequent classification; optimization of the neural net topology to classify urban environments, to produce thematic maps, and to analyze the urbanization processes. This work can considered as a first step in demonstrating how NN and FFT algorithms can contribute to the development of Image Information Mining (IMM) in Earth Observation

A Human-Centered Framework for the Understanding of Synthetic Aperture Radar Images

The limited usage of SAR data in the end-user community and in applicative contexts testified the failure of the recent literature, in which the research privileged the automatic extraction of information at the expense of users' experience with data. The development of new products and processing frameworks providing user-friendly representations and extraction of the physical information is a necessary condition for the full exploitation of SAR sensors. In this Book, the necessity to restore users’ centrality in remote sensing data analysis is analyzed and achieved through the introduction of two new classes of RGB SAR products obtained via multitemporal processing, whose principal characteristics are the ease of interpretation and the possibility to be processed with simple, end-user-oriented technique. These proposed approach aims to definitely fill the gap between the academy and the applications. The rationale is to provide ready-to-use images, in which the technical expertise with electromagnetic models, SAR imaging and image processing has been absorbed in the products formation phase. In such way, the idea that SAR images are too complicated to be interpreted and processed without a high technical expertise in order to extract physical information is overcame

Using Raster Sketches for Digital Image Retrieval

This research addresses the problem of content-based image retrieval using queries on image-object shape, completely in the raster domain. It focuses on the particularities of image databases encountered in typical topographic applications and presents the development of an environment for visual information management that enables such queries. The query consists of a user-provided raster sketch of the shape of an imaged object. The objective of the search is to retrieve images that contain an object sufficiently similar to the one specified in the query. The new contribution of this work combines the design of a comprehensive digital image database on-line query access strategy through the development of a feature library, image library and metadata library and the necessary matching tools. The matching algorithm is inspired by least-squares matching (lsm), and represents an extension of lsm to function with a variety of raster representations. The image retrieval strategy makes use of a hierarchical organization of linked feature (image-object) shapes within the feature library. The query results are ranked according to statistical scores and the user can subsequently narrow or broaden his/her search according to the previously obtained results and the purpose of the search

Using Raster Sketches for Digital Image Retrieval

This research addresses the problem of content-based image retrieval using queries on image-object shape, completely in the raster domain. It focuses on the particularities of image databases encountered in typical topographic applications and presents the development of an environment for visual information management that enables such queries. The query consists of a user-provided raster sketch of the shape of an imaged object. The objective of the search is to retrieve images that contain an object sufficiently similar to the one specified in the query. The new contribution of this work combines the design of a comprehensive digital image database on-line query access strategy through the development of a feature library, image library and metadata library and the necessary matching tools. The matching algorithm is inspired by least-squares matching (lsm), and represents an extension of lsm to function with a variety of raster representations. The image retrieval strategy makes use of a hierarchical organization of linked feature (image-object) shapes within the feature library. The query results are ranked according to statistical scores and the user can subsequently narrow or broaden his/her search according to the previously obtained results and the purpose of the search