Deep learning in remote sensing: a review

Standing at the paradigm shift towards data-intensive science, machine learning techniques are becoming increasingly important. In particular, as a major breakthrough in the field, deep learning has proven as an extremely powerful tool in many fields. Shall we embrace deep learning as the key to all? Or, should we resist a 'black-box' solution? There are controversial opinions in the remote sensing community. In this article, we analyze the challenges of using deep learning for remote sensing data analysis, review the recent advances, and provide resources to make deep learning in remote sensing ridiculously simple to start with. More importantly, we advocate remote sensing scientists to bring their expertise into deep learning, and use it as an implicit general model to tackle unprecedented large-scale influential challenges, such as climate change and urbanization.Comment: Accepted for publication IEEE Geoscience and Remote Sensing Magazin

A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community

In recent years, deep learning (DL), a re-branding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, natural language processing, etc. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV; e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should be aware of, if not at the leading edge of, of advancements like DL. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as it relates to (i) inadequate data sets, (ii) human-understandable solutions for modelling physical phenomena, (iii) Big Data, (iv) non-traditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.Comment: 64 pages, 411 references. To appear in Journal of Applied Remote Sensin

Binary Patterns Encoded Convolutional Neural Networks for Texture Recognition and Remote Sensing Scene Classification

Designing discriminative powerful texture features robust to realistic imaging conditions is a challenging computer vision problem with many applications, including material recognition and analysis of satellite or aerial imagery. In the past, most texture description approaches were based on dense orderless statistical distribution of local features. However, most recent approaches to texture recognition and remote sensing scene classification are based on Convolutional Neural Networks (CNNs). The d facto practice when learning these CNN models is to use RGB patches as input with training performed on large amounts of labeled data (ImageNet). In this paper, we show that Binary Patterns encoded CNN models, codenamed TEX-Nets, trained using mapped coded images with explicit texture information provide complementary information to the standard RGB deep models. Additionally, two deep architectures, namely early and late fusion, are investigated to combine the texture and color information. To the best of our knowledge, we are the first to investigate Binary Patterns encoded CNNs and different deep network fusion architectures for texture recognition and remote sensing scene classification. We perform comprehensive experiments on four texture recognition datasets and four remote sensing scene classification benchmarks: UC-Merced with 21 scene categories, WHU-RS19 with 19 scene classes, RSSCN7 with 7 categories and the recently introduced large scale aerial image dataset (AID) with 30 aerial scene types. We demonstrate that TEX-Nets provide complementary information to standard RGB deep model of the same network architecture. Our late fusion TEX-Net architecture always improves the overall performance compared to the standard RGB network on both recognition problems. Our final combination outperforms the state-of-the-art without employing fine-tuning or ensemble of RGB network architectures.Comment: To appear in ISPRS Journal of Photogrammetry and Remote Sensin

Object Detection in High Resolution Aerial Images and Hyperspectral Remote Sensing Images

With rapid developments in satellite and sensor technologies, there has been a dramatic increase in the availability of remotely sensed images. However, the exploration of these images still involves a tremendous amount of human interventions, which are tedious, time-consuming, and inefficient. To help imaging experts gain a complete understanding of the images and locate the objects of interest in a more accurate and efficient way, there is always an urgent need for developing automatic detection algorithms. In this work, we delve into the object detection problems in remote sensing applications, exploring the detection algorithms for both hyperspectral images (HSIs) and high resolution aerial images. In the first part, we focus on the subpixel target detection problem in HSIs with low spatial resolutions, where the objects of interest are much smaller than the image pixel spatial resolution. To this end, we explore the detection frameworks that integrate image segmentation techniques in designing the matched filters (MFs). In particular, we propose a novel image segmentation algorithm to identify the spatial-spectral coherent image regions, from which the background statistics were estimated for deriving the MFs. Extensive experimental studies were carried out to demonstrate the advantages of the proposed subpixel target detection framework. Our studies show the superiority of the approach when comparing to state-of-the-art methods. The second part of the thesis explores the object based image analysis (OBIA) framework for geospatial object detection in high resolution aerial images. Specifically, we generate a tree representation of the aerial images from the output of hierarchical image segmentation algorithms and reformulate the object detection problem into a tree matching task. We then proposed two tree-matching algorithms for the object detection framework. We demonstrate the efficiency and effectiveness of the proposed tree-matching based object detection framework. In the third part, we study object detection in high resolution aerial images from a machine learning perspective. We investigate both traditional machine learning based framework and end-to-end convolutional neural network (CNN) based approach for various object detection tasks. In the traditional detection framework, we propose to apply the Gaussian process classifier (GPC) to train an object detector and demonstrate the advantages of the probabilistic classification algorithm. In the CNN based approach, we proposed a novel scale transfer module that generates enhanced feature maps for object detection. Our results show the efficiency and competitiveness of the proposed algorithms when compared to state-of-the-art counterparts

A review of technical factors to consider when designing neural networks for semantic segmentation of Earth Observation imagery

Semantic segmentation (classification) of Earth Observation imagery is a crucial task in remote sensing. This paper presents a comprehensive review of technical factors to consider when designing neural networks for this purpose. The review focuses on Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Generative Adversarial Networks (GANs), and transformer models, discussing prominent design patterns for these ANN families and their implications for semantic segmentation. Common pre-processing techniques for ensuring optimal data preparation are also covered. These include methods for image normalization and chipping, as well as strategies for addressing data imbalance in training samples, and techniques for overcoming limited data, including augmentation techniques, transfer learning, and domain adaptation. By encompassing both the technical aspects of neural network design and the data-related considerations, this review provides researchers and practitioners with a comprehensive and up-to-date understanding of the factors involved in designing effective neural networks for semantic segmentation of Earth Observation imagery.Comment: 145 pages with 32 figure

Deep Learning Based Classification Techniques for Hyperspectral Images in Real Time

Remote sensing can be defined as the acquisition of information from a given scene without coming into physical contact with it, through the use of sensors, mainly located on aerial platforms, which capture information in different ranges of the electromagnetic spectrum. The objective of this thesis is the development of efficient schemes, based on the use of deep learning neural networks, for the classification of remotely sensed multi and hyperspectral land cover images. Efficient schemes are those that are capable of obtaining good results in terms of classification accuracy and that can be computed in a reasonable amount of time depending on the task performed. Regarding computational platforms, multicore architectures and Graphics Processing Units (GPUs) will be considered