Deep learning in remote sensing: a review

Standing at the paradigm shift towards data-intensive science, machine learning techniques are becoming increasingly important. In particular, as a major breakthrough in the field, deep learning has proven as an extremely powerful tool in many fields. Shall we embrace deep learning as the key to all? Or, should we resist a 'black-box' solution? There are controversial opinions in the remote sensing community. In this article, we analyze the challenges of using deep learning for remote sensing data analysis, review the recent advances, and provide resources to make deep learning in remote sensing ridiculously simple to start with. More importantly, we advocate remote sensing scientists to bring their expertise into deep learning, and use it as an implicit general model to tackle unprecedented large-scale influential challenges, such as climate change and urbanization.Comment: Accepted for publication IEEE Geoscience and Remote Sensing Magazin

A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community

In recent years, deep learning (DL), a re-branding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, natural language processing, etc. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV; e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should be aware of, if not at the leading edge of, of advancements like DL. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as it relates to (i) inadequate data sets, (ii) human-understandable solutions for modelling physical phenomena, (iii) Big Data, (iv) non-traditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.Comment: 64 pages, 411 references. To appear in Journal of Applied Remote Sensin

FPGA-based operational concept and payload data processing for the Flying Laptop satellite

Flying Laptop is the first small satellite developed by the Institute of Space Systems at the Universität Stuttgart. It is a test bed for an on-board computer with a reconfigurable, redundant and self-controlling high computational ability based on the field pro- grammable gate arrays (FPGAs). This Technical Note presents the operational concept and the on-board payload data processing of the satellite. The designed operational concept of Flying Laptop enables the achievement of mission goals such as technical demonstration, scientific Earth observation, and the payload data processing methods. All these capabilities expand its scientific usage and enable new possibilities for real-time applications. Its hierarchical architecture of the operational modes of subsys- tems and modules are developed in a state-machine diagram and tested by means of MathWorks Simulink-/Stateflow Toolbox. Furthermore, the concept of the on-board payload data processing and its implementation and possible applications are described

PERSIANN-CNN: Precipitation Estimation from Remotely Sensed Information Using Artificial Neural Networks-Convolutional Neural Networks

Abstract Accurate and timely precipitation estimates are critical for monitoring and forecasting natural disasters such as floods. Despite having high-resolution satellite information, precipitation estimation from remotely sensed data still suffers from methodological limitations. State-of-the-art deep learning algorithms, renowned for their skill in learning accurate patterns within large and complex datasets, appear well suited to the task of precipitation estimation, given the ample amount of high-resolution satellite data. In this study, the effectiveness of applying convolutional neural networks (CNNs) together with the infrared (IR) and water vapor (WV) channels from geostationary satellites for estimating precipitation rate is explored. The proposed model performances are evaluated during summer 2012 and 2013 over central CONUS at the spatial resolution of 0.08° and at an hourly time scale. Precipitation Estimation from Remotely Sensed Information Using Artificial Neural Networks (PERSIANN)–Cloud Classification System (CCS), which is an operational satellite-based product, and PERSIANN–Stacked Denoising Autoencoder (PERSIANN-SDAE) are employed as baseline models. Results demonstrate that the proposed model (PERSIANN-CNN) provides more accurate rainfall estimates compared to the baseline models at various temporal and spatial scales. Specifically, PERSIANN-CNN outperforms PERSIANN-CCS (and PERSIANN-SDAE) by 54% (and 23%) in the critical success index (CSI), demonstrating the detection skills of the model. Furthermore, the root-mean-square error (RMSE) of the rainfall estimates with respect to the National Centers for Environmental Prediction (NCEP) Stage IV gauge–radar data, for PERSIANN-CNN was lower than that of PERSIANN-CCS (PERSIANN-SDAE) by 37% (14%), showing the estimation accuracy of the proposed model

Solar Power Plant Detection on Multi-Spectral Satellite Imagery using Weakly-Supervised CNN with Feedback Features and m-PCNN Fusion

Most of the traditional convolutional neural networks (CNNs) implements bottom-up approach (feed-forward) for image classifications. However, many scientific studies demonstrate that visual perception in primates rely on both bottom-up and top-down connections. Therefore, in this work, we propose a CNN network with feedback structure for Solar power plant detection on middle-resolution satellite images. To express the strength of the top-down connections, we introduce feedback CNN network (FB-Net) to a baseline CNN model used for solar power plant classification on multi-spectral satellite data. Moreover, we introduce a method to improve class activation mapping (CAM) to our FB-Net, which takes advantage of multi-channel pulse coupled neural network (m-PCNN) for weakly-supervised localization of the solar power plants from the features of proposed FB-Net. For the proposed FB-Net CAM with m-PCNN, experimental results demonstrated promising results on both solar-power plant image classification and detection task.Comment: 9 pages, 9 figures, 4 table

Satellite-based precipitation estimation using watershed segmentation and growing hierarchical self-organizing map

This paper outlines the development of a multi-satellite precipitation estimation methodology that draws on techniques from machine learning and morphology to produce high-resolution, short-duration rainfall estimates in an automated fashion. First, cloud systems are identified from geostationary infrared imagery using morphology based watershed segmentation algorithm. Second, a novel pattern recognition technique, growing hierarchical self-organizing map (GHSOM), is used to classify clouds into a number of clusters with hierarchical architecture. Finally, each cloud cluster is associated with co-registered passive microwave rainfall observations through a cumulative histogram matching approach. The network was initially trained using remotely sensed geostationary infrared satellite imagery and hourly ground-radar data in lieu of a dense constellation of polar-orbiting spacecraft such as the proposed global precipitation measurement (GPM) mission. Ground-radar and gauge rainfall measurements were used to evaluate this technique for both warm (June 2004) and cold seasons (December 2004-February 2005) at various temporal (daily and monthly) and spatial (0.04 and 0.25) scales. Significant improvements of estimation accuracy are found classifying the clouds into hierarchical sub-layers rather than a single layer. Furthermore, 2-year (2003-2004) satellite rainfall estimates generated by the current algorithm were compared with gauge-corrected Stage IV radar rainfall at various time scales over continental United States. This study demonstrates the usefulness of the watershed segmentation and the GHSOM in satellite-based rainfall estimations

Building Footprint Generation Using Improved Generative Adversarial Networks

Building footprint information is an essential ingredient for 3-D reconstruction of urban models. The automatic generation of building footprints from satellite images presents a considerable challenge due to the complexity of building shapes. In this work, we have proposed improved generative adversarial networks (GANs) for the automatic generation of building footprints from satellite images. We used a conditional GAN with a cost function derived from the Wasserstein distance and added a gradient penalty term. The achieved results indicated that the proposed method can significantly improve the quality of building footprint generation compared to conditional generative adversarial networks, the U-Net, and other networks. In addition, our method nearly removes all hyperparameters tuning.Comment: 5 page