A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community

In recent years, deep learning (DL), a re-branding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, natural language processing, etc. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV; e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should be aware of, if not at the leading edge of, of advancements like DL. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as it relates to (i) inadequate data sets, (ii) human-understandable solutions for modelling physical phenomena, (iii) Big Data, (iv) non-traditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.Comment: 64 pages, 411 references. To appear in Journal of Applied Remote Sensin

Theory and Design of a Highly Compressed Dropped-Channel Polarimetric Synthetic Aperture Radar

Compressed sensing (CS) is a recent mathematical technique that leverages the sparsity in certain sets of data to solve an underdetermined system and recover a full set of data from a sub-Nyquist set of measurements of the data. Given the size and sparsity of the data, radar has been a natural choice to apply compressed sensing to, typically in the fast-time and slow-time domains. Polarimetric synthetic aperture radar (PolSAR) generates a particularly large amount of data for a given scene; however, the data tends to be sparse. Recently a technique was developed to recover a dropped PolSAR channel by leveraging antenna crosstalk information and using compressed sensing. In this dissertation, we build upon the initial concept of the dropped-channel PolSAR CS in three ways. First, we determine a metric which relates the measurement matrix to the l2 recovery error. The new metric is necessary given the deterministic nature of the measurement matrix. We then determine a range of antenna crosstalk required to recover a dropped PolSAR channel. Second, we propose a new antenna design that incorporates the relatively high levels of crosstalk required by a dropped-channel PolSAR system. Finally, we integrate fast- and slow-time compression schemes into the dropped-channel model in order to leverage sparsity in additional PolSAR domains and overall increase the compression ratio. The completion of these research tasks has allowed a more accurate description of a PolSAR system that compresses in fast-time, slow-time, and polarization; termed herein as highly compressed PolSAR. The description of a highly compressed PolSAR system is a big step towards the development of prototype hardware in the future

Cloud Removal in Sentinel-2 Imagery using a Deep Residual Neural Network and SAR-Optical Data Fusion

Optical remote sensing imagery is at the core of many Earth observation activities. The regular, consistent and global-scale nature of the satellite data is exploited in many applications, such as cropland monitoring, climate change assessment, land-cover and land-use classification, and disaster assessment. However, one main problem severely affects the temporal and spatial availability of surface observations, namely cloud cover. The task of removing clouds from optical images has been subject of studies since decades. The advent of the Big Data era in satellite remote sensing opens new possibilities for tackling the problem using powerful data-driven deep learning methods. In this paper, a deep residual neural network architecture is designed to remove clouds from multispectral Sentinel-2 imagery. SAR-optical data fusion is used to exploit the synergistic properties of the two imaging systems to guide the image reconstruction. Additionally, a novel cloud-adaptive loss is proposed to maximize the retainment of original information. The network is trained and tested on a globally sampled dataset comprising real cloudy and cloud-free images. The proposed setup allows to remove even optically thick clouds by reconstructing an optical representation of the underlying land surface structure

Patch-based Denoising Algorithms for Single and Multi-view Images

In general, all single and multi-view digital images are captured using sensors, where they are often contaminated with noise, which is an undesired random signal. Such noise can also be produced during transmission or by lossy image compression. Reducing the noise and enhancing those images is among the fundamental digital image processing tasks. Improving the performance of image denoising methods, would greatly contribute to single or multi-view image processing techniques, e.g. segmentation, computing disparity maps, etc. Patch-based denoising methods have recently emerged as the state-of-the-art denoising approaches for various additive noise levels. This thesis proposes two patch-based denoising methods for single and multi-view images, respectively. A modification to the block matching 3D algorithm is proposed for single image denoising. An adaptive collaborative thresholding filter is proposed which consists of a classification map and a set of various thresholding levels and operators. These are exploited when the collaborative hard-thresholding step is applied. Moreover, the collaborative Wiener filtering is improved by assigning greater weight when dealing with similar patches. For the denoising of multi-view images, this thesis proposes algorithms that takes a pair of noisy images captured from two different directions at the same time (stereoscopic images). The structural, maximum difference or the singular value decomposition-based similarity metrics is utilized for identifying locations of similar search windows in the input images. The non-local means algorithm is adapted for filtering these noisy multi-view images. The performance of both methods have been evaluated both quantitatively and qualitatively through a number of experiments using the peak signal-to-noise ratio and the mean structural similarity measure. Experimental results show that the proposed algorithm for single image denoising outperforms the original block matching 3D algorithm at various noise levels. Moreover, the proposed algorithm for multi-view image denoising can effectively reduce noise and assist to estimate more accurate disparity maps at various noise levels

Radar satellite imagery for humanitarian response. Bridging the gap between technology and application

This work deals with radar satellite imagery and its potential to assist of humanitarian operations. As the number of displaced people annually increases, both hosting countries and relief organizations face new challenges which are often related to unclear situations and lack of information on the number and location of people in need, as well as their environments. It was demonstrated in numerous studies that methods of earth observation can deliver this important information for the management of crises, the organization of refugee camps, and the mapping of environmental resources and natural hazards. However, most of these studies make use of -high-resolution optical imagery, while the role of radar satellites is widely neglected. At the same time, radar sensors have characteristics which make them highly suitable for humanitarian response, their potential to capture images through cloud cover and at night in the first place. Consequently, they potentially allow quicker response in cases of emergencies than optical imagery. This work demonstrates the currently unused potential of radar imagery for the assistance of humanitarian operations by case studies which cover the information needs of specific emergency situations. They are thematically grouped into topics related to population, natural hazards and the environment. Furthermore, the case studies address different levels of scientific objectives: The main intention is the development of innovative techniques of digital image processing and geospatial analysis as an answer on the identified existing research gaps. For this reason, novel approaches are presented on the mapping of refugee camps and urban areas, the allocation of biomass and environmental impact assessment. Secondly, existing methods developed for radar imagery are applied, refined, or adapted to specifically demonstrate their benefit in a humanitarian context. This is done for the monitoring of camp growth, the assessment of damages in cities affected by civil war, and the derivation of areas vulnerable to flooding or sea-surface changes. Lastly, to foster the integration of radar images into existing operational workflows of humanitarian data analysis, technically simple and easily-adaptable approaches are suggested for the mapping of rural areas for vaccination campaigns, the identification of changes within and around refugee camps, and the assessment of suitable locations for groundwater drillings. While the studies provide different levels of technical complexity and novelty, they all show that radar imagery can largely contribute to the provision of a variety of information which is required to make solid decisions and to effectively provide help in humanitarian operations. This work furthermore demonstrates that radar images are more than just an alternative image source for areas heavily affected by cloud cover. In fact, what makes them valuable is their information content regarding the characteristics of surfaces, such as shape, orientation, roughness, size, height, moisture, or conductivity. All these give decisive insights about man-made and natural environments in emergency situations and cannot be provided by optical images Finally, the findings of the case studies are put into a larger context, discussing the observed potential and limitations of the presented approaches. The major challenges are summarized which need be addressed to make radar imagery more useful in humanitarian operations in the context of upcoming technical developments. New radar satellites and technological progress in the fields of machine learning and cloud computing will bring new opportunities. At the same time, this work demonstrated the large need for further research, as well as for the collaboration and transfer of knowledge and experiences between scientists, users and relief workers in the field. It is the first extensive scientific compilation of this topic and the first step for a sustainable integration of radar imagery into operational frameworks to assist humanitarian work and to contribute to a more efficient provision of help to those in need.Die vorliegende Arbeit beschäftigt sich mit bildgebenden Radarsatelliten und ihrem potenziellen Beitrag zur Unterstützung humanitärer Einsätze. Die jährlich zunehmende Zahl an vertriebenen oder geflüchteten Menschen stellt sowohl Aufnahmeländer als auch humanitäre Organisationen vor große Herausforderungen, da sie oft mit unübersichtlichen Verhältnissen konfrontiert sind. Effektives Krisenmanagement, die Planung und Versorgung von Flüchtlingslagern, sowie der Schutz der betroffenen Menschen erfordern jedoch verlässliche Angaben über Anzahl und Aufenthaltsort der Geflüchteten und ihrer natürlichen Umwelt. Die Bereitstellung dieser Informationen durch Satellitenbilder wurde bereits in zahlreichen Studien aufgezeigt. Sie beruhen in der Regel auf hochaufgelösten optischen Aufnahmen, während bildgebende Radarsatelliten bisher kaum Anwendung finden. Dabei verfügen gerade Radarsatelliten über Eigenschaften, die hilfreich für humanitäre Einsätze sein können, allen voran ihre Unabhängigkeit von Bewölkung oder Tageslicht. Dadurch ermöglichen sie in Krisenfällen verglichen mit optischen Satelliten eine schnellere Reaktion. Diese Arbeit zeigt das derzeit noch ungenutzte Potenzial von Radardaten zur Unterstützung humanitärer Arbeit anhand von Fallstudien auf, in denen konkrete Informationen für ausgewählte Krisensituationen bereitgestellt werden. Sie sind in die Themenbereiche Bevölkerung, Naturgefahren und Ressourcen aufgeteilt, adressieren jedoch unterschiedliche wissenschaftliche Ansprüche: Der Hauptfokus der Arbeit liegt auf der Entwicklung von innovativen Methoden zur Verarbeitung von Radarbildern und räumlichen Daten als Antwort auf den identifizierten Forschungsbedarf in diesem Gebiet. Dies wird anhand der Kartierung von Flüchtlingslagern zur Abschätzung ihrer Bevölkerung, zur Bestimmung von Biomasse, sowie zur Ermittlung des Umwelteinflusses von Flüchtlingslagern aufgezeigt. Darüber hinaus werden existierende oder erprobte Ansätze für die Anwendung im humanitären Kontext angepasst oder weiterentwickelt. Dies erfolgt im Rahmen von Fallstudien zur Dynamik von Flüchtlingslagern, zur Ermittlung von Schäden an Gebäuden in Kriegsgebieten, sowie zur Erkennung von Risiken durch Überflutung. Zuletzt soll die Integration von Radardaten in bereits existierende Abläufe oder Arbeitsroutinen in der humanitären Hilfe anhand technisch vergleichsweise einfacher Ansätze vorgestellt und angeregt werden. Als Beispiele dienen hier die radargestützte Kartierung von entlegenen Gebieten zur Unterstützung von Impfkampagnen, die Identifizierung von Veränderungen in Flüchtlingslagern, sowie die Auswahl geeigneter Standorte zur Grundwasserentnahme. Obwohl sich die Fallstudien hinsichtlich ihres Innovations- und Komplexitätsgrads unterscheiden, zeigen sie alle den Mehrwert von Radardaten für die Bereitstellung von Informationen, um schnelle und fundierte Planungsentscheidungen zu unterstützen. Darüber hinaus wird in dieser Arbeit deutlich, dass Radardaten für humanitäre Zwecke mehr als nur eine Alternative in stark bewölkten Gebieten sind. Durch ihren Informationsgehalt zur Beschaffenheit von Oberflächen, beispielsweise hinsichtlich ihrer Rauigkeit, Feuchte, Form, Größe oder Höhe, sind sie optischen Daten überlegen und daher für viele Anwendungsbereiche im Kontext humanitärer Arbeit besonders. Die in den Fallstudien gewonnenen Erkenntnisse werden abschließend vor dem Hintergrund von Vor- und Nachteilen von Radardaten, sowie hinsichtlich zukünftiger Entwicklungen und Herausforderungen diskutiert. So versprechen neue Radarsatelliten und technologische Fortschritte im Bereich der Datenverarbeitung großes Potenzial. Gleichzeitig unterstreicht die Arbeit einen großen Bedarf an weiterer Forschung, sowie an Austausch und Zusammenarbeit zwischen Wissenschaftlern, Anwendern und Einsatzkräften vor Ort. Die vorliegende Arbeit ist die erste umfassende Darstellung und wissenschaftliche Aufarbeitung dieses Themenkomplexes. Sie soll als Grundstein für eine langfristige Integration von Radardaten in operationelle Abläufe dienen, um humanitäre Arbeit zu unterstützen und eine wirksame Hilfe für Menschen in Not ermöglichen

The SAR Handbook: Comprehensive Methodologies for Forest Monitoring and Biomass Estimation

This Synthetic Aperture Radar (SAR) handbook of applied methods for forest monitoring and biomass estimation has been developed by SERVIR in collaboration with SilvaCarbon to address pressing needs in the development of operational forest monitoring services. Despite the existence of SAR technology with all-weather capability for over 30 years, the applied use of this technology for operational purposes has proven difficult. This handbook seeks to provide understandable, easy-to-assimilate technical material to remote sensing specialists that may not have expertise on SAR but are interested in leveraging SAR technology in the forestry sector