A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community

In recent years, deep learning (DL), a re-branding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, natural language processing, etc. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV; e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should be aware of, if not at the leading edge of, of advancements like DL. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as it relates to (i) inadequate data sets, (ii) human-understandable solutions for modelling physical phenomena, (iii) Big Data, (iv) non-traditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.Comment: 64 pages, 411 references. To appear in Journal of Applied Remote Sensin

Large-Scale Urban Impervous Surfaces Estimation Through Incorporating Temporal and Spatial Information into Spectral Mixture Analysis

With rapid urbanization, impervious surfaces, a major component of urbanized areas, have increased concurrently. As a key indicator of environmental quality and urbanization intensity, an accurate estimation of impervious surfaces becomes essential. Numerous automated estimation approaches have been developed during the past decades. Among them, spectral mixture analysis (SMA) has been recognized as a powerful and widely employed technique. While SMA has proven valuable in impervious surface estimation, effects of temporal and spectral variability have not been successfully addressed. In particular, impervious surface estimation is likely to be sensitive to seasonal changes, majorly due to the shadowing effects of vegetation canopy in summer and the confusion between impervious surfaces and soil in winter. Moreover, endmember variability and multi-collinearity have adversely impacted the accurate estimation of impervious surface distribution with coarse resolution remote sensing imagery. Therefore, the main goal of this research is to incorporate temporal and spatial information, as well as geostatistical approaches, into SMA for improving large-scale urban impervious surface estimation. Specifically, three new approaches have been developed in this dissertation to improve the accuracy of large-scale impervious surface estimation. First, a phenology based temporal mixture analysis was developed to address seasonal sensitivity and spectral confusion issues with the multi-temporal MODIS NDVI data. Second, land use land cover information assisted temporal mixture analysis was proposed to handle the issue of endmember class variability through analyzing the spatial relationship between endmembers and surrounding environmental and socio-economic factors in support of the selection of an appropriate number and types of endmember classes. Third, a geostatistical temporal mixture analysis was developed to address endmember spectral variability by generating per-pixel spatial varied endmember spectra. Analysis results suggest that, first, with the proposed phenology based temporal mixture analysis, a significant phenophase differences between impervious surfaces and soil can be extracted and employed in unmxing analysis, which can facilitate their discrimination and successfully address the issue of seasonal sensitivity and spectral confusion. Second, with the analyzed spatial distribution relationship between endmembers and environmental and socio-economic factors, endmember classes can be identified with clear physical meanings throughout the whole study area, which can effectively improve the unmixing analysis results. Third, the use of the spatially varying per-pixel endmember generated from the geostatistical approach can effectively consider the endmember spectra spatial variability, overcome the endmember within-class variability issue, and improve the accuracy of impervious surface estimates. Major contributions of this research can be summarized as follows. First, instead of Landsat Thematic Mapper (TM) images, MODIS imageries with large geographic coverage and high temporal resolution have been successfully employed in this research, thus making timely and regional estimation of impervious surfaces possible. Second, this research proves that the incorporation of geographic knowledge (e.g. phonological knowledge, spatial interaction, and geostatistics) can effectively improve the spectral mixture analysis model, and therefore improve the estimation accuracy of urban impervious surfaces

Mixture of Latent Variable Models for Remotely Sensed Image Processing

The processing of remotely sensed data is innately an inverse problem where properties of spatial processes are inferred from the observations based on a generative model. Meaningful data inversion relies on well-defined generative models that capture key factors in the relationship between the underlying physical process and the measurements. Unfortunately, as two mainstream data processing techniques, both mixture models and latent variables models (LVM) are inadequate in describing the complex relationship between the spatial process and the remote sensing data. Consequently, mixture models, such as K-Means, Gaussian Mixture Model (GMM), Linear Discriminant Analysis (LDA) and Quadratic Discriminant Analysis (QDA), characterize a class by statistics in the original space, ignoring the fact that a class can be better represented by discriminative signals in the hidden/latent feature space, while LVMs, such as Principal Component Analysis (PCA), Independent Component Analysis (ICA) and Sparse Representation (SR), seek representational signals in the whole image scene that involves multiple spatial processes, neglecting the fact that signal discovery for individual processes is more efficient. Although the combined use of mixture model and LVMs is required for remote sensing data analysis, there is still a lack of systematic exploration on this important topic in remote sensing literature. Driven by the above considerations, this thesis therefore introduces a mixture of LVM (MLVM) framework for combining the mixture models and LVMs, under which three models are developed in order to address different aspects of remote sensing data processing: (1) a mixture of probabilistic SR (MPSR) is proposed for supervised classification of hyperspectral remote sensing imagery, considering that SR is an emerging and powerful technique for feature extraction and data representation; (2) a mixture model of K “Purified” means (K-P-Means) is proposed for addressing the spectral endmember estimation, which is a fundamental issue in remote sensing data analysis; (3) and a clustering-based PCA model is introduced for SAR image denoising. Under a unified optimization scheme, all models are solved via Expectation and Maximization (EM) algorithm, by iteratively estimating the two groups of parameters, i.e., the labels of pixels and the latent variables. Experiments on simulated data and real remote sensing data demonstrate the advantages of the proposed models in the respective applications

Mineral identification using data-mining in hyperspectral infrared imagery

Les applications de l’imagerie infrarouge dans le domaine de la géologie sont principalement des applications hyperspectrales. Elles permettent entre autre l’identification minérale, la cartographie, ainsi que l’estimation de la portée. Le plus souvent, ces acquisitions sont réalisées in-situ soit à l’aide de capteurs aéroportés, soit à l’aide de dispositifs portatifs. La découverte de minéraux indicateurs a permis d’améliorer grandement l’exploration minérale. Ceci est en partie dû à l’utilisation d’instruments portatifs. Dans ce contexte le développement de systèmes automatisés permettrait d’augmenter à la fois la qualité de l’exploration et la précision de la détection des indicateurs. C’est dans ce cadre que s’inscrit le travail mené dans ce doctorat. Le sujet consistait en l’utilisation de méthodes d’apprentissage automatique appliquées à l’analyse (au traitement) d’images hyperspectrales prises dans les longueurs d’onde infrarouge. L’objectif recherché étant l’identification de grains minéraux de petites tailles utilisés comme indicateurs minéral -ogiques. Une application potentielle de cette recherche serait le développement d’un outil logiciel d’assistance pour l’analyse des échantillons lors de l’exploration minérale. Les expériences ont été menées en laboratoire dans la gamme relative à l’infrarouge thermique (Long Wave InfraRed, LWIR) de 7.7m à 11.8 m. Ces essais ont permis de proposer une méthode pour calculer l’annulation du continuum. La méthode utilisée lors de ces essais utilise la factorisation matricielle non négative (NMF). En utlisant une factorisation du premier ordre on peut déduire le rayonnement de pénétration, lequel peut ensuite être comparé et analysé par rapport à d’autres méthodes plus communes. L’analyse des résultats spectraux en comparaison avec plusieurs bibliothèques existantes de données a permis de mettre en évidence la suppression du continuum. Les expérience ayant menés à ce résultat ont été conduites en utilisant une plaque Infragold ainsi qu’un objectif macro LWIR. L’identification automatique de grains de différents matériaux tels que la pyrope, l’olivine et le quartz a commencé. Lors d’une phase de comparaison entre des approches supervisées et non supervisées, cette dernière s’est montrée plus approprié en raison du comportement indépendant par rapport à l’étape d’entraînement. Afin de confirmer la qualité de ces résultats quatre expériences ont été menées. Lors d’une première expérience deux algorithmes ont été évalués pour application de regroupements en utilisant l’approche FCC (False Colour Composite). Cet essai a permis d’observer une vitesse de convergence, jusqu’a vingt fois plus rapide, ainsi qu’une efficacité significativement accrue concernant l’identification en comparaison des résultats de la littérature. Cependant des essais effectués sur des données LWIR ont montré un manque de prédiction de la surface du grain lorsque les grains étaient irréguliers avec présence d’agrégats minéraux. La seconde expérience a consisté, en une analyse quantitaive comparative entre deux bases de données de Ground Truth (GT), nommée rigid-GT et observed-GT (rigide-GT: étiquet manuel de la région, observée-GT:étiquetage manuel les pixels). La précision des résultats était 1.5 fois meilleur lorsque l’on a utlisé la base de données observed-GT que rigid-GT. Pour les deux dernières epxérience, des données venant d’un MEB (Microscope Électronique à Balayage) ainsi que d’un microscopie à fluorescence (XRF) ont été ajoutées. Ces données ont permis d’introduire des informations relatives tant aux agrégats minéraux qu’à la surface des grains. Les résultats ont été comparés par des techniques d’identification automatique des minéraux, utilisant ArcGIS. Cette dernière a montré une performance prometteuse quand à l’identification automatique et à aussi été utilisée pour la GT de validation. Dans l’ensemble, les quatre méthodes de cette thèse représentent des méthodologies bénéfiques pour l’identification des minéraux. Ces méthodes présentent l’avantage d’être non-destructives, relativement précises et d’avoir un faible coût en temps calcul ce qui pourrait les qualifier pour être utilisée dans des conditions de laboratoire ou sur le terrain.The geological applications of hyperspectral infrared imagery mainly consist in mineral identification, mapping, airborne or portable instruments, and core logging. Finding the mineral indicators offer considerable benefits in terms of mineralogy and mineral exploration which usually involves application of portable instrument and core logging. Moreover, faster and more mechanized systems development increases the precision of identifying mineral indicators and avoid any possible mis-classification. Therefore, the objective of this thesis was to create a tool to using hyperspectral infrared imagery and process the data through image analysis and machine learning methods to identify small size mineral grains used as mineral indicators. This system would be applied for different circumstances to provide an assistant for geological analysis and mineralogy exploration. The experiments were conducted in laboratory conditions in the long-wave infrared (7.7μm to 11.8μm - LWIR), with a LWIR-macro lens (to improve spatial resolution), an Infragold plate, and a heating source. The process began with a method to calculate the continuum removal. The approach is the application of Non-negative Matrix Factorization (NMF) to extract Rank-1 NMF and estimate the down-welling radiance and then compare it with other conventional methods. The results indicate successful suppression of the continuum from the spectra and enable the spectra to be compared with spectral libraries. Afterwards, to have an automated system, supervised and unsupervised approaches have been tested for identification of pyrope, olivine and quartz grains. The results indicated that the unsupervised approach was more suitable due to independent behavior against training stage. Once these results obtained, two algorithms were tested to create False Color Composites (FCC) applying a clustering approach. The results of this comparison indicate significant computational efficiency (more than 20 times faster) and promising performance for mineral identification. Finally, the reliability of the automated LWIR hyperspectral infrared mineral identification has been tested and the difficulty for identification of the irregular grain’s surface along with the mineral aggregates has been verified. The results were compared to two different Ground Truth(GT) (i.e. rigid-GT and observed-GT) for quantitative calculation. Observed-GT increased the accuracy up to 1.5 times than rigid-GT. The samples were also examined by Micro X-ray Fluorescence (XRF) and Scanning Electron Microscope (SEM) in order to retrieve information for the mineral aggregates and the grain’s surface (biotite, epidote, goethite, diopside, smithsonite, tourmaline, kyanite, scheelite, pyrope, olivine, and quartz). The results of XRF imagery compared with automatic mineral identification techniques, using ArcGIS, and represented a promising performance for automatic identification and have been used for GT validation. In overall, the four methods (i.e. 1.Continuum removal methods; 2. Classification or clustering methods for mineral identification; 3. Two algorithms for clustering of mineral spectra; 4. Reliability verification) in this thesis represent beneficial methodologies to identify minerals. These methods have the advantages to be a non-destructive, relatively accurate and have low computational complexity that might be used to identify and assess mineral grains in the laboratory conditions or in the field

Assessing skin lesion evolution from multispectral image sequences

During the evaluation of skin disease treatments, dermatologists have to clinically measure the evolution of the pathology severity of each patient during treatment periods. Such a process is sensitive to intra- and inter- dermatologist diagnosis. To make this severity measurement more objective we quantify the pathology severity using a new image processing based method. We focus on a hyperpigmentation disorder called melasma. During a treatment period, multispectral images are taken on patients receiving the same treatment. After co-registration and segmentation steps, we propose an algorithm to measure the intensity, the size and the homogeneity evolution of the pathological areas. Obtained results are compared with a dermatologist diagnosis using statistical tests on two clinical studies containing respectively 384 images from 16 patients and 352 images from 22 patients.This research report is an update of the report 8136. It describes methods and experiments in more details and provides more references.Lors de l'évaluation des traitements des maladies de peau, les dermatologues doivent mesurer la sévérité de la pathologie de chaque patient tout au long d'une période de traitement. Un tel procédé est sensible aux variations intra- et inter- dermatologues. Pour rendrecette mesure de sévérité plus robuste, nous proposons d'utiliser l'imagerie spectrale. Nous nous concentrons sur une pathologie d'hyperpigmentation cutanée appelée mélasma. Au cours d'une période de traitement, des images multispectrales sont acquises sur une population de patients sous traitement. Après des étapes de recalage des séries temporelles d'images et de classification des régions d'intérêt, nous proposons une méthodologie permettant de mesurer, dans le temps, la variation de contraste, de surface et d'homogénéité de la zone pathologique pour chaque patient. Les résultats obtenus sont comparés à un diagnostique clinique à l'aide de tests statistiques réalisés sur une étude clinique complète.Ce rapport de recherche est un complément du rapport de recherche 8136, afin de compléter la bibliographie, et de décrire plus en détail les méthodes et résultat

Summaries of the Sixth Annual JPL Airborne Earth Science Workshop

This publication contains the summaries for the Sixth Annual JPL Airborne Earth Science Workshop, held in Pasadena, California, on March 4-8, 1996. The main workshop is divided into two smaller workshops as follows: (1) The Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) workshop, on March 4-6. The summaries for this workshop appear in Volume 1; (2) The Airborne Synthetic Aperture Radar (AIRSAR) workshop, on March 6-8. The summaries for this workshop appear in Volume 2

Development of an unsupervised remote sensing methodology of detect surface leakage from terrestrial CO2 storage sites

Imperial Users onl

Semi-supervised and unsupervised kernel-based novelty detection with application to remote sensing images

The main challenge of new information technologies is to retrieve intelligible information from the large volume of digital data gathered every day. Among the variety of existing data sources, the satellites continuously observing the surface of the Earth are key to the monitoring of our environment. The new generation of satellite sensors are tremendously increasing the possibilities of applications but also increasing the need for efficient processing methodologies in order to extract information relevant to the users' needs in an automatic or semi-automatic way. This is where machine learning comes into play to transform complex data into simplified products such as maps of land-cover changes or classes by learning from data examples annotated by experts. These annotations, also called labels, may actually be difficult or costly to obtain since they are established on the basis of ground surveys. As an example, it is extremely difficult to access a region recently flooded or affected by wildfires. In these situations, the detection of changes has to be done with only annotations from unaffected regions. In a similar way, it is difficult to have information on all the land-cover classes present in an image while being interested in the detection of a single one of interest. These challenging situations are called novelty detection or one-class classification in machine learning. In these situations, the learning phase has to rely only on a very limited set of annotations, but can exploit the large set of unlabeled pixels available in the images. This setting, called semi-supervised learning, allows significantly improving the detection. In this Thesis we address the development of methods for novelty detection and one-class classification with few or no labeled information. The proposed methodologies build upon the kernel methods, which take place within a principled but flexible framework for learning with data showing potentially non-linear feature relations. The thesis is divided into two parts, each one having a different assumption on the data structure and both addressing unsupervised (automatic) and semi-supervised (semi-automatic) learning settings. The first part assumes the data to be formed by arbitrary-shaped and overlapping clusters and studies the use of kernel machines, such as Support Vector Machines or Gaussian Processes. An emphasis is put on the robustness to noise and outliers and on the automatic retrieval of parameters. Experiments on multi-temporal multispectral images for change detection are carried out using only information from unchanged regions or none at all. The second part assumes high-dimensional data to lie on multiple low dimensional structures, called manifolds. We propose a method seeking a sparse and low-rank representation of the data mapped in a non-linear feature space. This representation allows us to build a graph, which is cut into several groups using spectral clustering. For the semi-supervised case where few labels of one class of interest are available, we study several approaches incorporating the graph information. The class labels can either be propagated on the graph, constrain spectral clustering or used to train a one-class classifier regularized by the given graph. Experiments on the unsupervised and oneclass classification of hyperspectral images demonstrate the effectiveness of the proposed approaches

Very High Resolution (VHR) Satellite Imagery: Processing and Applications

Recently, growing interest in the use of remote sensing imagery has appeared to provide synoptic maps of water quality parameters in coastal and inner water ecosystems;, monitoring of complex land ecosystems for biodiversity conservation; precision agriculture for the management of soils, crops, and pests; urban planning; disaster monitoring, etc. However, for these maps to achieve their full potential, it is important to engage in periodic monitoring and analysis of multi-temporal changes. In this context, very high resolution (VHR) satellite-based optical, infrared, and radar imaging instruments provide reliable information to implement spatially-based conservation actions. Moreover, they enable observations of parameters of our environment at greater broader spatial and finer temporal scales than those allowed through field observation alone. In this sense, recent very high resolution satellite technologies and image processing algorithms present the opportunity to develop quantitative techniques that have the potential to improve upon traditional techniques in terms of cost, mapping fidelity, and objectivity. Typical applications include multi-temporal classification, recognition and tracking of specific patterns, multisensor data fusion, analysis of land/marine ecosystem processes and environment monitoring, etc. This book aims to collect new developments, methodologies, and applications of very high resolution satellite data for remote sensing. The works selected provide to the research community the most recent advances on all aspects of VHR satellite remote sensing

Optimizing Hyperspectral Image Processing with GPUs and Accelerators

Trabajo de Fin de Grado en Ingeniería de Computadores, Facultad de Informática UCM, Departamento de Arquitectura de Computadores y Automática, Curso 2021/2022.Durante siglos se han creado diversas teorías sobre el planeta en el que vivimos y con el desarrollo de las tecnologías se han podido conocer muchas de sus características, se han ido mejorando las técnicas que se utilizan hasta dar con las que se encuentran ahora, donde se analiza a través de satélites obteniendo imágenes hiperespectrales para un análisis píxel a píxel de los materiales que contiene. El análisis de imágenes hiperespectrales es una tarea ardua, al captar el material de el que se compone la imagen a través de un solo píxel se dificulta cuando ese píxel tiene más de un material, llamado el problema de la mezcla espectral, por ello se hace un desmezclado espectral a través de una cadena de procesamiento. Dependiendo de la imagen, algoritmos seleccionados para la cadena de desmezclado espectral y el factor tecnológico puede dar diversos rendimientos. La cadena de desmezcaldo espectral tiene tres fases, la primera fase se trata de obtener el numero de materiales o endmembers que tiene la imagen hiperespectral, la segunda fase se extrae los diferentes materiales que componen la imagen hiperespectral y la tercera fase se saca un mapa de abundancia de cada material. Se han seleccionado en el mismo orden los algoritmos VD, VCA e ISRA para completar la cadena de desmezclado espectral. En este proyecto se han implementado todas las fases de forma paralela, contribuyendo a la optimización de estos algoritmos de procesado de imágenes hiperespectrales en distintos paradigmas de programación paralela, como son: OpenACC, OpenMP y SYCL (oneAPI). Se utilizan este tipo de paradigmas ya que otro de los objetivos de este trabajo es poder ejecutar todos los algoritmos en sistemas heterogéneos, con todos los resultados obtenidos se hace una comparativa de rendimiento buscando la mejor combinación entre estos.For centuries, various theories have been created about the planet we live in, and with the development of technologies, many of its characteristics have been discovered. The techniques used for this matter have been improved and perfected, making satellites able to take hyperspectral images for a pixel-by-pixel analysis of the materials they contain. The analysis of hyperspectral images is an arduous task, capturing the material from which an image is composed through a single pixel becomes difficult when that pixel has more than one material, this is known as the problem of spectral mixing. For this reason, spectral unmixing is done through a chain of processing. Depending on the image, the selected algorithms for the spectral unmixing chain and the technological factor can lead to different performance results. The spectral unmixing chain has three phases: the first phase obtains the number of materials (or endmembers) present in the hyperspectral image, the second phase extracts what different materials that make up the hyperspectral image, and the third phase generates an abundance map of each material in the hyperspectral image. The VD, VCA and ISRA algorithms have been selected in that order to create the spectral unmixing chain. In this project, all the phases have been implemented using parallel computing, contributing to the optimization of these hyperspectral image processing algorithms in different parallel programming paradigms, such as: OpenACC, OpenMP and SYCL (oneAPI). This type of paradigm is used since one of the objectives of this work is to be able to execute all the algorithms in heterogeneous systems. With all the results obtained, a performance comparison is made looking for the best combination between them.Depto. de Arquitectura de Computadores y AutomáticaFac. de InformáticaTRUEunpu