More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification

Classification and identification of the materials lying over or beneath the Earth's surface have long been a fundamental but challenging research topic in geoscience and remote sensing (RS) and have garnered a growing concern owing to the recent advancements of deep learning techniques. Although deep networks have been successfully applied in single-modality-dominated classification tasks, yet their performance inevitably meets the bottleneck in complex scenes that need to be finely classified, due to the limitation of information diversity. In this work, we provide a baseline solution to the aforementioned difficulty by developing a general multimodal deep learning (MDL) framework. In particular, we also investigate a special case of multi-modality learning (MML) -- cross-modality learning (CML) that exists widely in RS image classification applications. By focusing on "what", "where", and "how" to fuse, we show different fusion strategies as well as how to train deep networks and build the network architecture. Specifically, five fusion architectures are introduced and developed, further being unified in our MDL framework. More significantly, our framework is not only limited to pixel-wise classification tasks but also applicable to spatial information modeling with convolutional neural networks (CNNs). To validate the effectiveness and superiority of the MDL framework, extensive experiments related to the settings of MML and CML are conducted on two different multimodal RS datasets. Furthermore, the codes and datasets will be available at https://github.com/danfenghong/IEEE_TGRS_MDL-RS, contributing to the RS community

Anomalous change detection in multi-temporal hyperspectral images

In latest years, the possibility to exploit the high amount of spectral information has made hyperspectral remote sensing a very promising approach to detect changes occurred in multi-temporal images. Detection of changes in images of the same area collected at different times is of crucial interest in military and civilian applications, spanning from wide area surveillance and damage assessment to geology and land cover. In military operations, the interest is in rapid location and tracking of objects of interest, people, vehicles or equipment that pose a potential threat. In civilian contexts, changes of interest may include different types of natural or manmade threats, such as the path of an impending storm or the source of a hazardous material spill. In this PhD thesis, the focus is on Anomalous Change Detection (ACD) in airborne hyperspectral images. The goal is the detection of small changes occurred in two images of the same scene, i.e. changes having size comparable with the sensor ground resolution. The objects of interest typically occupy few pixels of the image and change detection must be accomplished in a pixel-wise fashion. Moreover, since the images are in general not radiometrically comparable, because illumination, atmospheric and environmental conditions change from one acquisition to the other, pervasive and uninteresting changes must be accounted for in developing ACD strategies. ACD process can be distinguished into two main phases: a pre-processing step, which includes radiometric correction, image co-registration and noise filtering, and a detection step, where the pre-processed images are compared according to a defined criterion in order to derive a statistical ACD map highlighting the anomalous changes occurred in the scene. In the literature, ACD has been widely investigated providing valuable methods in order to cope with these problems. In this work, a general overview of ACD methods is given reviewing the most known pre-processing and detection methods proposed in the literature. The analysis has been conducted unifying different techniques in a common framework based on binary decision theory, where one has to test the two competing hypotheses H0 (change absent) and H1 (change present) on the basis of an observation vector derived from the radiance measured on each pixel of the two images. Particular emphasis has been posed on statistical approaches, where ACD is derived in the framework of Neymann Pearson theory and the decision rule is carried out on the basis of the statistical properties assumed for the two hypotheses distribution, the observation vector space and the secondary data exploited for the estimation of the unknown parameters. Typically, ACD techniques assume that the observation represents the realization of jointly Gaussian spatially stationary random process. Though such assumption is adopted because of its mathematical tractability, it may be quite simplistic to model the multimodality usually met in real data. A more appropriate model is that adopted to derive the well known RX anomaly detector which assumes the local Gaussianity of the hyperspectral data. In this framework, a new statistical ACD method has been proposed considering the local Gaussianity of the hyperspectral data. The assumption of local stationarity for the observations in the two hypotheses is taken into account by considering two different models, leading to two different detectors. In addition, when data are collected by airborne platforms, perfect co-registration between images is very difficult to achieve. As a consequence, a residual misregistration (RMR) error should be taken into account in developing ACD techniques. Different techniques have been proposed to cope with the performance degradation problem due to the RMR, embedding the a priori knowledge on the statistical properties of the RMR in the change detection scheme. In this context, a new method has been proposed for the estimation of the first and second order statistics of the RMR. The technique is based on a sequential strategy that exploits the Scale Invariant Feature Transform (SIFT) algorithm cascaded with the Minimum Covariance Determinant algorithm. The proposed method adapts the SIFT procedure to hyperspectral images and improves the robustness of the outliers filtering by means of a highly robust estimator of multivariate location. Then, the attention has been focused on noise filtering techniques aimed at enforcing the consistency of the ACD process. To this purpose, a new method has been proposed to mitigate the negative effects due to random noise. In particular, this is achieved by means of a band selection technique aimed at discarding spectral channels whose useful signal content is low compared with the noise contribution. Band selection is performed on a per-pixel basis by exploiting the estimates of the noise variance accounting also for the presence of the signal dependent noise component. Finally, the effectiveness of the proposed techniques has been extensively evaluated by employing different real hyperspectral datasets containing anomalous changes collected in different acquisition conditions and on different scenarios, highlighting advantages and drawbacks of each method. In summary, the main issues related to ACD in multi-temporal hyperspectral images have been examined in this PhD thesis. With reference to the pre-processing step, two original contributions have been offered: i) an unsupervised technique for the estimation of the RMR noise affecting hyperspectral images, and ii) an adaptive approach for ACD which mitigates the negative effects due to random noise. As to the detection step, a survey of the existing techniques has been carried out, highlighting the major drawbacks and disadvantages, and a novel contribution has been offered by presenting a new statistical ACD method which considers the local Gaussianity of the hyperspectral data

Context-dependent fusion with application to landmine detection.

Traditional machine learning and pattern recognition systems use a feature descriptor to describe the sensor data and a particular classifier (also called expert or learner ) to determine the true class of a given pattern. However, for complex detection and classification problems, involving data with large intra-class variations and noisy inputs, no single source of information can provide a satisfactory solution. As a result, combination of multiple classifiers is playing an increasing role in solving these complex pattern recognition problems, and has proven to be viable alternative to using a single classifier. In this thesis we introduce a new Context-Dependent Fusion (CDF) approach, We use this method to fuse multiple algorithms which use different types of features and different classification methods on multiple sensor data. The proposed approach is motivated by the observation that there is no single algorithm that can consistently outperform all other algorithms. In fact, the relative performance of different algorithms can vary significantly depending on several factions such as extracted features, and characteristics of the target class. The CDF method is a local approach that adapts the fusion method to different regions of the feature space. The goal is to take advantages of the strengths of few algorithms in different regions of the feature space without being affected by the weaknesses of the other algorithms and also avoiding the loss of potentially valuable information provided by few weak classifiers by considering their output as well. The proposed fusion has three main interacting components. The first component, called Context Extraction, partitions the composite feature space into groups of similar signatures, or contexts. Then, the second component assigns an aggregation weight to each detector\u27s decision in each context based on its relative performance within the context. The third component combines the multiple decisions, using the learned weights, to make a final decision. For Context Extraction component, a novel algorithm that performs clustering and feature discrimination is used to cluster the composite feature space and identify the relevant features for each cluster. For the fusion component, six different methods were proposed and investigated. The proposed approached were applied to the problem of landmine detection. Detection and removal of landmines is a serious problem affecting civilians and soldiers worldwide. Several detection algorithms on landmine have been proposed. Extensive testing of these methods has shown that the relative performance of different detectors can vary significantly depending on the mine type, geographical site, soil and weather conditions, and burial depth, etc. Therefore, multi-algorithm, and multi-sensor fusion is a critical component in land mine detection. Results on large and diverse real data collections show that the proposed method can identify meaningful and coherent clusters and that different expert algorithms can be identified for the different contexts. Our experiments have also indicated that the context-dependent fusion outperforms all individual detectors and several global fusion methods

Mineral identification using data-mining in hyperspectral infrared imagery

Les applications de l’imagerie infrarouge dans le domaine de la géologie sont principalement des applications hyperspectrales. Elles permettent entre autre l’identification minérale, la cartographie, ainsi que l’estimation de la portée. Le plus souvent, ces acquisitions sont réalisées in-situ soit à l’aide de capteurs aéroportés, soit à l’aide de dispositifs portatifs. La découverte de minéraux indicateurs a permis d’améliorer grandement l’exploration minérale. Ceci est en partie dû à l’utilisation d’instruments portatifs. Dans ce contexte le développement de systèmes automatisés permettrait d’augmenter à la fois la qualité de l’exploration et la précision de la détection des indicateurs. C’est dans ce cadre que s’inscrit le travail mené dans ce doctorat. Le sujet consistait en l’utilisation de méthodes d’apprentissage automatique appliquées à l’analyse (au traitement) d’images hyperspectrales prises dans les longueurs d’onde infrarouge. L’objectif recherché étant l’identification de grains minéraux de petites tailles utilisés comme indicateurs minéral -ogiques. Une application potentielle de cette recherche serait le développement d’un outil logiciel d’assistance pour l’analyse des échantillons lors de l’exploration minérale. Les expériences ont été menées en laboratoire dans la gamme relative à l’infrarouge thermique (Long Wave InfraRed, LWIR) de 7.7m à 11.8 m. Ces essais ont permis de proposer une méthode pour calculer l’annulation du continuum. La méthode utilisée lors de ces essais utilise la factorisation matricielle non négative (NMF). En utlisant une factorisation du premier ordre on peut déduire le rayonnement de pénétration, lequel peut ensuite être comparé et analysé par rapport à d’autres méthodes plus communes. L’analyse des résultats spectraux en comparaison avec plusieurs bibliothèques existantes de données a permis de mettre en évidence la suppression du continuum. Les expérience ayant menés à ce résultat ont été conduites en utilisant une plaque Infragold ainsi qu’un objectif macro LWIR. L’identification automatique de grains de différents matériaux tels que la pyrope, l’olivine et le quartz a commencé. Lors d’une phase de comparaison entre des approches supervisées et non supervisées, cette dernière s’est montrée plus approprié en raison du comportement indépendant par rapport à l’étape d’entraînement. Afin de confirmer la qualité de ces résultats quatre expériences ont été menées. Lors d’une première expérience deux algorithmes ont été évalués pour application de regroupements en utilisant l’approche FCC (False Colour Composite). Cet essai a permis d’observer une vitesse de convergence, jusqu’a vingt fois plus rapide, ainsi qu’une efficacité significativement accrue concernant l’identification en comparaison des résultats de la littérature. Cependant des essais effectués sur des données LWIR ont montré un manque de prédiction de la surface du grain lorsque les grains étaient irréguliers avec présence d’agrégats minéraux. La seconde expérience a consisté, en une analyse quantitaive comparative entre deux bases de données de Ground Truth (GT), nommée rigid-GT et observed-GT (rigide-GT: étiquet manuel de la région, observée-GT:étiquetage manuel les pixels). La précision des résultats était 1.5 fois meilleur lorsque l’on a utlisé la base de données observed-GT que rigid-GT. Pour les deux dernières epxérience, des données venant d’un MEB (Microscope Électronique à Balayage) ainsi que d’un microscopie à fluorescence (XRF) ont été ajoutées. Ces données ont permis d’introduire des informations relatives tant aux agrégats minéraux qu’à la surface des grains. Les résultats ont été comparés par des techniques d’identification automatique des minéraux, utilisant ArcGIS. Cette dernière a montré une performance prometteuse quand à l’identification automatique et à aussi été utilisée pour la GT de validation. Dans l’ensemble, les quatre méthodes de cette thèse représentent des méthodologies bénéfiques pour l’identification des minéraux. Ces méthodes présentent l’avantage d’être non-destructives, relativement précises et d’avoir un faible coût en temps calcul ce qui pourrait les qualifier pour être utilisée dans des conditions de laboratoire ou sur le terrain.The geological applications of hyperspectral infrared imagery mainly consist in mineral identification, mapping, airborne or portable instruments, and core logging. Finding the mineral indicators offer considerable benefits in terms of mineralogy and mineral exploration which usually involves application of portable instrument and core logging. Moreover, faster and more mechanized systems development increases the precision of identifying mineral indicators and avoid any possible mis-classification. Therefore, the objective of this thesis was to create a tool to using hyperspectral infrared imagery and process the data through image analysis and machine learning methods to identify small size mineral grains used as mineral indicators. This system would be applied for different circumstances to provide an assistant for geological analysis and mineralogy exploration. The experiments were conducted in laboratory conditions in the long-wave infrared (7.7μm to 11.8μm - LWIR), with a LWIR-macro lens (to improve spatial resolution), an Infragold plate, and a heating source. The process began with a method to calculate the continuum removal. The approach is the application of Non-negative Matrix Factorization (NMF) to extract Rank-1 NMF and estimate the down-welling radiance and then compare it with other conventional methods. The results indicate successful suppression of the continuum from the spectra and enable the spectra to be compared with spectral libraries. Afterwards, to have an automated system, supervised and unsupervised approaches have been tested for identification of pyrope, olivine and quartz grains. The results indicated that the unsupervised approach was more suitable due to independent behavior against training stage. Once these results obtained, two algorithms were tested to create False Color Composites (FCC) applying a clustering approach. The results of this comparison indicate significant computational efficiency (more than 20 times faster) and promising performance for mineral identification. Finally, the reliability of the automated LWIR hyperspectral infrared mineral identification has been tested and the difficulty for identification of the irregular grain’s surface along with the mineral aggregates has been verified. The results were compared to two different Ground Truth(GT) (i.e. rigid-GT and observed-GT) for quantitative calculation. Observed-GT increased the accuracy up to 1.5 times than rigid-GT. The samples were also examined by Micro X-ray Fluorescence (XRF) and Scanning Electron Microscope (SEM) in order to retrieve information for the mineral aggregates and the grain’s surface (biotite, epidote, goethite, diopside, smithsonite, tourmaline, kyanite, scheelite, pyrope, olivine, and quartz). The results of XRF imagery compared with automatic mineral identification techniques, using ArcGIS, and represented a promising performance for automatic identification and have been used for GT validation. In overall, the four methods (i.e. 1.Continuum removal methods; 2. Classification or clustering methods for mineral identification; 3. Two algorithms for clustering of mineral spectra; 4. Reliability verification) in this thesis represent beneficial methodologies to identify minerals. These methods have the advantages to be a non-destructive, relatively accurate and have low computational complexity that might be used to identify and assess mineral grains in the laboratory conditions or in the field