Matched Shrunken Cone Detector (MSCD): Bayesian Derivations and Case Studies for Hyperspectral Target Detection

Hyperspectral images (HSIs) possess non-negative properties for both hyperspectral signatures and abundance coefficients, which can be naturally modeled using cone-based representation. However, in hyperspectral target detection, cone-based methods are barely studied. In this paper, we propose a new regularized cone-based representation approach to hyperspectral target detection, as well as its two working models by incorporating into the cone representation l2-norm and l1-norm regularizations, respectively. We call the new approach the matched shrunken cone detector (MSCD). Also important, we provide principled derivations of the proposed MSCD from the Bayesian perspective: we show that MSCD can be derived by assuming a multivariate half-Gaussian distribution or a multivariate half-Laplace distribution as the prior distribution of the coefficients of the models. In the experimental studies, we compare the proposed MSCD with the subspace methods and the sparse representation-based methods for HSI target detection. Two real hyperspectral data sets are used for evaluating the detection performances on sub-pixel targets and full-pixel targets, respectively. Results show that the proposed MSCD can outperform other methods in both cases, demonstrating the competitiveness of the regularized cone-based representation

Essays on hyperspectral image analysis: classification and target detection

Over the past a few decades, hyperspectral imaging has drawn significant attention and become an important scientific tool for various fields of real-world applications. Among the research topics of hyperspectral image (HSI) analysis, two major topics -- HSI classification and HSI target detection have been intensively studied. Statistical learning has played a pivotal role in promoting the development of algorithms and methodologies for the two topics. Among the existing methods for HSI classification, sparse representation classification (SRC) has been widely investigated, which is based on the assumption that a signal can be represented by a linear combination of a small number of redundant bases (so called dictionary atoms). By virtue of the signal coherence in HSIs, a joint sparse model (JSM) has been successfully developed for HSI classification and has achieved promising performance. However, the JSM-based dictionary learning for HSIs is barely discussed. In addition, the non-negativity properties of coefficients in the JSM are also little touched. HSI target detection can be regarded as a special case of classification, i.e. a binary classification, but faces more challenges. Traditional statistical methods regard a test HSI pixel as a linear combination of several endmembers with corresponding fractions, i.e. based on the linear mixing model (LMM). However, due to the complicated environments in real-world problems, complex mixing effects may exist in HSIs and make the detection of targets more difficult. As a consequence, the performance of traditional LMM is limited. In this thesis, we focus on the topics of HSI classification and HSI target detection and propose five new methods to tackle the aforementioned issues in the two tasks. For the HSI classification, two new methods are proposed based on the JSM. The first proposed method focuses on the dictionary learning, which incorporates the JSM in the discriminative K-SVD learning algorithm, in order to learn a quality dictionary with rich information for improving the classification performance. The second proposed method focuses on developing the convex cone-based JSM, i.e. by incorporating the non-negativity constraints in the coefficients in the JSM. For the HSI target detection, three approaches are proposed based on the linear mixing model (LMM). The first approach takes account of interaction effects to tackle the mixing problems in HSI target detection. The second approach called matched shrunken subspace detector (MSSD) and the third approach, called matched cone shrunken detector (MSCD), both offer on Bayesian derivatives of regularisation constrained LMM. Specifically, the proposed MSSD is a regularised subspace-representation of LMM, while the proposed MSCD is a regularised cone-representation of LMM

Computational Multispectral Endoscopy

Minimal Access Surgery (MAS) is increasingly regarded as the de-facto approach in interventional medicine for conducting many procedures this is due to the reduced patient trauma and consequently reduced recovery times, complications and costs. However, there are many challenges in MAS that come as a result of viewing the surgical site through an endoscope and interacting with tissue remotely via tools, such as lack of haptic feedback; limited field of view; and variation in imaging hardware. As such, it is important best utilise the imaging data available to provide a clinician with rich data corresponding to the surgical site. Measuring tissue haemoglobin concentrations can give vital information, such as perfusion assessment after transplantation; visualisation of the health of blood supply to organ; and to detect ischaemia. In the area of transplant and bypass procedures measurements of the tissue tissue perfusion/total haemoglobin (THb) and oxygen saturation (SO2) are used as indicators of organ viability, these measurements are often acquired at multiple discrete points across the tissue using with a specialist probe. To acquire measurements across the whole surface of an organ one can use a specialist camera to perform multispectral imaging (MSI), which optically acquires sequential spectrally band limited images of the same scene. This data can be processed to provide maps of the THb and SO2 variation across the tissue surface which could be useful for intra operative evaluation. When capturing MSI data, a trade off often has to be made between spectral sensitivity and capture speed. The work in thesis first explores post processing blurry MSI data from long exposure imaging devices. It is of interest to be able to use these MSI data because the large number of spectral bands that can be captured, the long capture times, however, limit the potential real time uses for clinicians. Recognising the importance to clinicians of real-time data, the main body of this thesis develops methods around estimating oxy- and deoxy-haemoglobin concentrations in tissue using only monocular and stereo RGB imaging data

Quantitative Mapping of Soil Property Based on Laboratory and Airborne Hyperspectral Data Using Machine Learning

Soil visible and near-infrared spectroscopy provides a non-destructive, rapid and low-cost approach to quantify various soil physical and chemical properties based on their reflectance in the spectral range of 400–2500 nm. With an increasing number of large-scale soil spectral libraries established across the world and new space-borne hyperspectral sensors, there is a need to explore methods to extract informative features from reflectance spectra and produce accurate soil spectroscopic models using machine learning. Features generated from regional or large-scale soil spectral data play a key role in the quantitative spectroscopic model for soil properties. The Land Use/Land Cover Area Frame Survey (LUCAS) soil library was used to explore PLS-derived components and fractal features generated from soil spectra in this study. The gradient-boosting method performed well when coupled with extracted features on the estimation of several soil properties. Transfer learning based on convolutional neural networks (CNNs) was proposed to make the model developed from laboratory data transferable for airborne hyperspectral data. The soil clay map was successfully derived using HyMap imagery and the fine-tuned CNN model developed from LUCAS mineral soils, as deep learning has the potential to learn transferable features that generalise from the source domain to target domain. The external environmental factors like the presence of vegetation restrain the application of imaging spectroscopy. The reflectance data can be transformed into a vegetation suppressed domain with a force invariance approach, the performance of which was evaluated in an agricultural area using CASI airborne hyperspectral data. However, the relationship between vegetation and acquired spectra is complicated, and more efforts should put on removing the effects of external factors to make the model transferable from one sensor to another.:Abstract I Kurzfassung III Table of Contents V List of Figures IX List of Tables XIII List of Abbreviations XV 1 Introduction 1 1.1 Motivation 1 1.2 Soil spectra from different platforms 2 1.3 Soil property quantification using spectral data 4 1.4 Feature representation of soil spectra 5 1.5 Objectives 6 1.6 Thesis structure 7 2 Combining Partial Least Squares and the Gradient-Boosting Method for Soil Property Retrieval Using Visible Near-Infrared Shortwave Infrared Spectra 9 2.1 Abstract 10 2.2 Introduction 10 2.3 Materials and methods 13 2.3.1 The LUCAS soil spectral library 13 2.3.2 Partial least squares algorithm 15 2.3.3 Gradient-Boosted Decision Trees 15 2.3.4 Calculation of relative variable importance 16 2.3.5 Assessment 17 2.4 Results 17 2.4.1 Overview of the spectral measurement 17 2.4.2 Results of PLS regression for the estimation of soil properties 19 2.4.3 Results of PLS-GBDT for the estimation of soil properties 21 2.4.4 Relative important variables derived from PLS regression and the gradient-boosting method 24 2.5 Discussion 28 2.5.1 Dimension reduction for high-dimensional soil spectra 28 2.5.2 GBDT for quantitative soil spectroscopic modelling 29 2.6 Conclusions 30 3 Quantitative Retrieval of Organic Soil Properties from Visible Near-Infrared Shortwave Infrared Spectroscopy Using Fractal-Based Feature Extraction 31 3.1 Abstract 32 3.2 Introduction 32 3.3 Materials and Methods 35 3.3.1 The LUCAS topsoil dataset 35 3.3.2 Fractal feature extraction method 37 3.3.3 Gradient-boosting regression model 37 3.3.4 Evaluation 41 3.4 Results 42 3.4.1 Fractal features for soil spectroscopy 42 3.4.2 Effects of different step and window size on extracted fractal features 45 3.4.3 Modelling soil properties with fractal features 47 3.4.3 Comparison with PLS regression 49 3.5 Discussion 51 3.5.1 The importance of fractal dimension for soil spectra 51 3.5.2 Modelling soil properties with fractal features 52 3.6 Conclusions 53 4 Transfer Learning for Soil Spectroscopy Based on Convolutional Neural Networks and Its Application in Soil Clay Content Mapping Using Hyperspectral Imagery 55 4.1 Abstract 55 4.2 Introduction 56 4.3 Materials and Methods 59 4.3.1 Datasets 59 4.3.2 Methods 62 4.3.3 Assessment 67 4.4 Results and Discussion 67 4.4.1 Interpretation of mineral and organic soils from LUCAS dataset 67 4.4.2 1D-CNN and spectral index for LUCAS soil clay content estimation 69 4.4.3 Application of transfer learning for soil clay content mapping using the pre-trained 1D-CNN model 72 4.4.4 Comparison between spectral index and transfer learning 74 4.4.5 Large-scale soil spectral library for digital soil mapping at the local scale using hyperspectral imagery 75 4.5 Conclusions 75 5 A Case Study of Forced Invariance Approach for Soil Salinity Estimation in Vegetation-Covered Terrain Using Airborne Hyperspectral Imagery 77 5.1 Abstract 78 5.2 Introduction 78 5.3 Materials and Methods 81 5.3.1 Study area of Zhangye Oasis 81 5.3.2 Data description 82 5.3.3 Methods 83 5.3.3 Model performance assessment 85 5.4 Results and Discussion 86 5.4.1 The correlation between NDVI and soil salinity 86 5.4.2 Vegetation suppression performance using the Forced Invariance Approach 86 5.4.3 Estimation of soil properties using airborne hyperspectral data 88 5.5 Conclusions 90 6 Conclusions and Outlook 93 Bibliography 97 Acknowledgements 11

Illumination Invariant Deep Learning for Hyperspectral Data

Motivated by the variability in hyperspectral images due to illumination and the difficulty in acquiring labelled data, this thesis proposes different approaches for learning illumination invariant feature representations and classification models for hyperspectral data captured outdoors, under natural sunlight. The approaches integrate domain knowledge into learning algorithms and hence does not rely on a priori knowledge of atmospheric parameters, additional sensors or large amounts of labelled training data. Hyperspectral sensors record rich semantic information from a scene, making them useful for robotics or remote sensing applications where perception systems are used to gain an understanding of the scene. Images recorded by hyperspectral sensors can, however, be affected to varying degrees by intrinsic factors relating to the sensor itself (keystone, smile, noise, particularly at the limits of the sensed spectral range) but also by extrinsic factors such as the way the scene is illuminated. The appearance of the scene in the image is tied to the incident illumination which is dependent on variables such as the position of the sun, geometry of the surface and the prevailing atmospheric conditions. Effects like shadows can make the appearance and spectral characteristics of identical materials to be significantly different. This degrades the performance of high-level algorithms that use hyperspectral data, such as those that do classification and clustering. If sufficient training data is available, learning algorithms such as neural networks can capture variability in the scene appearance and be trained to compensate for it. Learning algorithms are advantageous for this task because they do not require a priori knowledge of the prevailing atmospheric conditions or data from additional sensors. Labelling of hyperspectral data is, however, difficult and time-consuming, so acquiring enough labelled samples for the learning algorithm to adequately capture the scene appearance is challenging. Hence, there is a need for the development of techniques that are invariant to the effects of illumination that do not require large amounts of labelled data. In this thesis, an approach to learning a representation of hyperspectral data that is invariant to the effects of illumination is proposed. This approach combines a physics-based model of the illumination process with an unsupervised deep learning algorithm, and thus requires no labelled data. Datasets that vary both temporally and spatially are used to compare the proposed approach to other similar state-of-the-art techniques. The results show that the learnt representation is more invariant to shadows in the image and to variations in brightness due to changes in the scene topography or position of the sun in the sky. The results also show that a supervised classifier can predict class labels more accurately and more consistently across time when images are represented using the proposed method. Additionally, this thesis proposes methods to train supervised classification models to be more robust to variations in illumination where only limited amounts of labelled data are available. The transfer of knowledge from well-labelled datasets to poorly labelled datasets for classification is investigated. A method is also proposed for enabling small amounts of labelled samples to capture the variability in spectra across the scene. These samples are then used to train a classifier to be robust to the variability in the data caused by variations in illumination. The results show that these approaches make convolutional neural network classifiers more robust and achieve better performance when there is limited labelled training data. A case study is presented where a pipeline is proposed that incorporates the methods proposed in this thesis for learning robust feature representations and classification models. A scene is clustered using no labelled data. The results show that the pipeline groups the data into clusters that are consistent with the spatial distribution of the classes in the scene as determined from ground truth

Deep learning methods for modelling forest biomass and structures from hyperspectral imagery

Forests affect the environment and ecosystems in multiple ways. Hence, understanding the forest processes and vegetation characteristics help us protect the environment better, reserve the biodiversity, and mitigate the hazardous impacts of climate change. There are studies in hyperspectral remote sensing that employ both empirical and artificial intelligence (AI) methods to analyze and predict the vegetation parameters. However, these methods have weaknesses. First, the empirical methods are inefficient because they cannot fully utilize a large amount of hyperspectral data. Secondly, even though the existing AI-based methods can achieve remarkable results, they are only validated on small-scale datasets that have simple forest structures. Thus, a robust technique that can effectively model complex forest structures on large-scale datasets is an open challenge. This thesis directly addresses the challenge by proposing a novel deep learning architecture that can jointly learn and model four discrete and twelve continuous forest parameters. The final model is comprised of three 3D convolution layers, a 3D multi-scale convolution block, a shared fully-connected layer, and two fully-connected layers for each learning task. The model uses a loss, namely focal loss, to address class imbalance problem and the gradient normalization for multi-task learning. Then, we record and compare the results of our comprehensive experiments. Overall, the proposed model reaches 78.32% class-balanced accuracy for the four classification tasks. For the regression tasks, the model achieves a notably low average mean absolute error (0.052) and high Pearson correlation coefficient (0.9) between predicted and target labels. In the end, the shortcomings of the thesis work are discussed and potential research areas for future work are suggested

CSTNet: A Dual-Branch Convolutional Network for Imaging of Reactive Flows using Chemical Species Tomography

Chemical Species Tomography (CST) has been widely used for in situ imaging of critical parameters, e.g. species concentration and temperature, in reactive flows. However, even with state-of-the-art computational algorithms the method is limited due to the inherently ill-posed and rank-deficient tomographic data inversion, and by high computational cost. These issues hinder its application for real-time flow diagnosis. To address them, we present here a novel CST-based convolutional neural Network (CSTNet) for high-fidelity, rapid, and simultaneous imaging of species concentration and temperature. CSTNet introduces a shared feature extractor that incorporates the CST measurement and sensor layout into the learning network. In addition, a dual-branch architecture is proposed for image reconstruction with crosstalk decoders that automatically learn the naturally correlated distributions of species concentration and temperature. The proposed CSTNet is validated both with simulated datasets, and with measured data from real flames in experiments using an industry-oriented sensor. Superior performance is found relative to previous approaches, in terms of robustness to measurement noise and millisecond-level computing time. This is the first time, to the best of our knowledge, that a deep learning-based algorithm for CST has been experimentally validated for simultaneous imaging of multiple critical parameters in reactive flows using a low-complexity optical sensor with severely limited number of laser beams.Comment: Submitted to IEEE Transactions on Neural Networks and Learning System

Authentication of Amadeo de Souza-Cardoso Paintings and Drawings With Deep Learning

Art forgery has a long-standing history that can be traced back to the Roman period and has become more rampant as the art market continues prospering. Reports disclosed that uncountable artworks circulating on the art market could be fake. Even some principal art museums and galleries could be exhibiting a good percentage of fake artworks. It is therefore substantially important to conserve cultural heritage, safeguard the interest of both the art market and the artists, as well as the integrity of artists’ legacies. As a result, art authentication has been one of the most researched and well-documented fields due to the ever-growing commercial art market in the past decades. Over the past years, the employment of computer science in the art world has flourished as it continues to stimulate interest in both the art world and the artificial intelligence arena. In particular, the implementation of Artificial Intelligence, namely Deep Learning algorithms and Neural Networks, has proved to be of significance for specialised image analysis. This research encompassed multidisciplinary studies on chemistry, physics, art and computer science. More specifically, the work presents a solution to the problem of authentication of heritage artwork by Amadeo de Souza-Cardoso, namely paintings, through the use of artificial intelligence algorithms. First, an authenticity estimation is obtained based on processing of images through a deep learning model that analyses the brushstroke features of a painting. Iterative, multi-scale analysis of the images is used to cover the entire painting and produce an overall indication of authenticity. Second, a mixed input, deep learning model is proposed to analyse pigments in a painting. This solves the image colour segmentation and pigment classification problem using hyperspectral imagery. The result is used to provide an indication of authenticity based on pigment classification and correlation with chemical data obtained via XRF analysis. Further algorithms developed include a deep learning model that tackles the pigment unmixing problem based on hyperspectral data. Another algorithm is a deep learning model that estimates hyperspectral images from sRGB images. Based on the established algorithms and results obtained, two applications were developed. First, an Augmented Reality mobile application specifically for the visualisation of pigments in the artworks by Amadeo. The mobile application targets the general public, i.e., art enthusiasts, museum visitors, art lovers or art experts. And second, a desktop application with multiple purposes, such as the visualisation of pigments and hyperspectral data. This application is designed for art specialists, i.e., conservators and restorers. Due to the special circumstances of the pandemic, trials on the usage of these applications were only performed within the Department of Conservation and Restoration at NOVA University Lisbon, where both applications received positive feedback.A falsificação de arte tem uma história de longa data que remonta ao período romano e tornou-se mais desenfreada à medida que o mercado de arte continua a prosperar. Relatórios revelaram que inúmeras obras de arte que circulam no mercado de arte podem ser falsas. Mesmo alguns dos principais museus e galerias de arte poderiam estar exibindo uma boa porcentagem de obras de arte falsas. Por conseguinte, é extremamente importante conservar o património cultural, salvaguardar os interesses do mercado da arte e dos artis- tas, bem como a integridade dos legados dos artistas. Como resultado, a autenticação de arte tem sido um dos campos mais pesquisados e bem documentados devido ao crescente mercado de arte comercial nas últimas décadas.Nos últimos anos, o emprego da ciência da computação no mundo da arte floresceu à medida que continua a estimular o interesse no mundo da arte e na arena da inteligência artificial. Em particular, a implementação da Inteligência Artificial, nomeadamente algoritmos de aprendizagem profunda (ou Deep Learning) e Redes Neuronais, tem-se revelado importante para a análise especializada de imagens.Esta investigação abrangeu estudos multidisciplinares em química, física, arte e informática. Mais especificamente, o trabalho apresenta uma solução para o problema da autenticação de obras de arte patrimoniais de Amadeo de Souza-Cardoso, nomeadamente pinturas, através da utilização de algoritmos de inteligência artificial. Primeiro, uma esti- mativa de autenticidade é obtida com base no processamento de imagens através de um modelo de aprendizagem profunda que analisa as características de pincelada de uma pintura. A análise iterativa e multiescala das imagens é usada para cobrir toda a pintura e produzir uma indicação geral de autenticidade. Em segundo lugar, um modelo misto de entrada e aprendizagem profunda é proposto para analisar pigmentos em uma pintura. Isso resolve o problema de segmentação de cores de imagem e classificação de pigmentos usando imagens hiperespectrais. O resultado é usado para fornecer uma indicação de autenticidade com base na classificação do pigmento e correlação com dados químicos obtidos através da análise XRF. Outros algoritmos desenvolvidos incluem um modelo de aprendizagem profunda que aborda o problema da desmistura de pigmentos com base em dados hiperespectrais. Outro algoritmo é um modelo de aprendizagem profunda estabelecidos e nos resultados obtidos, foram desenvolvidas duas aplicações. Primeiro, uma aplicação móvel de Realidade Aumentada especificamente para a visualização de pigmentos nas obras de Amadeo. A aplicação móvel destina-se ao público em geral, ou seja, entusiastas da arte, visitantes de museus, amantes da arte ou especialistas em arte. E, em segundo lugar, uma aplicação de ambiente de trabalho com múltiplas finalidades, como a visualização de pigmentos e dados hiperespectrais. Esta aplicação é projetada para especialistas em arte, ou seja, conservadores e restauradores. Devido às circunstâncias especiais da pandemia, os ensaios sobre a utilização destas aplicações só foram realizados no âmbito do Departamento de Conservação e Restauro da Universidade NOVA de Lisboa, onde ambas as candidaturas receberam feedback positivo

Matched shrunken subspace detectors for hyperspectral target detection

In this paper, we propose a new approach, called the matched shrunken subspace detector (MSSD), to target detection from hyperspectral images. The MSSD is developed by shrinking the abundance vectors of the target and background subspaces in the hypothesis models of the matched subspace detector (MSD), a popular subspace-based approach to target detection. The shrinkage is achieved by introducing simple l2-norm regularisation (also known as ridge regression or Tikhonov regularisation). We develop two types of MSSD, one with isotropic shrinkage and termed MSSD-i and the other with anisotropic shrinkage and termed MSSD-a. For these two new methods, we provide both the frequentist and Bayesian derivations. Experiments on a real hyperspectral imaging dataset called Hymap demonstrate that the proposed MSSD methods can outperform the original MSD for hyperspectral target detection