Illumination Invariant Deep Learning for Hyperspectral Data

Motivated by the variability in hyperspectral images due to illumination and the difficulty in acquiring labelled data, this thesis proposes different approaches for learning illumination invariant feature representations and classification models for hyperspectral data captured outdoors, under natural sunlight. The approaches integrate domain knowledge into learning algorithms and hence does not rely on a priori knowledge of atmospheric parameters, additional sensors or large amounts of labelled training data. Hyperspectral sensors record rich semantic information from a scene, making them useful for robotics or remote sensing applications where perception systems are used to gain an understanding of the scene. Images recorded by hyperspectral sensors can, however, be affected to varying degrees by intrinsic factors relating to the sensor itself (keystone, smile, noise, particularly at the limits of the sensed spectral range) but also by extrinsic factors such as the way the scene is illuminated. The appearance of the scene in the image is tied to the incident illumination which is dependent on variables such as the position of the sun, geometry of the surface and the prevailing atmospheric conditions. Effects like shadows can make the appearance and spectral characteristics of identical materials to be significantly different. This degrades the performance of high-level algorithms that use hyperspectral data, such as those that do classification and clustering. If sufficient training data is available, learning algorithms such as neural networks can capture variability in the scene appearance and be trained to compensate for it. Learning algorithms are advantageous for this task because they do not require a priori knowledge of the prevailing atmospheric conditions or data from additional sensors. Labelling of hyperspectral data is, however, difficult and time-consuming, so acquiring enough labelled samples for the learning algorithm to adequately capture the scene appearance is challenging. Hence, there is a need for the development of techniques that are invariant to the effects of illumination that do not require large amounts of labelled data. In this thesis, an approach to learning a representation of hyperspectral data that is invariant to the effects of illumination is proposed. This approach combines a physics-based model of the illumination process with an unsupervised deep learning algorithm, and thus requires no labelled data. Datasets that vary both temporally and spatially are used to compare the proposed approach to other similar state-of-the-art techniques. The results show that the learnt representation is more invariant to shadows in the image and to variations in brightness due to changes in the scene topography or position of the sun in the sky. The results also show that a supervised classifier can predict class labels more accurately and more consistently across time when images are represented using the proposed method. Additionally, this thesis proposes methods to train supervised classification models to be more robust to variations in illumination where only limited amounts of labelled data are available. The transfer of knowledge from well-labelled datasets to poorly labelled datasets for classification is investigated. A method is also proposed for enabling small amounts of labelled samples to capture the variability in spectra across the scene. These samples are then used to train a classifier to be robust to the variability in the data caused by variations in illumination. The results show that these approaches make convolutional neural network classifiers more robust and achieve better performance when there is limited labelled training data. A case study is presented where a pipeline is proposed that incorporates the methods proposed in this thesis for learning robust feature representations and classification models. A scene is clustered using no labelled data. The results show that the pipeline groups the data into clusters that are consistent with the spatial distribution of the classes in the scene as determined from ground truth

Search for Higgs-Boson Production in Association with a Top-Quark Pair in the Boosted Regime with the CMS Experiment

In 2012, the last missing piece of the Standard Model of particle physics has been discovered by the ATLAS and CMS collaborations, the Higgs boson. Still, some production modes of this particle remain undiscovered to date. One of them is the Higgs-boson production in association with a top-quark pair (ttH). A special feature of this process is the direct access to one of the most important Higgs-boson properties: the top-Higgs Yukawa coupling. The small cross section of ttH production is one of the reasons, why this process still remains unobserved. Especially for a Higgs boson decaying into a bottom-quark pair, the irreducible background by top-quark pair production is overwhelming. A further challenge is the ambiguous assignment of jets to the numerous final-state particles in the reconstruction of the collision event. A solution to this problem is provided by the investigation of a phase space, where the massive particles feature large transverse momenta, the boosted regime. For this special case, the decay products of the massive particles are collimated and the original massive particles can be reconstructed and identified with the help of dedicated algorithms. This thesis presents the implementation, optimization, and execution of the boosted analysis strategy in a search for ttH production with a Higgs-boson decay into a bottom-quark pair and a semileptonic top-quark pair decay. The search is performed based on proton-proton collisions at a center-of-mass energy of √s = 13 TeV recorded with the CMS experiment in 2015. It represents the first search for ttH production with a Higgs-boson decay into a bottom-quark pair and a semileptonic top-quark pair decay at a center-of-mass energy of √s = 13 TeV. Further, this search features the first dedicated investigation of the boosted regime in the search for ttH production

Cross-Spectral Face Recognition Between Near-Infrared and Visible Light Modalities.

In this thesis, improvement of face recognition performance with the use of images from the visible (VIS) and near-infrared (NIR) spectrum is attempted. Face recognition systems can be adversely affected by scenarios which encounter a significant amount of illumination variation across images of the same subject. Cross-spectral face recognition systems using images collected across the VIS and NIR spectrum can counter the ill-effects of illumination variation by standardising both sets of images. A novel preprocessing technique is proposed, which attempts the transformation of faces across both modalities to a feature space with enhanced correlation. Direct matching across the modalities is not possible due to the inherent spectral differences between NIR and VIS face images. Compared to a VIS light source, NIR radiation has a greater penetrative depth when incident on human skin. This fact, in addition to the greater number of scattering interactions within the skin by rays from the NIR spectrum can alter the morphology of the human face enough to disable a direct match with the corresponding VIS face. Several ways to bridge the gap between NIR-VIS faces have been proposed previously. Mostly of a data-driven approach, these techniques include standardised photometric normalisation techniques and subspace projections. A generative approach driven by a true physical model has not been investigated till now. In this thesis, it is proposed that a large proportion of the scattering interactions present in the NIR spectrum can be accounted for using a model for subsurface scattering. A novel subsurface scattering inversion (SSI) algorithm is developed that implements an inversion approach based on translucent surface rendering by the computer graphics field, whereby the reversal of the first order effects of subsurface scattering is attempted. The SSI algorithm is then evaluated against several preprocessing techniques, and using various permutations of feature extraction and subspace projection algorithms. The results of this evaluation show an improvement in cross spectral face recognition performance using SSI over existing Retinex-based approaches. The top performing combination of an existing photometric normalisation technique, Sequential Chain, is seen to be the best performing with a Rank 1 recognition rate of 92. 5%. In addition, the improvement in performance using non-linear projection models shows an element of non-linearity exists in the relationship between NIR and VIS

A novel band selection and spatial noise reduction method for hyperspectral image classification.

As an essential reprocessing method, dimensionality reduction (DR) can reduce the data redundancy and improve the performance of hyperspectral image (HSI) classification. A novel unsupervised DR framework with feature interpretability, which integrates both band selection (BS) and spatial noise reduction method, is proposed to extract low-dimensional spectral-spatial features of HSI. We proposed a new Neighboring band Grouping and Normalized Matching Filter (NGNMF) for BS, which can reduce the data dimension whilst preserve the corresponding spectral information. An enhanced 2-D singular spectrum analysis (E2DSSA) method is also proposed to extract the spatial context and structural information from each selected band, aiming to decrease the intra-class variability and reduce the effect of noise in the spatial domain. The support vector machine (SVM) classifier is used to evaluate the effectiveness of the extracted spectral-spatial low-dimensional features. Experimental results on three publicly available HSI datasets have fully demonstrated the efficacy of the proposed NGNMF-E2DSSA method, which has surpassed a number of state-of-the-art DR methods

Fusion of PCA and segmented-PCA domain multiscale 2-D-SSA for effective spectral-spatial feature extraction and data classification in hyperspectral imagery.

As hyperspectral imagery (HSI) contains rich spectral and spatial information, a novel principal component analysis (PCA) and segmented-PCA (SPCA)-based multiscale 2-D-singular spectrum analysis (2-D-SSA) fusion method is proposed for joint spectral–spatial HSI feature extraction and classification. Considering the overall spectra and adjacent band correlations of objects, the PCA and SPCA methods are utilized first for spectral dimension reduction, respectively. Then, multiscale 2-D-SSA is applied onto the SPCA dimension-reduced images to extract abundant spatial features at different scales, where PCA is applied again for dimensionality reduction. The obtained multiscale spatial features are then fused with the global spectral features derived from PCA to form multiscale spectral–spatial features (MSF-PCs). The performance of the extracted MSF-PCs is evaluated using the support vector machine (SVM) classifier. Experiments on four benchmark HSI data sets have shown that the proposed method outperforms other state-of-the-art feature extraction methods, including several deep learning approaches, when only a small number of training samples are available

Automatic Identification of Algae using Low-cost Multispectral Fluorescence Digital Microscopy, Hierarchical Classification & Deep Learning

Harmful algae blooms (HABs) can produce lethal toxins and are a rising global concern. In response to this threat, many organizations are monitoring algae populations to determine if a water body might be contaminated. However, identifying algae types in a water sample requires a human expert, a taxonomist, to manually identify organisms using an optical microscope. This is a tedious, time-consuming process that is prone to human error and bias. Since many facilities lack on-site taxonomists, they must ship their water samples off site, further adding to the analysis time. Given the urgency of this problem, this thesis hypothesizes that multispectral fluorescence microscopy with a deep learning hierarchical classification structure is the optimal method to automatically identify algae in water on-site. To test this hypothesis, a low-cost system was designed and built which was able generate one brightfield image and four fluorescence images. Each of the four fluorescence images was designed to target a different pigment in algae, resulting in a unique autofluorescence spectral fingerprint for different phyla groups. To complement this hardware system, a software framework was designed and developed. This framework used the prior taxonomic structure of algae to create a hierarchical classification structure. This hierarchical classifier divided the classification task into three steps which were phylum, genus, and species level classification. Deep learning models were used at each branch of this hierarchical classifier allowing the optimal set of features to be implicitly learned from the input data. In order to test the efficacy of the proposed hardware system and corresponding software framework, a dataset of nine algae from 4 different phyla groups was created. A number of preprocessing steps were required to prepare the data for analysis. These steps were flat field correction, thresholding and cropping. With this multispectral imaging data, a number of spatial and spectral features were extracted for use in the feature-extraction-based models. This dataset was used to determine the relative performance of 12 different model architectures, and the proposed multispectral hierarchical deep learning approach achieved the top classification accuracy of 97% to the species level. Further inspection revealed that a traditional feature extraction method was able to achieve 95% to the phyla level when only using the multispectral fluorescence data. These observations strongly support that: (1) the proposed low-cost multispectral fluorescence imaging system, and (2) the proposed hierarchical structure based on the taxonomy prior, in combination with (3) deep learning methods for feature learning, is an effective method to automatically classify algae

Screening for Neonatal Jaundice by Smartphone Sclera Imaging

Jaundice is observed in over 60% of neonates and must be carefully monitored. Ifsevere cases go unnoticed, death or permanent disability can result. Neonatal jaun-dice causes 100,000 deaths yearly, with low-income countries in Africa and SouthAsia particularly affected. There is an unmet need for an accessible and objectivescreening method. This thesis proposes a smartphone camera-based method forscreening based on quantification of yellow discolouration in the sclera.The primary aim is to develop and test an app to screen for neonatal jaundicethat requires only the smartphone itself. To this end, a novel ambient subtractionmethod is proposed and validated, with less dependence on external hardware orcolour cards than previous app-based methods. Another aim is to investigate thebenefits of screening via the sclera. An existing dataset of newborn sclera images(n=87) is used to show that sclera chromaticity can predict jaundice severity.The neoSCB app is developed to predict total serum bilirubin (TSB) fromambient-subtracted sclera chromaticity via a flash/ no-flash image pair. A studyis conducted in Accra, Ghana to evaluate the app. With 847 capture sessions, thisis the largest study on image-based jaundice detection to date. A model trained onsclera chromaticity is found to be more accurate than one based on skin. The modelis validated on an independent dataset collected at UCLH (n=38).The neoSCB app has a sensitivity of 100% and a specificity of 76% in iden-tifying neonates with TSB≥250μmol/L (n=179). This is equivalent to the TcB(JM-105) data collected concurrently, and as good as the best-performing app in theliterature (BiliCam). Following a one-time calibration, neoSCB works without spe-cialist equipment, which could help widen access to effective jaundice screening