Hyperspectral and Multispectral Image Fusion using Optimized Twin Dictionaries

Spectral or spatial dictionary has been widely used in fusing low-spatial-resolution hyperspectral (LH) images and high-spatial-resolution multispectral (HM) images. However, only using spectral dictionary is insufficient for preserving spatial information, and vice versa. To address this problem, a new LH and HM image fusion method termed OTD using optimized twin dictionaries is proposed in this paper. The fusion problem of OTD is formulated analytically in the framework of sparse representation, as an optimization of twin spectral-spatial dictionaries and their corresponding sparse coefficients. More specifically, the spectral dictionary representing the generalized spectrums and its spectral sparse coefficients are optimized by utilizing the observed LH and HM images in the spectral domain; and the spatial dictionary representing the spatial information and its spatial sparse coefficients are optimized by modeling the rest of high-frequency information in the spatial domain. In addition, without non-negative constraints, the alternating direction methods of multipliers (ADMM) are employed to implement the above optimization process. Comparison results with the related state-of-the-art fusion methods on various datasets demonstrate that our proposed OTD method achieves a better fusion performance in both spatial and spectral domains

Model Inspired Autoencoder for Unsupervised Hyperspectral Image Super-Resolution

This paper focuses on hyperspectral image (HSI) super-resolution that aims to fuse a low-spatial-resolution HSI and a high-spatial-resolution multispectral image to form a high-spatial-resolution HSI (HR-HSI). Existing deep learning-based approaches are mostly supervised that rely on a large number of labeled training samples, which is unrealistic. The commonly used model-based approaches are unsupervised and flexible but rely on hand-craft priors. Inspired by the specific properties of model, we make the first attempt to design a model inspired deep network for HSI super-resolution in an unsupervised manner. This approach consists of an implicit autoencoder network built on the target HR-HSI that treats each pixel as an individual sample. The nonnegative matrix factorization (NMF) of the target HR-HSI is integrated into the autoencoder network, where the two NMF parts, spectral and spatial matrices, are treated as decoder parameters and hidden outputs respectively. In the encoding stage, we present a pixel-wise fusion model to estimate hidden outputs directly, and then reformulate and unfold the model's algorithm to form the encoder network. With the specific architecture, the proposed network is similar to a manifold prior-based model, and can be trained patch by patch rather than the entire image. Moreover, we propose an additional unsupervised network to estimate the point spread function and spectral response function. Experimental results conducted on both synthetic and real datasets demonstrate the effectiveness of the proposed approach

Coupled Convolutional Neural Network with Adaptive Response Function Learning for Unsupervised Hyperspectral Super-Resolution

Due to the limitations of hyperspectral imaging systems, hyperspectral imagery (HSI) often suffers from poor spatial resolution, thus hampering many applications of the imagery. Hyperspectral super-resolution refers to fusing HSI and MSI to generate an image with both high spatial and high spectral resolutions. Recently, several new methods have been proposed to solve this fusion problem, and most of these methods assume that the prior information of the Point Spread Function (PSF) and Spectral Response Function (SRF) are known. However, in practice, this information is often limited or unavailable. In this work, an unsupervised deep learning-based fusion method - HyCoNet - that can solve the problems in HSI-MSI fusion without the prior PSF and SRF information is proposed. HyCoNet consists of three coupled autoencoder nets in which the HSI and MSI are unmixed into endmembers and abundances based on the linear unmixing model. Two special convolutional layers are designed to act as a bridge that coordinates with the three autoencoder nets, and the PSF and SRF parameters are learned adaptively in the two convolution layers during the training process. Furthermore, driven by the joint loss function, the proposed method is straightforward and easily implemented in an end-to-end training manner. The experiments performed in the study demonstrate that the proposed method performs well and produces robust results for different datasets and arbitrary PSFs and SRFs

System of System Integration for Hyperspectral Imaging Microscopy

Hyperspectral imaging (HSI) has become a leading tool in the medical field due to its capabilities for providing assessments of tissue pathology and separation of fluorescence signals. Acquisition speeds have been slow due to the need to acquire signal in many spectral bands and the light losses associated with technologies of spectral filtering. Traditional methods resulted in limited signal strength which placed limitations on time sensitive and photosensitive assays. For example, the distribution of cyclic adenosine monophosphate (cAMP) is largely undetermined because current microscope technologies lack the combination of speed, resolution, and spectral ability to accurately measure Forster resonance energy transfer (FRET). The work presented in this dissertation assesses the feasibility of integrating excitation-scanning hyperspectral imaging methods in widefield and confocal microscopy as a potential solution to improving acquisition speeds without compromising sensitivity and specificity. Our laboratory has previously proposed excitation-scanning approaches to improve signal-to-noise ratio (SNR) and showed that by using excitation-scanning, most-to-all emitted light at each excitation wavelength band can be detected which in turn, increases the SNR. This dissertation describes development and early feasibility studies for two novel prototype concepts as an alternative excitation-scanning HSI technology that may xvi increase acquisition speeds without compromising sensitivity or specificity. To achieve this, two new technologies for excitation-scanning HSI were conceptually designed: - LED-based spectral illumination for widefield microscopy - Supercontinuum-laser-based spectral illumination for spinning disk confocal microscopy. Next, design concepts were theoretically evaluated and optimized, leading to prototype testing. To evaluate the performance of each concept, prototype systems were integrated with other systems and subsystems, calibrated and feasibility assays were executed. This dissertation is divided into three main sections: 1) early development feasibility results of an excitation-scanning widefield system of systems prototype utilizing LED-based HSI, 2) Excitation-scanning HSI and image analysis methods used for endmember identification in fluorescence microscopy studies, and 3) early development feasibility of an excitation-scanning confocal SoS prototype utilizing a supercontinuum laser light source. Integration and testing results proved initial feasibility of both LED-based and broadband-based SoSs. The LED-based light source was successfully tested on a widefield microscope, while the broadband light source system was successfully tested on a confocal microscope. Feasibility for the LED-based system showed that further optical transmission optimization is needed to achieve high acquisition rates without compromising sensitivity or specificity. Early feasibility study results for the broadband-based system showed a successful proof of concept. Findings presented in this dissertation are expected to impact the fields of cellular physiology, medical sciences, and clinical diagnostics by providing the ability for high speed, high sensitivity microscopic imaging with spectroscopic discrimination

Hyperspectral Image Classification -- Traditional to Deep Models: A Survey for Future Prospects

Hyperspectral Imaging (HSI) has been extensively utilized in many real-life applications because it benefits from the detailed spectral information contained in each pixel. Notably, the complex characteristics i.e., the nonlinear relation among the captured spectral information and the corresponding object of HSI data make accurate classification challenging for traditional methods. In the last few years, Deep Learning (DL) has been substantiated as a powerful feature extractor that effectively addresses the nonlinear problems that appeared in a number of computer vision tasks. This prompts the deployment of DL for HSI classification (HSIC) which revealed good performance. This survey enlists a systematic overview of DL for HSIC and compared state-of-the-art strategies of the said topic. Primarily, we will encapsulate the main challenges of traditional machine learning for HSIC and then we will acquaint the superiority of DL to address these problems. This survey breakdown the state-of-the-art DL frameworks into spectral-features, spatial-features, and together spatial-spectral features to systematically analyze the achievements (future research directions as well) of these frameworks for HSIC. Moreover, we will consider the fact that DL requires a large number of labeled training examples whereas acquiring such a number for HSIC is challenging in terms of time and cost. Therefore, this survey discusses some strategies to improve the generalization performance of DL strategies which can provide some future guidelines