Approximate Message Passing in Coded Aperture Snapshot Spectral Imaging

We consider a compressive hyperspectral imaging reconstruction problem, where three-dimensional spatio-spectral information about a scene is sensed by a coded aperture snapshot spectral imager (CASSI). The approximate message passing (AMP) framework is utilized to reconstruct hyperspectral images from CASSI measurements, and an adaptive Wiener filter is employed as a three-dimensional image denoiser within AMP. We call our algorithm "AMP-3D-Wiener." The simulation results show that AMP-3D-Wiener outperforms existing widely-used algorithms such as gradient projection for sparse reconstruction (GPSR) and two-step iterative shrinkage/thresholding (TwIST) given the same amount of runtime. Moreover, in contrast to GPSR and TwIST, AMP-3D-Wiener need not tune any parameters, which simplifies the reconstruction process.Comment: to appear in Globalsip 201

mHealth hyperspectral learning for instantaneous spatiospectral imaging of hemodynamics

Hyperspectral imaging acquires data in both the spatial and frequency domains to offer abundant physical or biological information. However, conventional hyperspectral imaging has intrinsic limitations of bulky instruments, slow data acquisition rate, and spatiospectral tradeoff. Here we introduce hyperspectral learning for snapshot hyperspectral imaging in which sampled hyperspectral data in a small subarea are incorporated into a learning algorithm to recover the hypercube. Hyperspectral learning exploits the idea that a photograph is more than merely a picture and contains detailed spectral information. A small sampling of hyperspectral data enables spectrally informed learning to recover a hypercube from an RGB image. Hyperspectral learning is capable of recovering full spectroscopic resolution in the hypercube, comparable to high spectral resolutions of scientific spectrometers. Hyperspectral learning also enables ultrafast dynamic imaging, leveraging ultraslow video recording in an off-the-shelf smartphone, given that a video comprises a time series of multiple RGB images. To demonstrate its versatility, an experimental model of vascular development is used to extract hemodynamic parameters via statistical and deep-learning approaches. Subsequently, the hemodynamics of peripheral microcirculation is assessed at an ultrafast temporal resolution up to a millisecond, using a conventional smartphone camera. This spectrally informed learning method is analogous to compressed sensing; however, it further allows for reliable hypercube recovery and key feature extractions with a transparent learning algorithm. This learning-powered snapshot hyperspectral imaging method yields high spectral and temporal resolutions and eliminates the spatiospectral tradeoff, offering simple hardware requirements and potential applications of various machine-learning techniques.Comment: This paper will appear in PNAS Nexu

Mixture-Net: Low-Rank Deep Image Prior Inspired by Mixture Models for Spectral Image Recovery

This paper proposes a non-data-driven deep neural network for spectral image recovery problems such as denoising, single hyperspectral image super-resolution, and compressive spectral imaging reconstruction. Unlike previous methods, the proposed approach, dubbed Mixture-Net, implicitly learns the prior information through the network. Mixture-Net consists of a deep generative model whose layers are inspired by the linear and non-linear low-rank mixture models, where the recovered image is composed of a weighted sum between the linear and non-linear decomposition. Mixture-Net also provides a low-rank decomposition interpreted as the spectral image abundances and endmembers, helpful in achieving remote sensing tasks without running additional routines. The experiments show the MixtureNet effectiveness outperforming state-of-the-art methods in recovery quality with the advantage of architecture interpretability

Residual Degradation Learning Unfolding Framework with Mixing Priors across Spectral and Spatial for Compressive Spectral Imaging

To acquire a snapshot spectral image, coded aperture snapshot spectral imaging (CASSI) is proposed. A core problem of the CASSI system is to recover the reliable and fine underlying 3D spectral cube from the 2D measurement. By alternately solving a data subproblem and a prior subproblem, deep unfolding methods achieve good performance. However, in the data subproblem, the used sensing matrix is ill-suited for the real degradation process due to the device errors caused by phase aberration, distortion; in the prior subproblem, it is important to design a suitable model to jointly exploit both spatial and spectral priors. In this paper, we propose a Residual Degradation Learning Unfolding Framework (RDLUF), which bridges the gap between the sensing matrix and the degradation process. Moreover, a Mix$S^2$ Transformer is designed via mixing priors across spectral and spatial to strengthen the spectral-spatial representation capability. Finally, plugging the Mix$S^2$ Transformer into the RDLUF leads to an end-to-end trainable neural network RDLUF-Mix$S^2$. Experimental results establish the superior performance of the proposed method over existing ones.Comment: 10 pages, 5 figure

Versatile compressive microscope for hyperspectral transmission and fluorescence lifetime imaging

Increasing demand for multimodal characterization and imaging of new materials entails the combination of various methods in a single microscopic setup. Hyperspectral imaging of transmission spectra or photoluminescence (PL) decay imaging count among the most used methods. Nevertheless, these methods require very different working conditions and instrumentation. Therefore, combining the methods into a single microscopic system is seldom implemented. Here we demonstrate a novel versatile microscope based on single-pixel imaging, where we use a simple optical configuration to measure the hyperspectral information, as well as fluorescence lifetime imaging (FLIM). The maps are inherently spatially matched and can be taken with spectral resolution limited by the resolution of the used spectrometer (3 nm) or temporal resolution set by PL decay measurement (120 ps). We verify the system's performance by its comparison to the standard FLIM and non-imaging transmission spectroscopy. Our approach enabled us to switch between a broad field-of-view and micrometer resolution without changing the optical configuration. At the same time, the used design opens the possibility to add a variety of other characterization methods. This article demonstrates a simple, affordable way of complex material studies with huge versatility for the imaging parameters.publishedVersio

Advances in Raman hyperspectral compressive detection instrumentation for fast label free classification, quantitation and imaging

Multiple prototypes of hyperspectral compressive detection (CD) Raman spectrometers have previously been constructed in the Ben-Amotz lab and have proven to be useful for fast, label-free chemical identification, quantitation and imaging. The CD spectrometer consists of a volume holographic grating (VHG) that linearly disperses the Raman photons into its component wavelengths and all wavelengths are focused onto a digital micromirror devise (DMD). The DMD is an optical modulator that consists of an array of programmable 10μm mirrors that can reflect photons in either +12° or -12° to the incoming light. The DMD is tilted such that the +12° photons go back through the focusing lens and the VHG and is focused onto a single 150μm photon counting avalanche photodiode detector(APD). In chapter 1 of the thesis I describe the construction of a new CD Raman spectrometer capable of fast hyperspectral imaging that has better photon collection efficiency and fewer photon losses compared to its predecessors. The new spectrometer consists of a VHG and a DMD, however, the DMD is not tilted but is perpendicular to the incoming Raman photons. All the Raman photons modulated by the DMD are symmetrically detected in the +12° and -12° by two photon counting photomultiplier tube(PMT) detector modules. The new spectrometer avoids a double pass through the optics and hence has fewer losses associated due to reflection transmission of the optics. Full spectral measurements are made by consecutively scanning through columns of the DMD mirrors and measuring the intensity of photons associated with each wavelength. CD measurements are made by multiplexing wavelengths channels onto the detectors and can be done by applying optimal binary(OB) or Hadamard filters. The new optical design has a spectral window from 150cm-1 to 4000 cm-1 and the improvement in the photon collection efficiency allows classification and imaging speeds of 10μs per point with 13mW of laser power on the sample, and is significantly faster than measurements made with the previous prototype. In chapter 2 of the thesis I describe the construction of a new instrument which is equipped with both a hyperspectral CD spectrometer as well as a traditional Czerny Turner spectrometer. A flip mirror after the Raman microscope directs the Raman scattered beam either towards the CD spectrometer (with the mirror down) or towards the Czerny Turner spectrometer. This instrument allows us to perform head to head comparisons of the two spectrometers using the same Raman scattered photons emitted by the sample. The CD spectrometer uses hardware optical filters to perform compressed chemometric measurements to classify chemicals. The traditional spectrometer uses the CCD to measure full spectral data and chemometric analysis is performed to extract lower dimensional chemical information post measurement. Chemical classification results obtained using two sets of chemicals with differing degrees of spectral overlap show that CD classification is comparable to full spectral classification in the high signal regime. However, for signals consisting of less than 1000 total photon counts, CD classification outperforms full spectral classification. In chapter 3 of the thesis, Raman spectroscopy is used to probe changes in vibrational spectra of nucleotide solutions and hanging droplets containing RNA crystals at different pH. Self-modeling curve resolution (SMCR) applied to full Raman is used to extract solute correlated (SC) Raman spectral components that contain solute spectra with minimal interference from the surrounding solvent. The goal of these studies is to show that Raman spectroscopy can be used to study biological molecules in aqueous environments, with minimal sample preparation and without the need of labels

Physically Plausible Spectral Reconstruction

Spectral reconstruction algorithms recover spectra from RGB sensor responses. Recent methods—with the very best algorithms using deep learning—can already solve this problem with good spectral accuracy. However, the recovered spectra are physically incorrect in that they do not induce the RGBs from which they are recovered. Moreover, if the exposure of the RGB image changes then the recovery performance often degrades significantly—i.e., most contemporary methods only work for a fixed exposure. In this paper, we develop a physically accurate recovery method: the spectra we recover provably induce the same RGBs. Key to our approach is the idea that the set of spectra that integrate to the same RGB can be expressed as the sum of a unique fundamental metamer (spanned by the camera’s spectral sensitivities and linearly related to the RGB) and a linear combination of a vector space of metameric blacks (orthogonal to the spectral sensitivities). Physically plausible spectral recovery resorts to finding a spectrum that adheres to the fundamental metamer plus metameric black decomposition. To further ensure spectral recovery that is robust to changes in exposure, we incorporate exposure changes in the training stage of the developed method. In experiments we evaluate how well the methods recover spectra and predict the actual RGBs and RGBs under different viewing conditions (changing illuminations and/or cameras). The results show that our method generally improves the state-of-the-art spectral recovery (with more stabilized performance when exposure varies) and provides zero colorimetric error. Moreover, our method significantly improves the color fidelity under different viewing conditions, with up to a 60% reduction in some cases