Computational Spectral Imaging: A Contemporary Overview

Spectral imaging collects and processes information along spatial and spectral coordinates quantified in discrete voxels, which can be treated as a 3D spectral data cube. The spectral images (SIs) allow identifying objects, crops, and materials in the scene through their spectral behavior. Since most spectral optical systems can only employ 1D or maximum 2D sensors, it is challenging to directly acquire the 3D information from available commercial sensors. As an alternative, computational spectral imaging (CSI) has emerged as a sensing tool where the 3D data can be obtained using 2D encoded projections. Then, a computational recovery process must be employed to retrieve the SI. CSI enables the development of snapshot optical systems that reduce acquisition time and provide low computational storage costs compared to conventional scanning systems. Recent advances in deep learning (DL) have allowed the design of data-driven CSI to improve the SI reconstruction or, even more, perform high-level tasks such as classification, unmixing, or anomaly detection directly from 2D encoded projections. This work summarises the advances in CSI, starting with SI and its relevance; continuing with the most relevant compressive spectral optical systems. Then, CSI with DL will be introduced, and the recent advances in combining the physical optical design with computational DL algorithms to solve high-level tasks

Compact multispectral pushframe camera for nanosatellites

In this paper we present an evolution of the single-pixel camera architecture, called "pushframe," which addresses the limitations of pushbroom cameras in space-based applications. In particular, it is well-suited to observing fast-moving scenes while retaining high spatial resolution and sensitivity. We show that the system is capable of producing color images with good fidelity and scalable resolution performance. The principle of our design broadens the choice of spectral ranges that can be captured, making it suitable for wide spectral ranges of infrared imaging

Surgical spectral imaging

Recent technological developments have resulted in the availability of miniaturised spectral imaging sensors capable of operating in the multi- (MSI) and hyperspectral imaging (HSI) regimes. Simultaneous advances in image-processing techniques and artificial intelligence (AI), especially in machine learning and deep learning, have made these data-rich modalities highly attractive as a means of extracting biological information non-destructively. Surgery in particular is poised to benefit from this, as spectrally-resolved tissue optical properties can offer enhanced contrast as well as diagnostic and guidance information during interventions. This is particularly relevant for procedures where inherent contrast is low under standard white light visualisation. This review summarises recent work in surgical spectral imaging (SSI) techniques, taken from Pubmed, Google Scholar and arXiv searches spanning the period 2013–2019. New hardware, optimised for use in both open and minimally-invasive surgery (MIS), is described, and recent commercial activity is summarised. Computational approaches to extract spectral information from conventional colour images are reviewed, as tip-mounted cameras become more commonplace in MIS. Model-based and machine learning methods of data analysis are discussed in addition to simulation, phantom and clinical validation experiments. A wide variety of surgical pilot studies are reported but it is apparent that further work is needed to quantify the clinical value of MSI/HSI. The current trend toward data-driven analysis emphasises the importance of widely-available, standardised spectral imaging datasets, which will aid understanding of variability across organs and patients, and drive clinical translation

Chromatic Dispersion Based Wide-Band, Fiber-Coupled, Tunable Light Source for Hyperspectral Imaging

Hyperspectral imaging is a powerful label-free imaging technique that provides topological and spectral information at once. In this work, we have designed and characterized a hyperspectral source based on the chromatic dispersion property of off-the-shelf lenses and converted a supercontinuum laser light source into a hyperspectral imaging light source for 490 nm to 900 nm wavelength range with a spectral resolution of 3.5 nm to 18 nm respectively. The potential of the source was demonstrated by imaging two color dots with different absorption bands. Further, we generated the hypercube of the lily ovary and dense connective tissue and measured their spectral signature as a function of wavelength. We also imaged the lower tongue of a healthy volunteer at 540 nm, 630 nm, and white light. Our simple hyperspectral light source design can easily be incorporated in a standard endoscope or microscope to perform hyperspectral imaging

Compressive sampling using a pushframe camera

The recently described pushframe imager, a parallelized single pixel camera capturing with a pushbroom-like motion, is intrinsically suited to both remote-sensing and compressive sampling. It optically applies a 2D mask to the imaged scene, before performing light integration along a single spatial axis, but previous work has not made use of the architecture's potential for taking measurements sparsely. In this paper we develop a strongly performing static binarized noiselet compressive sampling mask design, tailored to pushframe hardware, allowing both a single exposure per motion time-step, and retention of 2D correlations in the scene. Results from simulated and real-world captures are presented, with performance shown to be similar to that of immobile — and hence inappropriate for satellite use — whole-scene imagers. A particular feature of our sampling approach is that the degree of compression can be varied without altering the pattern, and we demonstrate the utility of this for efficiently storing and transmitting multi-spectral images

New implementations of phase-contrast imaging

Phase-contrast imaging is a method of imaging widely used in biomedical research and applications. It is a label-free method that exploits intrinsic differences in the refractive index of different tissues to differentiate between biological structures under analysis. The basic principle of phase-contrast imaging has inspired a lot of implementations that are suited for different applications. This thesis explores multiple novel implementations of phase-contrast imaging in the following order. 1, We combined scanning Oblique Back-illumination Microscope (sOBM) and confocal microscope to produce phase and fluorescence contrast images in an endomicroscopy configuration. This dual-modality design provides co-registered, complementary labeled and unlabeled contrast of the sample. We further miniaturized the probe by dispensing the two optical fibers in our old design. And we presented proof of principle demonstrations with ex-vivo mouse colon tissue. 2, Then we explored sOBM-based phase and amplitude contrast imaging under different wavelengths. Hyperspectral imaging is achieved by multiplexing a wide-range supercontinuum laser with a Michaelson interferometer (similar to Fourier transform spectroscopy). It features simultaneous acquisition of hyperspectral phase and amplitude images with arbitrarily thick scattering biological samples. Proof-of-principle demonstrations are presented with chorioallantoic membrane of a chick embryo, illustrating the possibility of high-resolution hemodynamics imaging in thick tissue. 3, We focused on increasing the throughput of flow cytometry with principle of phase-contrast imaging and compressive sensing. By utilizing the linearity of scattered patterns under partially coherent illumination, our cytometer can detect multiple objects in the same field of view. By utilizing an optimized matched filter on pupil plane, it also provides increased information capacity of each measurement without sacrificing speed. We demonstrated a throughput of over 10,000 particles/s with accuracy over 91% in our results. 4, A fourth part, which describes the principle and preliminary results of a computational fluorescence endomicroscope is also included. It uses a numerical method to achieve sectioning effect and renders a pseudo-3D image stack with a single shot. The results are compared with true-3D image stack acquired with a confocal microscope

Random access spectral imaging

A salient goal of spectral imaging is to record a so-called hyperspectral data-cube, consisting of two spatial and one spectral dimension. Traditional approaches are based on either time-sequential scanning in either the spatial or spectral dimension: spatial scanning involves passing a fixed aperture over a scene in the manner of a raster scan and spectral scanning is generally based on the use of a tuneable filter, where typically a series of narrow-band images of a fixed field of view are recorded and assembled into the data-cube. Such techniques are suitable only when the scene in question is static or changes slower than the scan rate. When considering dynamic scenes a time-resolved (snapshot) spectral imaging technique is required. Such techniques acquire the whole data-cube in a single measurement, but require a trade-off in spatial and spectral resolution. These trade-offs prevent current snapshot spectral imaging techniques from achieving resolutions on par with time-sequential techniques. Any snapshot device needs to have an optical architecture that allows it to gather light from the scene and map it to the detector in a way that allows the spatial and spectral components can be de-multiplexed to reconstruct the data-cube. This process results in the decreased resolution of snapshot devices as it becomes a problem of mapping a 3D data-cube onto a 2D detector. The sheer volume of data present in the data-cube also presents a processing challenge, particularly in the case of real-time processing. This thesis describes a prototype snapshot spectral imaging device that employs a random-spatial-access technique to record spectra only from the regions of interest in the scene, thus enabling maximisation of integration time and minimisation of data volume and recording rate. The aim of this prototype is to demonstrate how a particular optical architecture will allow for the effect of some of the above mentioned bottlenecks to be removed. Underpinning the basic concept is the fact that in all practical scenes most of the spectrally interesting information is contained in relatively few pixels. The prototype system uses random-spatial-access to multiple points in the scene considered to be of greatest interest. This enables time-resolved high resolution spectrometry to be made simultaneously at points across the full field of view. The enabling technology for the prototype was a digital micromirror device (DMD), which is an array of switchable mirrors that was used to create a two channel system. One channel was to a conventional imaging camera, while the other was to a spectrometer. The DMD acted as a dynamic aperture to the spectrometer and could be used to open and close slits in any part of the spectrometer aperture. The imaging channel was used to guide the selection of points of interest from the scene. An extensive geometric calibration was performed to determine the relationships between the DMD and two channels of the system. Two demonstrations of the prototype are given in this thesis: a dynamic biological scene and a static scene sampled using statistical sampling methods enabled by the dynamic aperture of the system. The dynamic scene consisted of red blood cells in motion and also undergoing a process of de-oxygenation which resulted in a change in the spectrum. Ten red blood cells were tracked across the scene and the expected change in spectrum was observed. For the second example the prototype was modified for Raman spectroscopy by adding laser illumination, a mineral sample was scanned and used to test statistical sampling methods. These methods exploited the re-configurable aperture of the system to sample the scene using blind random sampling and a grid based sampling approach. Other spectral imaging systems have a fixed aperture and cannot operate such sampling schemes

Side Information in Coded Aperture Compressive Spectral Imaging

Coded aperture compressive spectral imagers sense a three-dimensional cube by using two-dimensional projections of the coded and spectrally dispersed source. These imagers systems often rely on FPA detectors, SLMs, micromirror devices (DMDs), and dispersive elements. The use of the DMDs to implement the coded apertures facilitates the capture of multiple projections, each admitting a different coded aperture pattern. The DMD allows not only to collect the sufficient number of measurements for spectrally rich scenes or very detailed spatial scenes but to design the spatial structure of the coded apertures to maximize the information content on the compressive measurements. Although sparsity is the only signal characteristic usually assumed for reconstruction in compressing sensing, other forms of prior information such as side information have been included as a way to improve the quality of the reconstructions. This paper presents the coded aperture design in a compressive spectral imager with side information in the form of RGB images of the scene. The use of RGB images as side information of the compressive sensing architecture has two main advantages: the RGB is not only used to improve the reconstruction quality but to optimally design the coded apertures for the sensing process. The coded aperture design is based on the RGB scene and thus the coded aperture structure exploits key features such as scene edges. Real reconstructions of noisy compressed measurements demonstrate the benefit of the designed coded apertures in addition to the improvement in the reconstruction quality obtained by the use of side information