Spectral imaging in preclinical research and clinical pathology.

Spectral imaging methods are attracting increased interest from researchers and practitioners in basic science, pre-clinical and clinical arenas. A combination of better labeling reagents and better optics creates opportunities to detect and measure multiple parameters at the molecular and cellular level. These tools can provide valuable insights into the basic mechanisms of life, and yield diagnostic and prognostic information for clinical applications. There are many multispectral technologies available, each with its own advantages and limitations. This chapter will present an overview of the rationale for spectral imaging, and discuss the hardware, software and sample labeling strategies that can optimize its usefulness in clinical settings

Studies of Single-Molecule Dynamics in Microorganisms

Fluorescence microscopy is one of the most extensively used techniques in the life sciences. Considering the non-invasive sample preparation, enabling live-cell compliant imaging, and the specific fluorescence labeling, allowing for a specific visualization of virtually any cellular compound, it is possible to localize even a single molecule in living cells. This makes modern fluorescence microscopy a powerful toolbox. In the recent decades, the development of new, "super-resolution" fluorescence microscopy techniques, which surpass the diffraction limit, revolutionized the field. Single-Molecule Localization Microscopy (SMLM) is a class of super-resolution microscopy methods and it enables resolution of down to tens of nanometers. SMLM methods like Photoactivated Localization Microscopy (PALM), (direct) Stochastic Optical Reconstruction Microscopy ((d)STORM), Ground-State Depletion followed by Individual Molecule Return (GSDIM) and Point Accumulation for Imaging in Nanoscale Topography (PAINT) have allowed to investigate both, the intracellular spatial organization of proteins and to observe their real-time dynamics at the single-molecule level in live cells. The focus of this thesis was the development of novel tools and strategies for live-cell SingleParticle Tracking PALM (sptPALM) imaging and implementing them for biological research. In the first part of this thesis, I describe the development of new Photoconvertible Fluorescent Proteins (pcFPs) which are optimized for sptPALM lowering the phototoxic damage caused by the imaging procedure. Furthermore, we show that we can utilize them together with Photoactivatable Fluorescent Proteins (paFPs) to enable multi-target labeling and read-out in a single color channel, which significantly simplifies the sample preparation and imaging routines as well as data analysis of multi-color PALM imaging of live cells. In parallel to developing new fluorescent proteins, I developed a high throughput data analysis pipeline. I have implemented this pipeline in my second project, described in the second part of this thesis, where I have investigated the protein organization and dynamics of the CRISPR-Cas antiviral defense mechanism of bacteria in vivo at a high spatiotemporal level with the sptPALM approach. I was successful to show the differences in the target search dynamics of the CRISPR effector complexes as well as of single Cas proteins for different target complementarities. I have also first data describing longer-lasting bound-times between effector complex and their potential targets in vivo, for which only in vitro data has been available till today. In summary, this thesis is a significant contribution for both, the advances of current sptPALM imaging methods, as well as for the understanding of the native behavior of CRISPR-Cas systems in vivo

Programmable Spectrometry -- Per-pixel Classification of Materials using Learned Spectral Filters

Many materials have distinct spectral profiles. This facilitates estimation of the material composition of a scene at each pixel by first acquiring its hyperspectral image, and subsequently filtering it using a bank of spectral profiles. This process is inherently wasteful since only a set of linear projections of the acquired measurements contribute to the classification task. We propose a novel programmable camera that is capable of producing images of a scene with an arbitrary spectral filter. We use this camera to optically implement the spectral filtering of the scene's hyperspectral image with the bank of spectral profiles needed to perform per-pixel material classification. This provides gains both in terms of acquisition speed --- since only the relevant measurements are acquired --- and in signal-to-noise ratio --- since we invariably avoid narrowband filters that are light inefficient. Given training data, we use a range of classical and modern techniques including SVMs and neural networks to identify the bank of spectral profiles that facilitate material classification. We verify the method in simulations on standard datasets as well as real data using a lab prototype of the camera

Reduced and coded sensing methods for x-ray based security

Current x-ray technologies provide security personnel with non-invasive sub-surface imaging and contraband detection in various portal screening applications such as checked and carry-on baggage as well as cargo. Computed tomography (CT) scanners generate detailed 3D imagery in checked bags; however, these scanners often require significant power, cost, and space. These tomography machines are impractical for many applications where space and power are often limited such as checkpoint areas. Reducing the amount of data acquired would help reduce the physical demands of these systems. Unfortunately this leads to the formation of artifacts in various applications, thus presenting significant challenges in reconstruction and classification. As a result, the goal is to maintain a certain level of image quality but reduce the amount of data gathered. For the security domain this would allow for faster and cheaper screening in existing systems or allow for previously infeasible screening options due to other operational constraints. While our focus is predominantly on security applications, many of the techniques can be extended to other fields such as the medical domain where a reduction of dose can allow for safer and more frequent examinations. This dissertation aims to advance data reduction algorithms for security motivated x-ray imaging in three main areas: (i) development of a sensing aware dimensionality reduction framework, (ii) creation of linear motion tomographic method of object scanning and associated reconstruction algorithms for carry-on baggage screening, and (iii) the application of coded aperture techniques to improve and extend imaging performance of nuclear resonance fluorescence in cargo screening. The sensing aware dimensionality reduction framework extends existing dimensionality reduction methods to include knowledge of an underlying sensing mechanism of a latent variable. This method provides an improved classification rate over classical methods on both a synthetic case and a popular face classification dataset. The linear tomographic method is based on non-rotational scanning of baggage moved by a conveyor belt, and can thus be simpler, smaller, and more reliable than existing rotational tomography systems at the expense of more challenging image formation problems that require special model-based methods. The reconstructions for this approach are comparable to existing tomographic systems. Finally our coded aperture extension of existing nuclear resonance fluorescence cargo scanning provides improved observation signal-to-noise ratios. We analyze, discuss, and demonstrate the strengths and challenges of using coded aperture techniques in this application and provide guidance on regimes where these methods can yield gains over conventional methods

Detection of the tulip breaking virus (TBV) in tulips using optical sensors

The tulip breaking virus (TBV) causes severe economic losses for countries that export tulips such as the Netherlands. Infected plants have to be removed from the field as soon as possible. There is an urgent need for a rapid and objective method of screening. In this study, four proximal optical sensing techniques for the detection of TBV in tulip plants were evaluated and compared with a visual assessment by crop experts as well as with an ELISA (enzyme immunoassay) analysis of the same plants. The optical sensor techniques used were an RGB color camera, a spectrophotometer measuring from 350 to 2500 nm, a spectral imaging camera covering a spectral range from 400 to 900 nm and a chlorophyll fluorescence imaging system that measures the photosynthetic activity. Linear discriminant classification was used to compare the results of these optical techniques and the visual assessment with the ELISA score. The spectral imaging system was the best optical technique and its error was only slightly larger than the visual assessment error. The experimental results appear to be promising, and they have led to further research to develop an autonomous robot for the detection and removal of diseased tulip plants in the open field. The application of this robot system will reduce the amount of insecticides and the considerable pressure on labor for selecting diseased plants by the crop expert. © 2010 The Author(s

High speed sCMOS-based oblique plane microscopy applied to the study of calcium dynamics in cardiac myocytes

blique plane microscopy (OPM) is a form of light sheet microscopy that uses a single high numerical aperture microscope objective for both fluorescence excitation and collection. In this paper, measurements of the relative collection efficiency of OPM are presented. An OPM system incorporating two sCMOS cameras is then introduced that enables single isolated cardiac myocytes to be studied continuously for 22 seconds in two dimensions at 667 frames per second with 960 × 200 pixels and for 30 seconds with 960 × 200 × 20 voxels at 25 volumes per second. In both cases OPM is able to record in two spectral channels, enabling intracellular calcium to be studied via the probe Fluo-4 AM simultaneously with the sarcolemma and transverse tubule network via the membrane dye Cellmask Orange. The OPM system was then applied to determine the spatial origin of spontaneous calcium waves for the first time and to measure the cell transverse tubule structure at their point of origin. Further results are presented to demonstrate that the OPM system can also be used to study calcium spark parameters depending on their relationship to the transverse tubule structure

Deep learning approach to Fourier ptychographic microscopy

Convolutional neural networks (CNNs) have gained tremendous success in solving complex inverse problems. The aim of this work is to develop a novel CNN framework to reconstruct video sequences of dynamic live cells captured using a computational microscopy technique, Fourier ptychographic microscopy (FPM). The unique feature of the FPM is its capability to reconstruct images with both wide field-of-view (FOV) and high resolution, i.e. a large space-bandwidth-product (SBP), by taking a series of low resolution intensity images. For live cell imaging, a single FPM frame contains thousands of cell samples with different morphological features. Our idea is to fully exploit the statistical information provided by these large spatial ensembles so as to make predictions in a sequential measurement, without using any additional temporal dataset. Specifically, we show that it is possible to reconstruct high-SBP dynamic cell videos by a CNN trained only on the first FPM dataset captured at the beginning of a time-series experiment. Our CNN approach reconstructs a 12800×10800 pixel phase image using only ∼25 seconds, a 50× speedup compared to the model-based FPM algorithm. In addition, the CNN further reduces the required number of images in each time frame by ∼ 6×. Overall, this significantly improves the imaging throughput by reducing both the acquisition and computational times. The proposed CNN is based on the conditional generative adversarial network (cGAN) framework. We further propose a mixed loss function that combines the standard image domain loss and a weighted Fourier domain loss, which leads to improved reconstruction of the high frequency information. Additionally, we also exploit transfer learning so that our pre-trained CNN can be further optimized to image other cell types. Our technique demonstrates a promising deep learning approach to continuously monitor large live-cell populations over an extended time and gather useful spatial and temporal information with sub-cellular resolution.We would like to thank NVIDIA Corporation for supporting us with the GeForce Titan Xp through the GPU Grant Program. (NVIDIA Corporation; GeForce Titan Xp through the GPU Grant Program)First author draf

Accurate detection of dysmorphic nuclei using dynamic programming and supervised classification

A vast array of pathologies is typified by the presence of nuclei with an abnormal morphology. Dysmorphic nuclear phenotypes feature dramatic size changes or foldings, but also entail much subtler deviations such as nuclear protrusions called blebs. Due to their unpredictable size, shape and intensity, dysmorphic nuclei are often not accurately detected in standard image analysis routines. To enable accurate detection of dysmorphic nuclei in confocal and widefield fluorescence microscopy images, we have developed an automated segmentation algorithm, called Blebbed Nuclei Detector (BleND), which relies on two-pass thresholding for initial nuclear contour detection, and an optimal path finding algorithm, based on dynamic programming, for refining these contours. Using a robust error metric, we show that our method matches manual segmentation in terms of precision and outperforms state-of-the-art nuclear segmentation methods. Its high performance allowed for building and integrating a robust classifier that recognizes dysmorphic nuclei with an accuracy above 95%. The combined segmentation-classification routine is bound to facilitate nucleus-based diagnostics and enable real-time recognition of dysmorphic nuclei in intelligent microscopy workflows