Deep learning approach to scalable imaging through scattering media

We propose a deep learning technique to exploit “deep speckle correlations”. Our work paves the way to a highly scalable deep learning approach for imaging through scattering media.Published versio

Deep speckle correlation: a deep learning approach toward scalable imaging through scattering media

Imaging through scattering is an important yet challenging problem. Tremendous progress has been made by exploiting the deterministic input–output “transmission matrix” for a fixed medium. However, this “one-to-one” mapping is highly susceptible to speckle decorrelations – small perturbations to the scattering medium lead to model errors and severe degradation of the imaging performance. Our goal here is to develop a new framework that is highly scalable to both medium perturbations and measurement requirement. To do so, we propose a statistical “one-to-all” deep learning (DL) technique that encapsulates a wide range of statistical variations for the model to be resilient to speckle decorrelations. Specifically, we develop a convolutional neural network (CNN) that is able to learn the statistical information contained in the speckle intensity patterns captured on a set of diffusers having the same macroscopic parameter. We then show for the first time, to the best of our knowledge, that the trained CNN is able to generalize and make high-quality object predictions through an entirely different set of diffusers of the same class. Our work paves the way to a highly scalable DL approach for imaging through scattering media.National Science Foundation (NSF) (1711156); Directorate for Engineering (ENG). (1711156 - National Science Foundation (NSF); Directorate for Engineering (ENG))First author draf

Illumination coding meets uncertainty learning: toward reliable AI-augmented phase imaging

We propose a physics-assisted deep learning (DL) framework for large space-bandwidth product (SBP) phase imaging. We design an asymmetric coded illumination scheme to encode high-resolution phase information across a wide field-of-view. We then develop a matching DL algorithm to provide large-SBP phase estimation. We show that this illumination coding scheme is highly scalable in achieving flexible resolution, and robust to experimental variations. We demonstrate this technique on both static and dynamic biological samples, and show that it can reliably achieve 5X resolution enhancement across 4X FOVs using only five multiplexed measurements -- more than 10X data reduction over the state-of-the-art. Typical DL algorithms tend to provide over-confident predictions, whose errors are only discovered in hindsight. We develop an uncertainty learning framework to overcome this limitation and provide predictive assessment to the reliability of the DL prediction. We show that the predicted uncertainty maps can be used as a surrogate to the true error. We validate the robustness of our technique by analyzing the model uncertainty. We quantify the effect of noise, model errors, incomplete training data, and "out-of-distribution" testing data by assessing the data uncertainty. We further demonstrate that the predicted credibility maps allow identifying spatially and temporally rare biological events. Our technique enables scalable AI-augmented large-SBP phase imaging with dependable predictions.Published versio

Deep learning approach to Fourier ptychographic microscopy

Convolutional neural networks (CNNs) have gained tremendous success in solving complex inverse problems. The aim of this work is to develop a novel CNN framework to reconstruct video sequences of dynamic live cells captured using a computational microscopy technique, Fourier ptychographic microscopy (FPM). The unique feature of the FPM is its capability to reconstruct images with both wide field-of-view (FOV) and high resolution, i.e. a large space-bandwidth-product (SBP), by taking a series of low resolution intensity images. For live cell imaging, a single FPM frame contains thousands of cell samples with different morphological features. Our idea is to fully exploit the statistical information provided by these large spatial ensembles so as to make predictions in a sequential measurement, without using any additional temporal dataset. Specifically, we show that it is possible to reconstruct high-SBP dynamic cell videos by a CNN trained only on the first FPM dataset captured at the beginning of a time-series experiment. Our CNN approach reconstructs a 12800×10800 pixel phase image using only ∼25 seconds, a 50× speedup compared to the model-based FPM algorithm. In addition, the CNN further reduces the required number of images in each time frame by ∼ 6×. Overall, this significantly improves the imaging throughput by reducing both the acquisition and computational times. The proposed CNN is based on the conditional generative adversarial network (cGAN) framework. We further propose a mixed loss function that combines the standard image domain loss and a weighted Fourier domain loss, which leads to improved reconstruction of the high frequency information. Additionally, we also exploit transfer learning so that our pre-trained CNN can be further optimized to image other cell types. Our technique demonstrates a promising deep learning approach to continuously monitor large live-cell populations over an extended time and gather useful spatial and temporal information with sub-cellular resolution.We would like to thank NVIDIA Corporation for supporting us with the GeForce Titan Xp through the GPU Grant Program. (NVIDIA Corporation; GeForce Titan Xp through the GPU Grant Program)First author draf

Deep learning approach to Fourier ptychographic microscopy

Convolutional neural networks (CNNs) have gained tremendous success in solving complex inverse problems. The aim of this work is to develop a novel CNN framework to reconstruct video sequence of dynamic live cells captured using a computational microscopy technique, Fourier ptychographic microscopy (FPM). The unique feature of the FPM is its capability to reconstruct images with both wide field-of-view (FOV) and high resolution, i.e. a large space-bandwidth-product (SBP), by taking a series of low resolution intensity images. For live cell imaging, a single FPM frame contains thousands of cell samples with different morphological features. Our idea is to fully exploit the statistical information provided by this large spatial ensemble so as to make predictions in a sequential measurement, without using any additional temporal dataset. Specifically, we show that it is possible to reconstruct high-SBP dynamic cell videos by a CNN trained only on the first FPM dataset captured at the beginning of a time-series experiment. Our CNN approach reconstructs a 12800X10800 pixels phase image using only ~25 seconds, a 50X speedup compared to the model-based FPM algorithm. In addition, the CNN further reduces the required number of images in each time frame by ~6X. Overall, this significantly improves the imaging throughput by reducing both the acquisition and computational times. The proposed CNN is based on the conditional generative adversarial network (cGAN) framework. Additionally, we also exploit transfer learning so that our pre-trained CNN can be further optimized to image other cell types. Our technique demonstrates a promising deep learning approach to continuously monitor large live-cell populations over an extended time and gather useful spatial and temporal information with sub-cellular resolution

A deep-learning approach for high-speed Fourier ptychographic microscopy

We demonstrate a new convolutional neural network architecture to perform Fourier ptychographic Microscopy (FPM) reconstruction, which achieves high-resolution phase recovery with considerably less data than standard FPM.https://www.researchgate.net/profile/Thanh_Nguyen68/publication/325829575_A_deep-learning_approach_for_high-speed_Fourier_ptychographic_microscopy/links/5b2beec20f7e9b0df5ba4872/A-deep-learning-approach-for-high-speed-Fourier-ptychographic-microscopy.pdfhttps://www.researchgate.net/profile/Thanh_Nguyen68/publication/325829575_A_deep-learning_approach_for_high-speed_Fourier_ptychographic_microscopy/links/5b2beec20f7e9b0df5ba4872/A-deep-learning-approach-for-high-speed-Fourier-ptychographic-microscopy.pdfhttps://www.researchgate.net/profile/Thanh_Nguyen68/publication/325829575_A_deep-learning_approach_for_high-speed_Fourier_ptychographic_microscopy/links/5b2beec20f7e9b0df5ba4872/A-deep-learning-approach-for-high-speed-Fourier-ptychographic-microscopy.pdfhttps://www.researchgate.net/profile/Thanh_Nguyen68/publication/325829575_A_deep-learning_approach_for_high-speed_Fourier_ptychographic_microscopy/links/5b2beec20f7e9b0df5ba4872/A-deep-learning-approach-for-high-speed-Fourier-ptychographic-microscopy.pdfhttps://www.researchgate.net/profile/Thanh_Nguyen68/publication/325829575_A_deep-learning_approach_for_high-speed_Fourier_ptychographic_microscopy/links/5b2beec20f7e9b0df5ba4872/A-deep-learning-approach-for-high-speed-Fourier-ptychographic-microscopy.pdfhttps://www.researchgate.net/profile/Thanh_Nguyen68/publication/325829575_A_deep-learning_approach_for_high-speed_Fourier_ptychographic_microscopy/links/5b2beec20f7e9b0df5ba4872/A-deep-learning-approach-for-high-speed-Fourier-ptychographic-microscopy.pdfPublished versio

Scalable and reliable deep learning for computational microscopy in complex media

Emerging deep learning based computational microscopy techniques promise novel imaging capabilities beyond traditional techniques. In this talk, I will discuss two microscopy applications. First, high space-bandwidth product microscopy typically requires a large number of measurements. I will present a novel physics-assisted deep learning (DL) framework for large space-bandwidth product (SBP) phase imaging [1], enabling significant reduction of the required measurements, opening up real-time applications. In this technique, we design asymmetric coded illumination patterns to encode high-resolution phase information across a wide field-of-view. We then develop a matching DL algorithm to provide large-SBP phase estimation. We demonstrate this technique on both static and dynamic biological samples, and show that it can reliably achieve 5× resolution enhancement across 4× FOVs using only five multiplexed measurements. In addition, we develop an uncertainty learning framework to provide predictive assessment to the reliability of the DL prediction. We show that the predicted uncertainty maps can be used as a surrogate to the true error. We validate the robustness of our technique by analyzing the model uncertainty. We quantify the effect of noise, model errors, incomplete training data, and “out-of-distribution” testing data by assessing the data uncertainty. We further demonstrate that the predicted credibility maps allow identifying spatially and temporally rare biological events. Our technique enables scalable DL-augmented large-SBP phase imaging with reliable predictions and uncertainty quantifications. Second, I will turn to the pervasive problem of imaging in scattering media. I will discuss a new deep learning- based technique that is highly generalizable and resilient to statistical variations of the scattering media [2]. We develop a statistical ‘one-to-all’ deep learning technique that encapsulates a wide range of statistical variations for the model to be resilient to speckle decorrelations. Specifically, we develop a convolutional neural network (CNN) that is able to learn the statistical information contained in the speckle intensity patterns captured on a set of diffusers having the same macroscopic parameter. We then show that the trained CNN is able to generalize and make high-quality object predictions through an entirely different set of diffusers of the same class. Our work paves the way to a highly scalable deep learning approach for imaging through scattering media. REFERENCES [1] Xue, Y., Cheng, S., Li, Y., and Tian, L., “Illumination coding meets uncertainty learning: toward reliable ai-augmented phase imaging,” arXiv:1901.02038 (2019). [2] Li, Y., Xue, Y., and Tian, L., “Deep speckle correlation: a deep learning approach toward scalable imaging through scattering media,” Optica 5, 1181 (2018)