Structured illumination microscopy with unknown patterns and a statistical prior

Structured illumination microscopy (SIM) improves resolution by down-modulating high-frequency information of an object to fit within the passband of the optical system. Generally, the reconstruction process requires prior knowledge of the illumination patterns, which implies a well-calibrated and aberration-free system. Here, we propose a new \textit{algorithmic self-calibration} strategy for SIM that does not need to know the exact patterns {\it a priori}, but only their covariance. The algorithm, termed PE-SIMS, includes a Pattern-Estimation (PE) step requiring the uniformity of the sum of the illumination patterns and a SIM reconstruction procedure using a Statistical prior (SIMS). Additionally, we perform a pixel reassignment process (SIMS-PR) to enhance the reconstruction quality. We achieve 2$\times$ better resolution than a conventional widefield microscope, while remaining insensitive to aberration-induced pattern distortion and robust against parameter tuning

Superresolution fluorescence microscopy with structured illumination

Cotutela Universitat Politècnica de Catalunya i Aix-Marseille UniversitéThe resolution of a conventional fluorescence microscope image is diffraction limited which achieves a spatial resolution of 200nm lateral and 500nm axial. Recently, many superresolution fluorescence microscopy techniques have been developed which allow the observation of many biological structures beyond the diffraction limit. Structured illumination microscopy (SIM) is one of them. The principle of SIM is based on using a harmonic light grid which down modulates the high spatial frequencies of the sample into the observable region of the microscope. The resolution enhancement is highly dependent on the reconstruction technique, which restores the high spatial frequencies of the sample to their original position. Common SIM reconstructions require the perfect knowledge of the illumination pattern. However, to perfectly control the harmonic illumination patterns on the sample plane is not easy in experimental implementations and this makes the experimental setup very technical. Reconstructing SIM images assuming the perfect knowledge of the illumination intensity patterns may, therefore, introduce artifacts on the estimated sample due to the misalignment of the grid that can occur during experimental acquisitions. To tackle this drawback of SIM, in this these, we have developed blind-SIM reconstruction strategies which are independent of the illumination patterns. Using the 3D blind-SIM reconstruction strategies we extended the harmonic SIM to speckle illumination microscopy which uses random unknown speckle patterns that need no control, unlike the harmonic grid patterns. For harmonic-SIM images, since incorporating some information about illumination patterns is valuable, we have developed a 3D positive filtered blind-SIM reconstruction which confines the iterative estimation of the illuminations in the vicinity of the Fourier peaks (using carefully designed Fourier filter masks) in the Fourier space. Using blind-SIM reconstruction techniques a lateral resolution of about 100nm and axial resolution of about 200nm is obtained in both speckle and harmonic SIM. In addition, to reduce the out-of-focus problem in widefield images, a simple computational technique which is based on reconstructing 2D data with 3D PSF is developed based on blind-SIM reconstruction. Moreover, to combine the functionalities of SIM and light sheet microscopy, as a proof of concept, we have developed a simple microscope setup which produces a structured light sheet illumination pattern.La microscopie de fluorescence optique est l’un des outils les plus puissants pour étudier les structures cellulaires et moléculaires au niveau subcellulaire. La résolution d’une image de microscope conventionnel à fluorescence est limitée par la diffraction, ce qui permet d’obtenir une résolution spatiale latérale de 200nm et axiale de 500nm. Récemment, de nombreuses techniques de microscopie de fluorescence de super-résolution ont été développées pour permettre d’observer de nombreuses structures biologiques au-delà de la limite de diffraction. La microscopie d’illumination structurée (SIM) est l’une de ces technologies. Le principe de la SIM est basé sur l’utilisation d’une grille de lumière harmonique qui permet de translater les hautes fréquences spatiales de l’échantillon vers la région d’observation du microscope. L’amélioration de la résolution de cette technologie de microscopie dépend fortement de la technique de reconstruction, qui rétablit les hautes fréquences spatiales de l’échantillon dans leur position d’origine. Les méthodes classiques de reconstruction SIM nécessitent une connaissance parfaite de l’illumination de l’échantillon. Cependant, l’implémentation d’un contrôle parfait de l’illumination harmonique sur le plan de l’échantillon n’est pas facile expérimentalement et il présente un grand défi. L’hypothèse de la connaissance parfaite de l’intensité de la lumière illuminant l’échantillon en SIM peut donc introduire des artefacts sur l’image reconstruite de l’échantillon, à cause des erreurs d’alignement de la grille qui peuvent se présenter lors de l’acquisition expérimentale. Afin de surmonter ce défi, nous avons développé dans cette thèse des stratégies de reconstruction «aveugle» qui sont indépendantes de d’illumination. À l’aide de ces stratégies de reconstruction dites «blind-SIM», nous avons étendu la SIM harmonique pour l’appliquer aux cas de «SIM-speckle» qui utilisent des illuminations aléatoires et inconnues qui contrairement à l’illumination harmonique, ne nécessitent pas de controle. Comme il est utile de récupérer des informations sur l’illumination en SIM harmonique, nous avons développé une reconstruction blind-SIM tridimensionnel et filtrée qui confine l’estimation itérative des illuminations au voisinage des pics dans l’espace de Fourier, en utilisant des masques de filtre de Fourier soigneusement conçus. En utilisant des techniques de reconstruction blind-SIM, une résolution latérale d’environ 100 nm et une résolution axiale d’environ 200 nm sont obtenues, à la fois en SIM harmonique et en SIM speckle. En outre, pour réduire le problème de focalisation dans les images de champ large, une technique de calcul simple qui repose sur la reconstruction bidimensionnel de données à partir de PSF tridimensionnel est développée. En outre, afin de combiner à la fois les fonctionnalités de la SIM et de la microscopie á nappe de lumière, en tant que preuve de concept, nous avons développé une configuration de microscope simple qui produit une nappe de lumière structuréePostprint (published version

Superresolution fluorescence microscopy with structured illumination

The resolution of a conventional fluorescence microscope image is diffraction limited which achieves a spatial resolution of 200nm lateral and 500nm axial. Recently, many superresolution fluorescence microscopy techniques have been developed which allow the observation of many biological structures beyond the diffraction limit. Structured illumination microscopy (SIM) is one of them. The principle of SIM is based on using a harmonic light grid which down modulates the high spatial frequencies of the sample into the observable region of the microscope. The resolution enhancement is highly dependent on the reconstruction technique, which restores the high spatial frequencies of the sample to their original position. Common SIM reconstructions require the perfect knowledge of the illumination pattern. However, to perfectly control the harmonic illumination patterns on the sample plane is not easy in experimental implementations and this makes the experimental setup very technical. Reconstructing SIM images assuming the perfect knowledge of the illumination intensity patterns may, therefore, introduce artifacts on the estimated sample due to the misalignment of the grid that can occur during experimental acquisitions. To tackle this drawback of SIM, in this these, we have developed blind-SIM reconstruction strategies which are independent of the illumination patterns. Using the 3D blind-SIM reconstruction strategies we extended the harmonic SIM to speckle illumination microscopy which uses random unknown speckle patterns that need no control, unlike the harmonic grid patterns. For harmonic-SIM images, since incorporating some information about illumination patterns is valuable, we have developed a 3D positive filtered blind-SIM reconstruction which confines the iterative estimation of the illuminations in the vicinity of the Fourier peaks (using carefully designed Fourier filter masks) in the Fourier space. Using blind-SIM reconstruction techniques a lateral resolution of about 100nm and axial resolution of about 200nm is obtained in both speckle and harmonic SIM. In addition, to reduce the out-of-focus problem in widefield images, a simple computational technique which is based on reconstructing 2D data with 3D PSF is developed based on blind-SIM reconstruction. Moreover, to combine the functionalities of SIM and light sheet microscopy, as a proof of concept, we have developed a simple microscope setup which produces a structured light sheet illumination pattern.La microscopie de fluorescence optique est l’un des outils les plus puissants pour étudier les structures cellulaires et moléculaires au niveau subcellulaire. La résolution d’une image de microscope conventionnel à fluorescence est limitée par la diffraction, ce qui permet d’obtenir une résolution spatiale latérale de 200nm et axiale de 500nm. Récemment, de nombreuses techniques de microscopie de fluorescence de super-résolution ont été développées pour permettre d’observer de nombreuses structures biologiques au-delà de la limite de diffraction. La microscopie d’illumination structurée (SIM) est l’une de ces technologies. Le principe de la SIM est basé sur l’utilisation d’une grille de lumière harmonique qui permet de translater les hautes fréquences spatiales de l’échantillon vers la région d’observation du microscope. L’amélioration de la résolution de cette technologie de microscopie dépend fortement de la technique de reconstruction, qui rétablit les hautes fréquences spatiales de l’échantillon dans leur position d’origine. Les méthodes classiques de reconstruction SIM nécessitent une connaissance parfaite de l’illumination de l’échantillon. Cependant, l’implémentation d’un contrôle parfait de l’illumination harmonique sur le plan de l’échantillon n’est pas facile expérimentalement et il présente un grand défi. L’hypothèse de la connaissance parfaite de l’intensité de la lumière illuminant l’échantillon en SIM peut donc introduire des artefacts sur l’image reconstruite de l’échantillon, à cause des erreurs d’alignement de la grille qui peuvent se présenter lors de l’acquisition expérimentale. Afin de surmonter ce défi, nous avons développé dans cette thèse des stratégies de reconstruction «aveugle» qui sont indépendantes de d’illumination. À l’aide de ces stratégies de reconstruction dites «blind-SIM», nous avons étendu la SIM harmonique pour l’appliquer aux cas de «SIM-speckle» qui utilisent des illuminations aléatoires et inconnues qui contrairement à l’illumination harmonique, ne nécessitent pas de controle. Comme il est utile de récupérer des informations sur l’illumination en SIM harmonique, nous avons développé une reconstruction blind-SIM tridimensionnel et filtrée qui confine l’estimation itérative des illuminations au voisinage des pics dans l’espace de Fourier, en utilisant des masques de filtre de Fourier soigneusement conçus. En utilisant des techniques de reconstruction blind-SIM, une résolution latérale d’environ 100 nm et une résolution axiale d’environ 200 nm sont obtenues, à la fois en SIM harmonique et en SIM speckle. En outre, pour réduire le problème de focalisation dans les images de champ large, une technique de calcul simple qui repose sur la reconstruction bidimensionnel de données à partir de PSF tridimensionnel est développée. En outre, afin de combiner à la fois les fonctionnalités de la SIM et de la microscopie á nappe de lumière, en tant que preuve de concept, nous avons développé une configuration de microscope simple qui produit une nappe de lumière structuré

Super-resolution photoacoustic imaging via flow induced absorption fluctuations

In deep tissue photoacoustic imaging the spatial resolution is inherently limited by the acoustic wavelength. We present an approach for surpassing the acoustic diffraction limit by exploiting temporal fluctuations in the sample absorption distribution, such as those induced by flowing particles. In addition to enhanced resolution, our approach inherently provides background reduction, and can be implemented with any conventional photoacoustic imaging system. The considerable resolution increase is made possible by adapting notions from super-resolution optical fluctuations imaging (SOFI) developed for blinking fluorescent molecules, to flowing acoustic emitters. By generalizing SOFI mathematical analysis to complex valued signals, we demonstrate super-resolved photoacoustic images that are free from oscillations caused by band-limited detection. The presented technique holds potential for contrast-agent free micro-vessels imaging, as red blood cells provide a strong endogenous source of naturally fluctuating absorption

Development of Microscopy Systems for Super-Resolution, Whole-Slide, Hyperspectral, and Confocal Imaging

Optical microscope is an important tool for researchers to study small objects. In this thesis, we will focus on the improvement of traditional microscope systems from several aspects including resolution, field of view, speed, cost, compactness, multimodality. In particular, we will investigate computational imaging methods that bypass the limitations with traditional microscope systems by combining the optical hardware design and image processing algorithm. Examples will include optimizing illumination strategy for the Fourier ptychography (FP), developing field-portable high-resolution microscope using a cellphone lens, investigating pattern-illuminated FP for fluorescence microscopy, demonstrating multimodal microscopic imaging with the use of liquid crystal display, achieving fast and accurate autofocusing for whole slide imaging system

Advances in super-resolution photoacoustic imaging

Photoacoustic (PA) imaging (PAI), or optoacoustic imaging, is a hybrid imaging modality that combines optical absorption contrast and ultrasound image formation. In PAI, the target is excited by a short laser pulse and subsequently absorbs the photon energy, leading to a transient local temperature rise. The temperature rise induces a local pressure rise that propagates as acoustic waves. As acoustic waves generally undergo less scattering and attenuation in tissue compared with light, PAI can provide high-resolution images in both the optical (quasi)ballistic and (quasi)diffusive regimes (1,2). Based on the image formation methods, PAI can be classified into two categories: photoacoustic microscopy (PAM) and photoacoustic computed tomography (PACT). PAM uses a focused excitation light beam and/or a focused single-element ultrasonic transducer for direct image formation through position scanning (1,2). PAM has a maximum imaging depth ranging from a few hundred micrometers to a few millimeters with spatial resolution ranging from sub-micrometer to sub-millimeter (2,3). PAM can be further classified into optical-resolution PAM (OR-PAM) and acoustic-resolution PAM (AR-PAM). For both OR-PAM and AR-PAM, the axial resolution is determined by the bandwidth of the ultrasonic transducer (4). OR-PAM works in the optical (quasi)ballistic regime, whereas the light is tightly focused that it can penetrate about one optical transport mean free path (~1 mm in soft tissue). Therefore, the lateral resolution of OR-PAM is mainly determined by the optical focal spot size (4-6). The optical focusing is diffraction-limited as λ/2NA, where λ is the light wavelength, and NA is the numerical aperture of objective lens. On the contrary, in AR-PAM, the laser is loosely focused to fulfill the entire acoustic focal spot, thereby penetrating a few optical transport mean free paths, i.e., in the quasi-diffusive regime. The lateral resolution of AR-PAM is thus determined by the size of acoustic focus (4,7,8), limited by acoustic diffraction. In PACT, the object is illuminated with a wide-field laser beam in the diffusive regime, and the generated acoustic waves are detected at multiple locations or by using a multi-element transducer array. The image formed by PACT is reconstructed by an inverse algorithm. The spatial resolution of PACT is fundamentally limited by acoustic diffraction, and additionally affected by the directionality and spacing of the detector elements (9). Recently, several studies have shown that sub-diffraction imaging of biological samples can be achieved through PAI by breaking optical-diffraction limit in the (quasi)ballistic regime or acoustic-diffraction limit in the (quasi)diffusive regime, which have opened new possibilities for fundamental biological studies. Yao et al. developed a photoimprint PAM using the intensity-dependent photobleaching effect and acquired a melanoma cell PA image with a lateral resolution of 90 nm (10). Danielli et al. reported a label-free PA nanoscopy based on the optical-absorption saturation effect and acquired a mitochondria PA image with a lateral resolution of 88 nm (11). Chaigne et al. exploited the sample-dynamics-induced inherent temporal fluctuation in the PA signals and achieved a resolution enhancement of about 1.4 over conventional PACT (12). Murray et al. broke the acoustic diffraction limit by implementing a blind speckle illumination and block-FISTA reconstruction algorithm and achieved a resolution close to the acoustic speckle size (13). Dean-Ben et al. also overcame the acoustic diffraction limit by incorporating rapid sequential acquisition of 3D PA images of flowing absorbing particles and further enhanced the visibility of structures under limited-view tomographic conditions (14). Conkey et al. optimized wavefront shaping with photoacoustic feedback and achieved up to ten times improvement in signal-to-noise ratio and five to six times sub-acoustic-diffraction resolution (15). In this concise review, we summarize and analyze the recent development in super-resolution (SR) PAI (SR-PAI) in both the optical (quasi)ballistic and (quasi)diffusive regime, as well as their representative applications. We also discuss the current challenges in SR-PAI and envision the potential breakthroughs

Minimally-Invasive Lens-Free Computational Microendoscopy

Ultra-miniaturized imaging tools are vital for numerous biomedical applications. Such minimally-invasive imagers allow for navigation into hard-to-reach regions and, for example, observation of deep brain activity in freely moving animals with minimal ancillary tissue damage. Conventional solutions employ distal microlenses. However, as lenses become smaller and thus less invasive they develop greater optical aberrations, requiring bulkier compound designs with restricted field-of-view. In addition, tools capable of 3-dimensional volumetric imaging require components that physically scan the focal plane, which ultimately increases the distal complexity, footprint, and weight. Simply put, minimally-invasive imaging systems have limited information capacity due to their given cross-sectional area. This thesis explores minimally-invasive lens-free microendoscopy enabled by a successful integration of signal processing, optical hardware, and image reconstruction algorithms. Several computational microendoscopy architectures that simultaneously achieve miniaturization and high information content are presented. Leveraging the computational imaging techniques enables color-resolved imaging with wide field-of-view, and 3-dimensional volumetric reconstruction of an unknown scene using a single camera frame without any actuated parts, further advancing the performance versus invasiveness of microendoscopy

Advances in super-resolution photoacoustic imaging

Photoacoustic (PA) imaging (PAI), or optoacoustic imaging, is a hybrid imaging modality that combines optical absorption contrast and ultrasound image formation. In PAI, the target is excited by a short laser pulse and subsequently absorbs the photon energy, leading to a transient local temperature rise. The temperature rise induces a local pressure rise that propagates as acoustic waves. As acoustic waves generally undergo less scattering and attenuation in tissue compared with light, PAI can provide high-resolution images in both the optical (quasi)ballistic and (quasi)diffusive regimes (1,2). Based on the image formation methods, PAI can be classified into two categories: photoacoustic microscopy (PAM) and photoacoustic computed tomography (PACT). PAM uses a focused excitation light beam and/or a focused single-element ultrasonic transducer for direct image formation through position scanning (1,2). PAM has a maximum imaging depth ranging from a few hundred micrometers to a few millimeters with spatial resolution ranging from sub-micrometer to sub-millimeter (2,3). PAM can be further classified into optical-resolution PAM (OR-PAM) and acoustic-resolution PAM (AR-PAM). For both OR-PAM and AR-PAM, the axial resolution is determined by the bandwidth of the ultrasonic transducer (4). OR-PAM works in the optical (quasi)ballistic regime, whereas the light is tightly focused that it can penetrate about one optical transport mean free path (~1 mm in soft tissue). Therefore, the lateral resolution of OR-PAM is mainly determined by the optical focal spot size (4-6). The optical focusing is diffraction-limited as λ/2NA, where λ is the light wavelength, and NA is the numerical aperture of objective lens. On the contrary, in AR-PAM, the laser is loosely focused to fulfill the entire acoustic focal spot, thereby penetrating a few optical transport mean free paths, i.e., in the quasi-diffusive regime. The lateral resolution of AR-PAM is thus determined by the size of acoustic focus (4,7,8), limited by acoustic diffraction. In PACT, the object is illuminated with a wide-field laser beam in the diffusive regime, and the generated acoustic waves are detected at multiple locations or by using a multi-element transducer array. The image formed by PACT is reconstructed by an inverse algorithm. The spatial resolution of PACT is fundamentally limited by acoustic diffraction, and additionally affected by the directionality and spacing of the detector elements (9). Recently, several studies have shown that sub-diffraction imaging of biological samples can be achieved through PAI by breaking optical-diffraction limit in the (quasi)ballistic regime or acoustic-diffraction limit in the (quasi)diffusive regime, which have opened new possibilities for fundamental biological studies. Yao et al. developed a photoimprint PAM using the intensity-dependent photobleaching effect and acquired a melanoma cell PA image with a lateral resolution of 90 nm (10). Danielli et al. reported a label-free PA nanoscopy based on the optical-absorption saturation effect and acquired a mitochondria PA image with a lateral resolution of 88 nm (11). Chaigne et al. exploited the sample-dynamics-induced inherent temporal fluctuation in the PA signals and achieved a resolution enhancement of about 1.4 over conventional PACT (12). Murray et al. broke the acoustic diffraction limit by implementing a blind speckle illumination and block-FISTA reconstruction algorithm and achieved a resolution close to the acoustic speckle size (13). Dean-Ben et al. also overcame the acoustic diffraction limit by incorporating rapid sequential acquisition of 3D PA images of flowing absorbing particles and further enhanced the visibility of structures under limited-view tomographic conditions (14). Conkey et al. optimized wavefront shaping with photoacoustic feedback and achieved up to ten times improvement in signal-to-noise ratio and five to six times sub-acoustic-diffraction resolution (15). In this concise review, we summarize and analyze the recent development in super-resolution (SR) PAI (SR-PAI) in both the optical (quasi)ballistic and (quasi)diffusive regime, as well as their representative applications. We also discuss the current challenges in SR-PAI and envision the potential breakthroughs

Fourier ptychography: current applications and future promises

Traditional imaging systems exhibit a well-known trade-off between the resolution and the field of view of their captured images. Typical cameras and microscopes can either “zoom in” and image at high-resolution, or they can “zoom out” to see a larger area at lower resolution, but can rarely achieve both effects simultaneously. In this review, we present details about a relatively new procedure termed Fourier ptychography (FP), which addresses the above trade-off to produce gigapixel-scale images without requiring any moving parts. To accomplish this, FP captures multiple low-resolution, large field-of-view images and computationally combines them in the Fourier domain into a high-resolution, large field-of-view result. Here, we present details about the various implementations of FP and highlight its demonstrated advantages to date, such as aberration recovery, phase imaging, and 3D tomographic reconstruction, to name a few. After providing some basics about FP, we list important details for successful experimental implementation, discuss its relationship with other computational imaging techniques, and point to the latest advances in the field while highlighting persisting challenges