Structured illumination microscopy with unknown patterns and a statistical prior

Structured illumination microscopy (SIM) improves resolution by down-modulating high-frequency information of an object to fit within the passband of the optical system. Generally, the reconstruction process requires prior knowledge of the illumination patterns, which implies a well-calibrated and aberration-free system. Here, we propose a new \textit{algorithmic self-calibration} strategy for SIM that does not need to know the exact patterns {\it a priori}, but only their covariance. The algorithm, termed PE-SIMS, includes a Pattern-Estimation (PE) step requiring the uniformity of the sum of the illumination patterns and a SIM reconstruction procedure using a Statistical prior (SIMS). Additionally, we perform a pixel reassignment process (SIMS-PR) to enhance the reconstruction quality. We achieve 2$\times$ better resolution than a conventional widefield microscope, while remaining insensitive to aberration-induced pattern distortion and robust against parameter tuning

Comparison of Computational Methods Developed to Address Depth-variant Imaging in Fluorescence Microscopy

In three-dimensional fluorescence microscopy, the image formation process is inherently depth variant (DV) due to the refractive index mismatch between imaging layers, which causes depth-induced spherical aberration (SA). In this study, we present a quantitative comparison among different image restoration techniques developed based on a DV imaging model for microscopy in order to assess their ability to correct SA and their impact on restoration. The imaging models approximate DV imaging by either stratifying the object space or image space. For the reconstruction purpose, we used regularized DV algorithms with object stratification method such as the Expectation Maximization (EM), Conjugate Gradient; Principal Component Analysis based expectation maximization (PCA-EM), and Inverse filtering (IF). Reconstructions from simulated data and measured data show that better restoration results are achieved with the DV PCA-EM method than the other DV algorithms in terms of execution time and restoration quality of the image

THREE-DIMENSIONAL (3D) IMAGE FORMATION AND RECONSTRUCTION FOR STRUCTURED ILLUMINATION MICROSCOPY USING A 3D TUNABLE PATTERN

Specific needs in live-cell microscopy necessitate moving fluorescence microscopy toward 3D imaging with enhanced spatial and temporal resolution. Exciting the sample by non-uniform illumination instead of uniform illumination is the main idea of developing techniques to address Abbes diffraction limit in the conventional widefield fluorescence microscopy. In this dissertation, we characterized a novel tunable structured illumination microscopy (SIM) system using a Fresnel biprism illuminated by multiple linear incoherent sources (slits), in which the lateral and axial modulation frequencies of the 3D structured illumination (SI) pattern can be tuned separately. This is a unique feature, which is not the case of the conventional SIM systems. First, in order to take advantage of the tunable-frequency 2D-SIM system (using a single slit), we present a computational approach to reconstruct optical-sectioned images with super-resolution enhancement (OS-SR) by combining data from two lateral modulation frequencies. Moreover, a computational approach to reduce residual fringes evident in the restored images from the Fresnel biprism-based incoherent tunable SIM system is proposed. Second, the 3D SI pattern and the forward imaging model for the tunable-frequency 3D-SIM system (using multiple slits) are verified experimentally and two reconstruction methods have been used to evaluate the achieved OS and SR capabilities. Third, we presented the design of the 3D SI system used in a tunable 3D-SIM setup and discussed its performance in terms of synthetic optical transfer function (OTF). By designing the slit element, we can engineer the frequency response of our 3D-SIM system to always operate at the highest OS and SR performance for a given imaging application. This is the first 3D-SIM setup that enables independent control of the achieved OS and SR capabilities. Finally, we proposed and implemented a new 3D iterative deconvolution approach based on a model that takes into account the axial scanning of the specimen during the data acquisition as in commercial microscopes. The method minimizes the mean squared error using the conjugate gradient descent optimization method. To our knowledge, such a restoration method has not been published to date

Solutions to linear problems in aberrated optical systems

Linear problems are possibly the kindest problems in physics and mathematics. Given sufficient information, the linear equations describing such problems are intrinsically solvable. The solution can be written as a vector having undergone a linear transformation in a vector space; extracting the solution is simply a matter of inverting the transformation. In an ideal optical system, the problem of extracting the object under investigation would be well defined, and the solution trivial to implement. However, real optical systems are all aberrated in some way, and these aberrations obfuscate the information, scrambling it and rendering it inextricable. The process of detangling the object from the aberrated system is no longer a trivial problem or even a uniquely solvable one, and represents one of the great challenges in optics today. This thesis provides a review of the theory behind optical microscopy in the presence of absent information, an architecture for the modern physical and computational methods used to solve the linear inversion problem, and three distinct application spaces of relevance. I hope you find it useful

Superresolution fluorescence microscopy with structured illumination

The resolution of a conventional fluorescence microscope image is diffraction limited which achieves a spatial resolution of 200nm lateral and 500nm axial. Recently, many superresolution fluorescence microscopy techniques have been developed which allow the observation of many biological structures beyond the diffraction limit. Structured illumination microscopy (SIM) is one of them. The principle of SIM is based on using a harmonic light grid which down modulates the high spatial frequencies of the sample into the observable region of the microscope. The resolution enhancement is highly dependent on the reconstruction technique, which restores the high spatial frequencies of the sample to their original position. Common SIM reconstructions require the perfect knowledge of the illumination pattern. However, to perfectly control the harmonic illumination patterns on the sample plane is not easy in experimental implementations and this makes the experimental setup very technical. Reconstructing SIM images assuming the perfect knowledge of the illumination intensity patterns may, therefore, introduce artifacts on the estimated sample due to the misalignment of the grid that can occur during experimental acquisitions. To tackle this drawback of SIM, in this these, we have developed blind-SIM reconstruction strategies which are independent of the illumination patterns. Using the 3D blind-SIM reconstruction strategies we extended the harmonic SIM to speckle illumination microscopy which uses random unknown speckle patterns that need no control, unlike the harmonic grid patterns. For harmonic-SIM images, since incorporating some information about illumination patterns is valuable, we have developed a 3D positive filtered blind-SIM reconstruction which confines the iterative estimation of the illuminations in the vicinity of the Fourier peaks (using carefully designed Fourier filter masks) in the Fourier space. Using blind-SIM reconstruction techniques a lateral resolution of about 100nm and axial resolution of about 200nm is obtained in both speckle and harmonic SIM. In addition, to reduce the out-of-focus problem in widefield images, a simple computational technique which is based on reconstructing 2D data with 3D PSF is developed based on blind-SIM reconstruction. Moreover, to combine the functionalities of SIM and light sheet microscopy, as a proof of concept, we have developed a simple microscope setup which produces a structured light sheet illumination pattern.La microscopie de fluorescence optique est l’un des outils les plus puissants pour étudier les structures cellulaires et moléculaires au niveau subcellulaire. La résolution d’une image de microscope conventionnel à fluorescence est limitée par la diffraction, ce qui permet d’obtenir une résolution spatiale latérale de 200nm et axiale de 500nm. Récemment, de nombreuses techniques de microscopie de fluorescence de super-résolution ont été développées pour permettre d’observer de nombreuses structures biologiques au-delà de la limite de diffraction. La microscopie d’illumination structurée (SIM) est l’une de ces technologies. Le principe de la SIM est basé sur l’utilisation d’une grille de lumière harmonique qui permet de translater les hautes fréquences spatiales de l’échantillon vers la région d’observation du microscope. L’amélioration de la résolution de cette technologie de microscopie dépend fortement de la technique de reconstruction, qui rétablit les hautes fréquences spatiales de l’échantillon dans leur position d’origine. Les méthodes classiques de reconstruction SIM nécessitent une connaissance parfaite de l’illumination de l’échantillon. Cependant, l’implémentation d’un contrôle parfait de l’illumination harmonique sur le plan de l’échantillon n’est pas facile expérimentalement et il présente un grand défi. L’hypothèse de la connaissance parfaite de l’intensité de la lumière illuminant l’échantillon en SIM peut donc introduire des artefacts sur l’image reconstruite de l’échantillon, à cause des erreurs d’alignement de la grille qui peuvent se présenter lors de l’acquisition expérimentale. Afin de surmonter ce défi, nous avons développé dans cette thèse des stratégies de reconstruction «aveugle» qui sont indépendantes de d’illumination. À l’aide de ces stratégies de reconstruction dites «blind-SIM», nous avons étendu la SIM harmonique pour l’appliquer aux cas de «SIM-speckle» qui utilisent des illuminations aléatoires et inconnues qui contrairement à l’illumination harmonique, ne nécessitent pas de controle. Comme il est utile de récupérer des informations sur l’illumination en SIM harmonique, nous avons développé une reconstruction blind-SIM tridimensionnel et filtrée qui confine l’estimation itérative des illuminations au voisinage des pics dans l’espace de Fourier, en utilisant des masques de filtre de Fourier soigneusement conçus. En utilisant des techniques de reconstruction blind-SIM, une résolution latérale d’environ 100 nm et une résolution axiale d’environ 200 nm sont obtenues, à la fois en SIM harmonique et en SIM speckle. En outre, pour réduire le problème de focalisation dans les images de champ large, une technique de calcul simple qui repose sur la reconstruction bidimensionnel de données à partir de PSF tridimensionnel est développée. En outre, afin de combiner à la fois les fonctionnalités de la SIM et de la microscopie á nappe de lumière, en tant que preuve de concept, nous avons développé une configuration de microscope simple qui produit une nappe de lumière structuré

Superresolution fluorescence microscopy with structured illumination

Cotutela Universitat Politècnica de Catalunya i Aix-Marseille UniversitéThe resolution of a conventional fluorescence microscope image is diffraction limited which achieves a spatial resolution of 200nm lateral and 500nm axial. Recently, many superresolution fluorescence microscopy techniques have been developed which allow the observation of many biological structures beyond the diffraction limit. Structured illumination microscopy (SIM) is one of them. The principle of SIM is based on using a harmonic light grid which down modulates the high spatial frequencies of the sample into the observable region of the microscope. The resolution enhancement is highly dependent on the reconstruction technique, which restores the high spatial frequencies of the sample to their original position. Common SIM reconstructions require the perfect knowledge of the illumination pattern. However, to perfectly control the harmonic illumination patterns on the sample plane is not easy in experimental implementations and this makes the experimental setup very technical. Reconstructing SIM images assuming the perfect knowledge of the illumination intensity patterns may, therefore, introduce artifacts on the estimated sample due to the misalignment of the grid that can occur during experimental acquisitions. To tackle this drawback of SIM, in this these, we have developed blind-SIM reconstruction strategies which are independent of the illumination patterns. Using the 3D blind-SIM reconstruction strategies we extended the harmonic SIM to speckle illumination microscopy which uses random unknown speckle patterns that need no control, unlike the harmonic grid patterns. For harmonic-SIM images, since incorporating some information about illumination patterns is valuable, we have developed a 3D positive filtered blind-SIM reconstruction which confines the iterative estimation of the illuminations in the vicinity of the Fourier peaks (using carefully designed Fourier filter masks) in the Fourier space. Using blind-SIM reconstruction techniques a lateral resolution of about 100nm and axial resolution of about 200nm is obtained in both speckle and harmonic SIM. In addition, to reduce the out-of-focus problem in widefield images, a simple computational technique which is based on reconstructing 2D data with 3D PSF is developed based on blind-SIM reconstruction. Moreover, to combine the functionalities of SIM and light sheet microscopy, as a proof of concept, we have developed a simple microscope setup which produces a structured light sheet illumination pattern.La microscopie de fluorescence optique est l’un des outils les plus puissants pour étudier les structures cellulaires et moléculaires au niveau subcellulaire. La résolution d’une image de microscope conventionnel à fluorescence est limitée par la diffraction, ce qui permet d’obtenir une résolution spatiale latérale de 200nm et axiale de 500nm. Récemment, de nombreuses techniques de microscopie de fluorescence de super-résolution ont été développées pour permettre d’observer de nombreuses structures biologiques au-delà de la limite de diffraction. La microscopie d’illumination structurée (SIM) est l’une de ces technologies. Le principe de la SIM est basé sur l’utilisation d’une grille de lumière harmonique qui permet de translater les hautes fréquences spatiales de l’échantillon vers la région d’observation du microscope. L’amélioration de la résolution de cette technologie de microscopie dépend fortement de la technique de reconstruction, qui rétablit les hautes fréquences spatiales de l’échantillon dans leur position d’origine. Les méthodes classiques de reconstruction SIM nécessitent une connaissance parfaite de l’illumination de l’échantillon. Cependant, l’implémentation d’un contrôle parfait de l’illumination harmonique sur le plan de l’échantillon n’est pas facile expérimentalement et il présente un grand défi. L’hypothèse de la connaissance parfaite de l’intensité de la lumière illuminant l’échantillon en SIM peut donc introduire des artefacts sur l’image reconstruite de l’échantillon, à cause des erreurs d’alignement de la grille qui peuvent se présenter lors de l’acquisition expérimentale. Afin de surmonter ce défi, nous avons développé dans cette thèse des stratégies de reconstruction «aveugle» qui sont indépendantes de d’illumination. À l’aide de ces stratégies de reconstruction dites «blind-SIM», nous avons étendu la SIM harmonique pour l’appliquer aux cas de «SIM-speckle» qui utilisent des illuminations aléatoires et inconnues qui contrairement à l’illumination harmonique, ne nécessitent pas de controle. Comme il est utile de récupérer des informations sur l’illumination en SIM harmonique, nous avons développé une reconstruction blind-SIM tridimensionnel et filtrée qui confine l’estimation itérative des illuminations au voisinage des pics dans l’espace de Fourier, en utilisant des masques de filtre de Fourier soigneusement conçus. En utilisant des techniques de reconstruction blind-SIM, une résolution latérale d’environ 100 nm et une résolution axiale d’environ 200 nm sont obtenues, à la fois en SIM harmonique et en SIM speckle. En outre, pour réduire le problème de focalisation dans les images de champ large, une technique de calcul simple qui repose sur la reconstruction bidimensionnel de données à partir de PSF tridimensionnel est développée. En outre, afin de combiner à la fois les fonctionnalités de la SIM et de la microscopie á nappe de lumière, en tant que preuve de concept, nous avons développé une configuration de microscope simple qui produit une nappe de lumière structuréePostprint (published version

Holistic improvement of image acquisition and reconstruction in fluorescence microscopy

Recent developments in microscopic imaging led to a better understanding of intra- and intercellular metabolic processes and, for example, to visualize structural properties of viral pathogens. In this thesis, the imaging process of widefield and confocal scanning microscopy techniques is treated holistically to highlight general strategies and maximise their information content. Poisson or shot noise is assumed to be the fundamental noise process for the given measurements. A stable focus position is a basic condition for e.g. long-term measurements in order to provide reliable information about potential changes inside the Field-of-view. While newer microscopy systems can be equipped with hardware autofocus, this is not yet the widespread standard. For image-based focus analysis, different metrics for ideal, noisy and aberrated, in case of spherical aberration and astigmatism, measurements are presented. A stable focus position is also relevant in the example of 2-photon confocal imaging and at the same time the situation is aggravated in the given example, the measurement of the retina in the living mouse. In addition to the natural drift of the focal position, which can be evaluated by means of previously introduced metrics, rhythmic heartbeat, respiration, unrhythmic muscle twitching and movement of the mouse kept in artificial sleep are added. A dejittering algorithm is presented for the measurement data obtained under these circumstances. Using the additional information about the sample distribution in ISM, a method for reconstructing 3D from 2D image data is presented in the form of thick slice unmixing. This method can further be used for suppression of light generated outside the focal layer of 3D data stacks and is compared to selective layer multi-view deconvolution. To reduce phototoxicity and save valuable measurement time for a 3D stack, the method of zLEAP is presented, by which omitted Z-planes are subsequently calculated and inserted