Single-shot self-supervised object detection in microscopy

Object detection is a fundamental task in digital microscopy, where machine learning has made great strides in overcoming the limitations of classical approaches. The training of state-of-the-art machine-learning methods almost universally relies on vast amounts of labeled experimental data or the ability to numerically simulate realistic datasets. However, experimental data are often challenging to label and cannot be easily reproduced numerically. Here, we propose a deep-learning method, named LodeSTAR (Localization and detection from Symmetries, Translations And Rotations), that learns to detect microscopic objects with sub-pixel accuracy from a single unlabeled experimental image by exploiting the inherent roto-translational symmetries of this task. We demonstrate that LodeSTAR outperforms traditional methods in terms of accuracy, also when analyzing challenging experimental data containing densely packed cells or noisy backgrounds. Furthermore, by exploiting additional symmetries we show that LodeSTAR can measure other properties, e.g., vertical position and polarizability in holographic microscopy

Computer Vision Approaches to Liquid-Phase Transmission Electron Microscopy

Electron microscopy (EM) is a technique that exploits the interaction between electron and matter to produce high resolution images down to atomic level. In order to avoid undesired scattering in the electron path, EM samples are conventionally imaged in solid state under vacuum conditions. Recently, this limit has been overcome by the realization of liquid-phase electron microscopy (LP EM), a technique that enables the analysis of samples in their liquid native state. LP EM paired with a high frame rate acquisition direct detection camera allows tracking the motion of particles in liquids, as well as their temporal dynamic processes. In this research work, LP EM is adopted to image the dynamics of particles undergoing Brownian motion, exploiting their natural rotation to access all the particle views, in order to reconstruct their 3D structure via tomographic techniques. However, specific computer vision-based tools were designed around the limitations of LP EM in order to elaborate the results of the imaging process. Consequently, different deblurring and denoising approaches were adopted to improve the quality of the images. Therefore, the processed LP EM images were adopted to reconstruct the 3D model of the imaged samples. This task was performed by developing two different methods: Brownian tomography (BT) and Brownian particle analysis (BPA). The former tracks in time a single particle, capturing its dynamics evolution over time. The latter is an extension in time of the single particle analysis (SPA) technique. Conventionally it is paired to cryo-EM to reconstruct 3D density maps starting from thousands of EM images by capturing hundreds of particles of the same species frozen on a grid. On the contrary, BPA has the ability to process image sequences that may not contain thousands of particles, but instead monitors individual particle views across consecutive frames, rather than across a single frame

Fast and Accurate Nanoparticle Characterization Using Deep-Learning-Enhanced Off-Axis Holography

Characterization of suspended nanoparticles in their native environment plays a central role in a wide range of fields, from medical diagnostics and nanoparticleenhanced drug delivery to nanosafety and environmental nanopollution assessment. Standard optical approaches for nanoparticle sizing assess the size via the diffusion constant and, as a consequence, require long trajectories and that the medium has a known and uniform viscosity. However, in most biological applications, only short trajectories are available, while simultaneously, the medium viscosity is unknown and tends to display spatiotemporal variations. In this work, we demonstrate a label-free method to quantify not only size but also refractive index of individual subwavelength particles using 2 orders of magnitude shorter trajectories than required by standard methods and without prior knowledge about the physicochemical properties of the medium. We achieved this by developing a weighted average convolutional neural network to analyze holographic images of single particles, which was successfully applied to distinguish and quantify both size and refractive index of subwavelength silica andpolystyrene particles without prior knowledge of solute viscosity or refractive index. We further demonstrate how these features make it possible to temporally resolve aggregation dynamics of 31 nm polystyrene nanoparticles, revealing previously unobserved time-resolved dynamics of the monomer number and fractal dimension of individual subwavelength aggregates

Pushing the Boundaries of Biomolecule Characterization through Deep Learning

The importance of studying biological molecules in living organisms can hardly be overstated as they regulate crucial processes in living matter of all kinds.Their ubiquitous nature makes them relevant for disease diagnosis, drug development, and for our fundamental understanding of the complex systems of biology.However, due to their small size, they scatter too little light on their own to be directly visible and available for study.Thus, it is necessary to develop characterization methods which enable their elucidation even in the regime of very faint signals. Optical systems, utilizing the relatively low intrusiveness of visible light, constitute one such approach of characterization. However, the optical systems currently capable of analyzing single molecules in the nano-sized regime today either require the species of interest to be tagged with visible labels like fluorescence or chemically restrained on a surface to be analyzed.Ergo, there exist effectively no methods of characterizing very small biomolecules under naturally relevant conditions through unobtrusive probing. Nanofluidic Scattering Microscopy is a method introduced in this thesis which bridges this gap by enabling the real-time label-free size-and-weight determination of freely diffusing molecules directly in small nano-sized channels. However, the molecule signals are so faint, and the background noise so complex with high spatial and temporal variation, that standard methods of data analysis are incapable of elucidating the molecules\u27 properties of relevance in any but the least challenging conditions.To remedy the weak signal, and realize the method\u27s full potential, this thesis\u27 focus is the development of a versatile deep-learning based computer-vision platform to overcome the bottleneck of data analysis. We find that said platform has considerably increased speed, accuracy, precision and limit of detection compared to standard methods, constituting even a lower detection limit than any other method of label-free optical characterization currently available. In this regime, hitherto elusive species of biomolecules become accessible for study, potentially opening up entirely new avenues of biological research. These results, along with many others in the context of deep learning for optical microscopy in biological applications, suggest that deep learning is likely to be pivotal in solving the complex image analysis problems of the present and enabling new regimes of study within microscopy-based research in the near future

Multi-material spectral photon-counting micro-CT with minimum residual decomposition and self-supervised deep denoising

Spectral micro-CT imaging with direct-detection energy discriminating photon counting detectors having small pixel size (< 100×100 μm2) is mainly hampered by: i) the limited energy resolution of the imaging device due to charge sharing effects and ii) the unavoidable noise amplification in the images resulting from basis material decomposition. In this work, we present a cone-beam micro-CT setup that includes a CdTe photon counting detector implementing a charge summing hardware solution to correct for the charge-sharing issue and an innovative image processing pipeline based on accurate modeling of the spectral response of the imaging system, an improved basis material decomposition (BMD) algorithm named minimum-residual BMD (MR-BMD), and self-supervised deep convolutional denoising. Experimental tomographic projections having a pixel size of 45×45 μm2 of a plastinated mouse sample including I, Ba, and Gd small cuvettes were acquired. Results demonstrate the capability of the combined hardware and software tools to sharply discriminate even between materials having their K-Edge separated by a few keV, such as e.g., I and Ba. By evaluating the quality of the reconstructed decomposed images (water, bone, I, Ba, and Gd), the quantitative performances of the spectral system are here assessed and discusse

Random scattering of surface plasmons for sensing and tracking

In this thesis, a single particle biosensing setup, capable of sensing and tracking single nanoscale biological particles, is proposed and investigated theoretically. The setup is based on monitoring the speckle pattern intensity distribution arising due to random scattering of surface plasmon polaritons (SPPs) from a metal surface. An analyte particle close to the surface will additionally scatter light, perturbing the speckle pattern. From this speckle pattern perturbation, the analyte particle can be detected and tracked. Theoretical sensitivity analysis predicts a biological particle on the order of 10nm in radius gives a fractional intensity perturbation to the speckle intensity of 10^4, comparable to intensity contrasts used in existing interferometric scattering sensing techniques. A formula for the minimum detectable particle size is derived. In addition, an algorithm is derived capable of extracting the particle trajectory in the single scattering regime from the change to the speckle intensity perturbation over time and shown to be capable of errors of approximately 1nm on simulated data under optimal noise conditions. The effect of multiple scattering on the speckle pattern perturbation is studied, and it is shown that, by tuning the scattering mean free path and individual scatterer properties of a random nanostructure of scatterers on the metal surface, one can increase the magnitude of the speckle field perturbation by up to the order of 10^2. A neural network based localisation algorithm is developed to calculate the analyte particle position based on the speckle intensity perturbation and its performance on simulated data is studied. Mean errors on the order of 20nm were found, depending on the size of the region over which the particle must be tracked. Unlike the single scattering tracking algorithm, the neural network algorithm continues to function in the multiple scattering regime.Open Acces