Deep Learning Methods for Detection and Tracking of Particles in Fluorescence Microscopy Images

Studying the dynamics of sub-cellular structures such as receptors, filaments, and vesicles is a prerequisite for investigating cellular processes at the molecular level. In addition, it is important to characterize the dynamic behavior of virus structures to gain a better understanding of infection mechanisms and to develop novel drugs. To investigate the dynamics of fluorescently labeled sub-cellular and viral structures, time-lapse fluorescence microscopy is the most often used imaging technique. Due to the limited spatial resolution of microscopes caused by diffraction, these very small structures appear as bright, blurred spots, denoted as particles, in microscopy images. To draw statistically meaningful biological conclusions, a large number of such particles need to be analyzed. However, since manual analysis of fluorescent particles is very time consuming, fully automated computer-based methods are indispensable. We introduce novel deep learning methods for detection and tracking of multiple particles in fluorescence microscopy images. We propose a particle detection method based on a convolutional neural network which performs image-to-image mapping by density map regression and uses the adaptive wing loss. For particle tracking, we present a recurrent neural network that exploits past and future information in both forward and backward direction. Assignment probabilities across multiple detections as well as the probabilities for missing detections are computed jointly. To resolve tracking ambiguities using future information, several track hypotheses are propagated to later time points. In addition, we developed a novel probabilistic deep learning method for particle tracking, which is based on a recurrent neural network mimicking classical Bayesian filtering. The method includes both aleatoric and epistemic uncertainty, and provides valuable information about the reliability of the computed trajectories. Short and long-term temporal dependencies of individual object dynamics are exploited for state prediction, and assigned detections are used to update the predicted states. Moreover, we developed a convolutional Long Short-Term Memory neural network for combined particle tracking and colocalization analysis in two-channel microscopy image sequences. The network determines colocalization probabilities, and colocalization information is exploited to improve tracking. Short and long-term temporal dependencies of object motion as well as image intensities are taken into account to compute assignment probabilities jointly across multiple detections. We also introduce a deep learning method for probabilistic particle detection and tracking. For particle detection, temporal information is integrated to regress a density map and determine sub-pixel particle positions. For tracking, a fully Bayesian neural network is presented that mimics classical Bayesian filtering and takes into account both aleatoric and epistemic uncertainty. Uncertainty information of individual particle detections is considered. Network training for the developed deep learning-based particle tracking methods relies only on synthetic data, avoiding the need of time-consuming manual annotation. We performed an extensive evaluation of our methods based on image data of the Particle Tracking Challenge as well as on fluorescence microscopy images displaying virus proteins of HCV and HIV, chromatin structures, and cell-surface receptors. It turned out that the methods outperform previous methods

Improving FRAP and SPT for mobility and interaction measurements of molecules and nanoparticles in biomaterials

An increasing amount of pharmaceutical technologies are being developed in which nanoparticles play a crucial role. The rational development of these technologies requires detailed knowledge of the mobility and interaction of the nanoparticles inside complex biomaterials. The aim of this PhD thesis is to improve fluorescence microscopy based methods that allow to extract this information from time sequences of images. In particular, the fluorescence microscopy techniques Fluorescence Recovery After Photobleaching (FRAP) and Single Particle Tracking (SPT) are considered. FRAP modelling is revisited in order to incorporate the effect of the microscope's scanning laser beam on the shape of the photobleached region. The new model should lead to more straightforward an accurate FRAP measurements. SPT is the main focus of the PhD thesis, starting with an investigation of how motion during image acquisition affects the experimental uncertainty with which the nanoparticle positions are determined. This knowledge is used to develop a method that is able to identify interactions between nanoparticles in high detail, by scanning their trajectories for correlated positions. The method is proven to be useful in the context of drug delivery, where it was used to study the intracellular trafficking of polymeric gene complexes. Besides SPT data analysis, it is also explored how light sheet illumination, which allows to strongly reduce the out of focus fluorescence that degrades the contrast in SPT experiments, can be generated by a planar waveguide that is incorporated on a disposable chip. The potential as platform for diagnostic measurements was demonstrated by using the chip to perform SPT size and concentration measurements of cell-derived membrane vesicles. The results of this PhD thesis are expected to contribute to the effort of making accurate SPT and FRAP measurements of nanoparticle properties in biomaterials more accessible to the pharmaceutical research community

Pseudotyping exosomes for enhanced protein delivery in mammalian cells.

Exosomes are cell-derived nanovesicles that hold promise as living vehicles for intracellular delivery of therapeutics to mammalian cells. This potential, however, is undermined by the lack of effective methods to load exosomes with therapeutic proteins and to facilitate their uptake by target cells. Here, we demonstrate how a vesicular stomatitis virus glycoprotein (VSVG) can both load protein cargo onto exosomes and increase their delivery ability via a pseudotyping mechanism. By fusing a set of fluorescent and luminescent reporters with VSVG, we show the successful targeting and incorporation of VSVG fusions into exosomes by gene transfection and fluorescence tracking. We subsequently validate our system by live cell imaging of VSVG and its participation in endosomes/exosomes that are ultimately released from transfected HEK293 cells. We show that VSVG pseudotyping of exosomes does not affect the size or distributions of the exosomes, and both the full-length VSVG and the VSVG without the ectodomain are shown to integrate into the exosomal membrane, suggesting that the ectodomain is not required for protein loading. Finally, exosomes pseudotyped with full-length VSVG are internalized by multiple-recipient cell types to a greater degree compared to exosomes loaded with VSVG without the ectodomain, confirming a role of the ectodomain in cell tropism. In summary, our work introduces a new genetically encoded pseudotyping platform to load and enhance the intracellular delivery of therapeutic proteins via exosome-based vehicles to target cells

Application of Super-Resolution and Advanced Quantitative Microscopy to the Spatio-Temporal Analysis of Influenza Virus Replication

With an estimated three to five million human cases annually and the potential to infect domestic and wild animal populations, influenza viruses are one of the greatest health and economic burdens to our society, and pose an ongoing threat of large-scale pandemics. Despite our knowledge of many important aspects of influenza virus biology, there is still much to learn about how influenza viruses replicate in infected cells, for instance, how they use entry receptors or exploit host cell trafficking pathways. These gaps in our knowledge are due, in part, to the difficulty of directly observing viruses in living cells. In recent years, advances in light microscopy, including super-resolution microscopy and single-molecule imaging, have enabled many viral replication steps to be visualised dynamically in living cells. In particular, the ability to track single virions and their components, in real time, now allows specific pathways to be interrogated, providing new insights to various aspects of the virus-host cell interaction. In this review, we discuss how state-of-the-art imaging technologies, notably quantitative live-cell and super-resolution microscopy, are providing new nanoscale and molecular insights into influenza virus replication and revealing new opportunities for developing antiviral strategies

From single-molecule spectroscopy to super-resolution imaging of the neuron: a review.

For more than 20 years, single-molecule spectroscopy has been providing invaluable insights into nature at the molecular level. The field has received a powerful boost with the development of the technique into super-resolution imaging methods, ca. 10 years ago, which overcome the limitations imposed by optical diffraction. Today, single molecule super-resolution imaging is routinely used in the study of macromolecular function and structure in the cell. Concomitantly, computational methods have been developed that provide information on numbers and positions of molecules at the nanometer-scale. In this overview, we outline the technical developments that have led to the emergence of localization microscopy techniques from single-molecule spectroscopy. We then provide a comprehensive review on the application of the technique in the field of neuroscience research.This work was supported by grants from the UK Engineering and Physical Sciences Research Council (EPSRC), The Wellcome Trust, Alzheimer’s Research UK, the Medical Research Council (MRC), and the Biotechnology and Biological Sciences Resesarch Council (BBSRC)

Imaging Three-Dimensional Single Molecule Dynamics in its Cellular Context

Three-dimensional single molecule microscopy enables the study of dynamic processes in living cells at the level of individual molecules. Multifocal plane microscopy (MUM) is an example of such a modality and has been shown to be capable of capturing the rapid subcellular trafficking of single molecules in thick samples by simultaneously imaging distinct focal planes within the sample. Regardless of the specific modality, however, the obtained 3D trajectories of single molecules often do not fully reveal the biological significance of the observed dynamics. This is because the missing cellular context is often also needed in order to properly understand the events observed at the molecular level. We introduce the remote focusing-MUM (rMUM) modality, which enables 3D single molecule imaging with the simultaneous z-stack imaging of the surrounding cellular structures. Using rMUM, we demonstrate the 3D tracking of prostate-specific membrane antigen (PSMA) with a PSMA-specific antibody in a prostate cancer cell. PSMA is an important biomarker for prostate cancer cells. As such, it is a common target for antibody-based therapies. For example, of particular interest is the use of PSMA-specific antibodies that are conjugated with a toxin that kills prostate cancer cells. We analyze here the pathways of PSMA-specific antibodies, from prior to their first binding to PSMA at the plasma membrane to their arrival at, and continued movement in, sorting endosomes. By making possible the observation of single molecule dynamics within the relevant cellular context, rMUM allows, in our current application, the identification and analysis of different stages of the PSMA-specific antibody trafficking pathway