Particle detection and tracking in fluorescence time-lapse imaging: a contrario approach

This paper proposes a probabilistic approach for the detection and the tracking of particles in fluorescent time-lapse imaging. In the presence of a very noised and poor-quality data, particles and trajectories can be characterized by an a contrario model, that estimates the probability of observing the structures of interest in random data. This approach, first introduced in the modeling of human visual perception and then successfully applied in many image processing tasks, leads to algorithms that neither require a previous learning stage, nor a tedious parameter tuning and are very robust to noise. Comparative evaluations against a well-established baseline show that the proposed approach outperforms the state of the art.Comment: Published in Journal of Machine Vision and Application

Deep Learning Methods for Detection and Tracking of Particles in Fluorescence Microscopy Images

Studying the dynamics of sub-cellular structures such as receptors, filaments, and vesicles is a prerequisite for investigating cellular processes at the molecular level. In addition, it is important to characterize the dynamic behavior of virus structures to gain a better understanding of infection mechanisms and to develop novel drugs. To investigate the dynamics of fluorescently labeled sub-cellular and viral structures, time-lapse fluorescence microscopy is the most often used imaging technique. Due to the limited spatial resolution of microscopes caused by diffraction, these very small structures appear as bright, blurred spots, denoted as particles, in microscopy images. To draw statistically meaningful biological conclusions, a large number of such particles need to be analyzed. However, since manual analysis of fluorescent particles is very time consuming, fully automated computer-based methods are indispensable. We introduce novel deep learning methods for detection and tracking of multiple particles in fluorescence microscopy images. We propose a particle detection method based on a convolutional neural network which performs image-to-image mapping by density map regression and uses the adaptive wing loss. For particle tracking, we present a recurrent neural network that exploits past and future information in both forward and backward direction. Assignment probabilities across multiple detections as well as the probabilities for missing detections are computed jointly. To resolve tracking ambiguities using future information, several track hypotheses are propagated to later time points. In addition, we developed a novel probabilistic deep learning method for particle tracking, which is based on a recurrent neural network mimicking classical Bayesian filtering. The method includes both aleatoric and epistemic uncertainty, and provides valuable information about the reliability of the computed trajectories. Short and long-term temporal dependencies of individual object dynamics are exploited for state prediction, and assigned detections are used to update the predicted states. Moreover, we developed a convolutional Long Short-Term Memory neural network for combined particle tracking and colocalization analysis in two-channel microscopy image sequences. The network determines colocalization probabilities, and colocalization information is exploited to improve tracking. Short and long-term temporal dependencies of object motion as well as image intensities are taken into account to compute assignment probabilities jointly across multiple detections. We also introduce a deep learning method for probabilistic particle detection and tracking. For particle detection, temporal information is integrated to regress a density map and determine sub-pixel particle positions. For tracking, a fully Bayesian neural network is presented that mimics classical Bayesian filtering and takes into account both aleatoric and epistemic uncertainty. Uncertainty information of individual particle detections is considered. Network training for the developed deep learning-based particle tracking methods relies only on synthetic data, avoiding the need of time-consuming manual annotation. We performed an extensive evaluation of our methods based on image data of the Particle Tracking Challenge as well as on fluorescence microscopy images displaying virus proteins of HCV and HIV, chromatin structures, and cell-surface receptors. It turned out that the methods outperform previous methods

Inferring Latent States and Refining Force Estimates via Hierarchical Dirichlet Process Modeling in Single Particle Tracking Experiments

Optical microscopy provides rich spatio-temporal information characterizing in vivo molecular motion. However, effective forces and other parameters used to summarize molecular motion change over time in live cells due to latent state changes, e.g., changes induced by dynamic micro-environments, photobleaching, and other heterogeneity inherent in biological processes. This study focuses on techniques for analyzing Single Particle Tracking (SPT) data experiencing abrupt state changes. We demonstrate the approach on GFP tagged chromatids experiencing metaphase in yeast cells and probe the effective forces resulting from dynamic interactions that reflect the sum of a number of physical phenomena. State changes are induced by factors such as microtubule dynamics exerting force through the centromere, thermal polymer fluctuations, etc. Simulations are used to demonstrate the relevance of the approach in more general SPT data analyses. Refined force estimates are obtained by adopting and modifying a nonparametric Bayesian modeling technique, the Hierarchical Dirichlet Process Switching Linear Dynamical System (HDP-SLDS), for SPT applications. The HDP-SLDS method shows promise in systematically identifying dynamical regime changes induced by unobserved state changes when the number of underlying states is unknown in advance (a common problem in SPT applications). We expand on the relevance of the HDP-SLDS approach, review the relevant background of Hierarchical Dirichlet Processes, show how to map discrete time HDP-SLDS models to classic SPT models, and discuss limitations of the approach. In addition, we demonstrate new computational techniques for tuning hyperparameters and for checking the statistical consistency of model assumptions directly against individual experimental trajectories; the techniques circumvent the need for "ground-truth" and subjective information.Comment: 25 pages, 6 figures. Differs only typographically from PLoS One publication available freely as an open-access article at http://journals.plos.org/plosone/article?id=10.1371/journal.pone.013763

Motion patterns of subviral particles: Digital tracking, image data processing and analysis

At the Institute of Virology, Philipps-University, Marburg, Germany, currently research on the understanding of the transport mechanisms of Ebola- and Marburgvirus nucleocapsids is carried out. This research demands a profound knowledge about the various motion characteristics of the nucleocapids. The analysis of large amounts of samples by conventional manual evaluation is a laborious task and does not always lead to reproducible and comparable results. In a cooperation between the Institute of Virology, Marburg, and the Institute for Biomedical Engineering, University of Applied Sciences, Giessen, Germany, algorithms are developed and programmed that enable an automatic tracking of subviral particles in fluorescence microscopic image sequences. The algorithms form an interface between the biologic and the algorithmic domain. Furthermore, methods to automatically parameterize and classify subviral particle motions are created. Geometric and mathematical approaches, like curvature-, fractal dimension- and mean squared displacement-determination are applied. Statistical methods are used to compare the measured subviral particle motion parameters between different biological samples. In this thesis, the biological, mathematical and algorithmic basics are described and the state of the art methods of other research groups are presented and compared. The algorithms to track, parameterize, classify and statistically analyze subviral particle tracks are presented in the Methods section. All methods are evaluated with simulated data and/or compared to data validated by a virologist. The methods are applied to a set of real fluorescence microscopic image sequences of Marburgvirus infected live-cells. The Results chapter shows that subviral particle motion can be successfully analyzed using the presented tracking and analysis methods. Furthermore, differences between the subviral particle motions in the analyzed groups could be detected. However, further optimization with manually evaluated data can improve the results. The methods developed in this project enhance the knowledge about nucleocapsid transport and may be valuable for the development of effective antiviral agents to cure Ebola- and Marburgvirus diseases. The thesis concludes with a chapter Discussion and Conclusions

Probabilistic Tracking and Behavior Identification of Fluorescent Particles

Explicit and tractable characterizations of the dynamical behavior of virus particles are pivotal for a thorough understanding of the infection mechanisms of viruses. This thesis deals with the problem of extracting symbolic representations of the dynamical behavior of fluorescent particles from fluorescence microscopy image sequences. The focus is on the behavior of virus particles such as fusion with the cell membrane. A numerical representation is obtained by tracking the particles in the image sequences. We have investigated probabilistic tracking approaches, including approaches based on the Kalman filter as well as based on particle filters. For reasons of efficiency and robustness, we developed a tracking approach based on the probabilistic data association (PDA) algorithm in combination with an ellipsoidal sampling scheme that exploits effectively the image data via parametric appearance models. To track objects in close proximity, we compute the support that each image position provides to each tracked object relative to the support provided to the object's neighbors. After tracking, the problem of mapping the trajectory information computed by the tracking approaches to symbolic representations of the behavior arises. To compute symbolic representations of behaviors related to the fusion of single virus particles with the cell membrane based on their intensity over time, we developed a layered probabilistic approach based on stochastic hybrid systems as well as hidden Markov models (HMMs). We use a maxbelief strategy to efficiently combine both representations. The layered approach describes the intensity, intensity models, and behaviors of single virus particles. We introduce models for the evolution of the intensity and the behavior. To compute estimates for the intensity, intensity models, and behaviors we use a hybrid particle filter and the Viterbi algorithm. The developed approaches have been applied to synthetic images as well as to real microscopy image sequences displaying human immunodeficiency virus (HIV-1) particles. We have performed an extensive quantitative evaluation of the performance and a comparison with several existing approaches. It turned out that our approaches outperform previous ones, thus yielding more accurate and more reliable information about the behavior of virus particles. Moreover, we have successfully applied our tracking approaches to 3D image sequences displaying herpes simplex virus (HSV) replication compartments. We also applied the tracking approaches to image data displaying microtubule tips and analyzed their motion. In addition, our tracking approaches were successfully applied to the 2D and 3D image data of a Particle Tracking Challenge

Protein Tracking by CNN-Based Candidate Pruning and Two-Step Linking with Bayesian Network

Protein trafficking plays a vital role in understanding many biological processes and disease. Automated tracking of protein vesicles is challenging due to their erratic behaviour, changing appearance, and visual clutter. In this paper we present a novel tracking approach which utilizes a two-step linking process that exploits a probabilistic graphical model to predict tracklet linkage. The vesicles are initially detected with help of a candidate selection process, where the candidates are identified by a multi-scale spot enhancing filter. Subsequently, these candidates are pruned and selected by a light weight convolutional neural network. At the linking stage, the tracklets are formed based on the distance and the detection assignment which is implemented via combinatorial optimization algorithm. Each tracklet is described by a number of parameters used to evaluate the probability of tracklets connection by the inference over the Bayesian network. The tracking results are presented for confocal fluorescence microscopy data of protein trafficking in epithelial cells. The proposed method achieves a root mean square error (RMSE) of 1.39 for the vesicle localisation and of 0.7 representing the degree of track matching with ground truth. The presented method is also evaluated against the state-of-the-art “Trackmate“ framework

Unraveling the Thousand Word Picture: An Introduction to Super-Resolution Data Analysis

Super-resolution microscopy provides direct insight into fundamental biological processes occurring at length scales smaller than light’s diffraction limit. The analysis of data at such scales has brought statistical and machine learning methods into the mainstream. Here we provide a survey of data analysis methods starting from an overview of basic statistical techniques underlying the analysis of super-resolution and, more broadly, imaging data. We subsequently break down the analysis of super-resolution data into four problems: the localization problem, the counting problem, the linking problem, and what we’ve termed the interpretation problem

Quantitative analysis of microscopy

Particle tracking is an essential tool for the study of dynamics of biological processes. The dynamics of these processes happens in three-dimensional (3D) space as the biological structures themselves are 3D. The focus of this thesis is on the development of single particle tracking methods for analysis of the dynamics of biological processes through the use of image processing techniques. Firstly, introduced is a novel particle tracking method that works with two-dimensional (2D) image data. This method uses the theory of Haar-like features for particle detection and trajectory linking is achieved using a combination of three Kalman filters within an interacting multiple models framework. The trajectory linking process utilises an extended state space variable which better describe the morphology and intensity profiles of the particles under investigation at their current position. This tracking method is validated using both 2D synthetically generated images as well as 2D experimentally collected images. It is shown that this method outperforms 14 other stateof-the-art methods. Next this method is used to analyse the dynamics of fluorescently labelled particles using a live-cell fluorescence microscopy technique, specifically a variant of the super-resolution (SR) method PALM, spt-PALM. From this application, conclusions about the organisation of the proteins under investigation at the cell membrane are drawn. Introduced next is a second particle tracking method which is highly efficient and capable of working with both 2D and 3D image data. This method uses a novel Haar-inspired feature for particle detection, drawing inspiration from the type of particles to be detected which are typically circular in 2D space and spherical in 3D image space. Trajectory linking in this method utilises a global nearest neighbour methodology incorporating both motion models to describe the motion of the particles under investigation and a further extended state space variable describing many more aspects of the particles to be linked. This method is validated using a variety of both 2D and 3D synthetic image data. The methods performance is compared with 14 other state-of-the-art methods showing it to be one of the best overall performing methods. Finally, analysis tools to study a SR image restoration method developed by our research group, referred to as Translation Microscopy (TRAM) are investigated [1]. TRAM can be implemented on any standardised microscope and deliver an improvement in resolution of up to 7-fold. However, the results from TRAM and other SR imaging methods require specialised tools to validate and analyse them. Tools have been developed to validate that TRAM performs correctly using a specially designed ground truth. Furthermore, through analysis of results on a biological sample corroborate other published results based on the size of biological structures, showing again that TRAM performs as expected.EPSC

Quantitative Live-Cell Imaging of Human Immunodeficiency Virus (HIV-1) Assembly

Advances in fluorescence methodologies make it possible to investigate biological systems in unprecedented detail. Over the last few years, quantitative live-cell imaging has increasingly been used to study the dynamic interactions of viruses with cells and is expected to become even more indispensable in the future. Here, we describe different fluorescence labeling strategies that have been used to label HIV-1 for live cell imaging and the fluorescence based methods used to visualize individual aspects of virus-cell interactions. This review presents an overview of experimental methods and recent experiments that have employed quantitative microscopy in order to elucidate the dynamics of late stages in the HIV-1 replication cycle. This includes cytosolic interactions of the main structural protein, Gag, with itself and the viral RNA genome, the recruitment of Gag and RNA to the plasma membrane, virion assembly at the membrane and the recruitment of cellular proteins involved in HIV-1 release to the nascent budding site