VisionBlocks: A Social Computer Vision Framework

Vision Blocks (http://visionblocks.org) is an on demand, in-browser, customizable computer vision application publishing platform for masses. It empowers end-users (consumers)to create novel solutions for themselves that they would not easily obtain off-the-shelf. By transferring design capability to the consumers, we enable creation and dissemination of custom products and algorithms. We adapt a visual programming paradigm to codify vision algorithms for general use. As a proof of-concept, we implement computer vision algorithms such as motion tracking, face detection, change detection and others. We demonstrate their applications on real-time video. Our studies show that end users (non programmers) only need 50% more time to build such systems when compared to the most experienced researchers. We made progress towards closing the gap between researchers and consumers by finding that users rate the intuitiveness of the approach in a level 6% less than researchers. We discuss different application scenarios where such study will be useful and argue its benefit for computer vision research community. We believe that enabling users with ability to create application will be first step towards creating social computer vision applications and platform.Alfred P. Sloan Foundation (Research Fellowship

Parallelized computational 3D video microscopy of freely moving organisms at multiple gigapixels per second

To study the behavior of freely moving model organisms such as zebrafish (Danio rerio) and fruit flies (Drosophila) across multiple spatial scales, it would be ideal to use a light microscope that can resolve 3D information over a wide field of view (FOV) at high speed and high spatial resolution. However, it is challenging to design an optical instrument to achieve all of these properties simultaneously. Existing techniques for large-FOV microscopic imaging and for 3D image measurement typically require many sequential image snapshots, thus compromising speed and throughput. Here, we present 3D-RAPID, a computational microscope based on a synchronized array of 54 cameras that can capture high-speed 3D topographic videos over a 135-cm^2 area, achieving up to 230 frames per second at throughputs exceeding 5 gigapixels (GPs) per second. 3D-RAPID features a 3D reconstruction algorithm that, for each synchronized temporal snapshot, simultaneously fuses all 54 images seamlessly into a globally-consistent composite that includes a coregistered 3D height map. The self-supervised 3D reconstruction algorithm itself trains a spatiotemporally-compressed convolutional neural network (CNN) that maps raw photometric images to 3D topography, using stereo overlap redundancy and ray-propagation physics as the only supervision mechanism. As a result, our end-to-end 3D reconstruction algorithm is robust to generalization errors and scales to arbitrarily long videos from arbitrarily sized camera arrays. The scalable hardware and software design of 3D-RAPID addresses a longstanding problem in the field of behavioral imaging, enabling parallelized 3D observation of large collections of freely moving organisms at high spatiotemporal throughputs, which we demonstrate in ants (Pogonomyrmex barbatus), fruit flies, and zebrafish larvae

Experimental and computational analyses reveal that environmental restrictions shape HIV-1 spread in 3D cultures

Here, using an integrative experimental and computational approach, Imle et al. show how cell motility and density affect HIV cell-associated transmission in a three-dimensional tissue-like culture system of CD4+ T cells and collagen, and how different collagen matrices restrict infection by cell-free virions

Microfluidics for bacterial chemotaxis

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Civil and Environmental Engineering, 2011.Cataloged from PDF version of thesis.Includes bibliographical references (p. 136-151).Bacterial chemotaxis, a remarkable behavioral trait which allows bacteria to sense and respond to chemical gradients in the environment, has implications in a broad range of fields including but not limited to disease pathogenesis, in-situ bioremediation and marine biogeochemistry. And therefore, studying bacterial chemotaxis is of significant importance to scientists and engineers alike. Microfluidics has revolutionized the way we study the motile behavior of cells by enabling observations at high spatial and temporal resolution in carefully controlled microenvironments. This thesis aims to explore the potential of microfluidic technology in studying bacterial behavior by investigating different aspects of bacterial chemotaxis on a microfluidic platform. We quantified population-scale transport parameters of bacteria using videomicroscopy and cell tracking in controlled chemoattractant gradients. Previously, transport parameters have been derived theoretically from single-cell swimming behavior using probabilistic models, but the mechanistic foundations of this up-scaling process have not been proven experimentally. The parameter estimates computed directly from single-cell swimming information showed good agreement with literature values providing the experimental verification of the upscaling from single cells to population-scale models. Furthermore, we also developed a diffusion-based microfluidic device to generate steady, arbitrarily shaped chemical gradients. Steady gradients, linear or nonlinear, are often a useful model of the bacterial microenvironment to study chemotaxis in the limit of slow patch diffusion or fast motility of free swimming bacterial cells. Observed cell distribution along the gradients showed good agreement with predictions from the bacterial transport equation, providing the first quantification of chemotaxis in steady nonlinear gradients. Also, by observing the time series of the bacterial distributions in different scaled gradients (both steady and unsteady) generated using microfluidic devices, the bacterial response was found to be invariant up to an 87-fold change in ambient chemoattractant concentration. These observations provide an explanation for the ability of bacteria to cope with a broad range of chemical concentrations and gradients in the environment, by means of a flexible sensing network that allows them to rescale their response to take maximum advantage of signals, while discounting less-informative background information. Finally, a microfluidic lattice habitat was developed to study the fate of a chemotactic bacterial population under the pressure of predation. It was observed that the demographic and spatial organization of the bacterial prey population depended on the predator-to-prey ratio as well as on the degree of heterogeneity of the habitat structure. These results represent a first step towards predator-prey microcosms and pave the way for future predator-prey metapopulation studies.by Tanvir Ahmed.Ph.D

Tracking Interacting Objects in Image Sequences

Object tracking in image sequences is a key challenge in computer vision. Its goal is to follow objects that move or evolve over time while preserving the identity of each object. However, most existing approaches focus on one class of objects and model only very simple interactions, such as the fact that different objects do not occupy the same spatial location at a given time instance. They ignore that objects may interact in more complex ways. For example, in a parking lot, a person may get in a car and become invisible in the scene. In this thesis, we focus on tracking interacting objects in image sequences. We show that by exploiting the relationship between different objects, we can achieve more reliable tracking results. We explore a wide range of applications, such as tracking players and the ball in team sports, tracking cars and people in a parking lot and tracking dividing cells in biomedical imagery. We start by tracking the ball in team sports, which is a very challenging task because the ball is often occluded by the players. We propose a sequential approach that tracks the players first, and then tracks the ball by deciding which player, if any, is in possession of the ball at any given time. This is very different from standard approaches that first attempt to track the ball and only then to assign possession. We show that our method substantially increases performance when applied to long basketball and soccer sequences. We then focus on simultaneously tracking interacting objects. We achieve this by formulating the tracking problem as a network-flow Mixed Integer Program, and expressing the fact that one object can appear or disappear at locations of another in terms of linear flow constraints. We demonstrate our method on scenes involving cars and passengers, bags being carried and dropped by people, and balls being passed from one player to the next in team sports. In particular, we show that by estimating jointly and globally the trajectories of different types of objects, the presence of the ones which were not initially detected based solely on image evidence can be inferred from the detections of the others. We finally extend our approach to dividing cells in biomedical imagery. In this case, cells interact by overlapping with each other and giving birth to daughter cells. We propose a novel approach to automatically detecting and tracking cell populations in time-lapse images. Unlike earlier approaches that rely on linking a predetermined and potentially incomplete set of detections, we generate an overcomplete set of competing detection hypotheses. We then perform detection and tracking simultaneously by solving an integer program to find the optimal and consistent subset. This eliminates the need for heuristics to handle missed detections due to occlusions and complex morphology. We demonstrate the effectiveness of our approach on a range of challenging image sequences consisting of clumped cells and show that it outperforms the state-of-the-art techniques

Scalable Inference for Multi-Target Tracking of Proliferating Cells

With the continuous advancements in microscopy techniques such as improved image quality, faster acquisition and reduced photo-toxicity, the amount of data recorded in the life sciences is rapidly growing. Clearly, the size of the data renders manual analysis intractable, calling for automated cell tracking methods. Cell tracking – in contrast to other tracking scenarios – exhibits several difficulties: low signal to noise ratio in the images, high cell density and sometimes cell clusters, radical morphology changes, but most importantly cells divide – which is often the focus of the experiment. These peculiarities have been targeted by tracking-byassignment methods that first extract a set of detection hypotheses and then track those over time. Improving the general quality of these cell tracking methods is difficult, because every cell type, surrounding medium, and microscopy setting leads to recordings with specific properties and problems. This unfortunately implies that automated approaches will not become perfect any time soon but manual proof reading by experts will remain necessary for the time being. In this thesis we focus on two different aspects, firstly on scaling previous and developing new solvers to deal with longer videos and more cells, and secondly on developing a specialized pipeline for detecting and tracking tuberculosis bacteria. The most powerful tracking-by-assignment methods are formulated as probabilistic graphical models and solved as integer linear programs. Because those integer linear programs are in general NP-hard, increasing the problem size will lead to an explosion of computational cost. We begin by reformulating one of these models in terms of a constrained network flow, and show that it can be solved more efficiently. Building on the successful application of network flow algorithms in the pedestrian tracking literature, we develop a heuristic to integrate constraints – here for divisions – into such a network flow method. This allows us to obtain high quality approximations to the tracking solution while providing a polynomial runtime guarantee. Our experiments confirm this much better scaling behavior to larger problems. However, this approach is single threaded and does not utilize available resources of multi-core machines yet. To parallelize the tracking problem we present a simple yet effective way of splitting long videos into intervals that can be tracked independently, followed by a sparse global stitching step that resolves disagreements at the cuts. Going one step further, we propose a microservices based software design for ilastik that allows to distribute all required computation for segmentation, object feature extraction, object classification and tracking across the nodes of a cluster or in the cloud. Finally, we discuss the use case of detecting and tracking tuberculosis bacteria in more detail, because no satisfying automated method to this important problem existed before. One peculiarity of these elongated cells is that they build dense clusters in which it is hard to outline individuals. To cope with that we employ a tracking-by-assignment model that allows competing detection hypotheses and selects the best set of detections while considering the temporal context during tracking. To obtain these hypotheses, we develop a novel algorithm that finds diverseM- best solutions of tree-shaped graphical models by dynamic programming. First experiments with the pipeline indicate that it can greatly reduce the required amount of human intervention for analyzing tuberculosis treatment