Cognitive visual tracking and camera control

Cognitive visual tracking is the process of observing and understanding the behaviour of a moving person. This paper presents an efficient solution to extract, in real-time, high-level information from an observed scene, and generate the most appropriate commands for a set of pan-tilt-zoom (PTZ) cameras in a surveillance scenario. Such a high-level feedback control loop, which is the main novelty of our work, will serve to reduce uncertainties in the observed scene and to maximize the amount of information extracted from it. It is implemented with a distributed camera system using SQL tables as virtual communication channels, and Situation Graph Trees for knowledge representation, inference and high-level camera control. A set of experiments in a surveillance scenario show the effectiveness of our approach and its potential for real applications of cognitive vision

Modern Statistical/Machine Learning Techniques for Bio/Neuro-imaging Applications

Developments in modern bio-imaging techniques have allowed the routine collection of a vast amount of data from various techniques. The challenges lie in how to build accurate and efficient models to draw conclusions from the data and facilitate scientific discoveries. Fortunately, recent advances in statistics, machine learning, and deep learning provide valuable tools. This thesis describes some of our efforts to build scalable Bayesian models for four bio-imaging applications: (1) Stochastic Optical Reconstruction Microscopy (STORM) Imaging, (2) particle tracking, (3) voltage smoothing, (4) detect color-labeled neurons in c elegans and assign identity to the detections

Globally Optimal Cell Tracking using Integer Programming

We propose a novel approach to automatically tracking cell populations in time-lapse images. To account for cell occlusions and overlaps, we introduce a robust method that generates an over-complete set of competing detection hypotheses. We then perform detection and tracking simultaneously on these hypotheses by solving to optimality an integer program with only one type of flow variables. This eliminates the need for heuristics to handle missed detections due to occlusions and complex morphology. We demonstrate the effectiveness of our approach on a range of challenging sequences consisting of clumped cells and show that it outperforms state-of-the-art techniques.Comment: Engin T\"uretken and Xinchao Wang contributed equally to this wor

Multi-target tracking using appearance models for identity maintenance

This thesis considers perception systems for urban environments. It focuses on the task of tracking dynamic objects and in particular on methods that can maintain the identities of targets through periods of ambiguity. Examples of such ambiguous situations occur when targets interact with each other, or when they are occluded by other objects or the environment. With the development of self driving cars, the push for autonomous delivery of packages, and an increasing use of technology for security, surveillance and public-safety applications, robust perception in crowded urban spaces is more important than ever before. A critical part of perception systems is the ability to understand the motion of objects in a scene. Tracking strategies that merge closely-spaced targets together into groups have been shown to offer improved robustness, but in doing so sacrifice the concept of target identity. Additionally, the primary sensor used for the tracking task may not provide the information required to reason about the identity of individual objects. There are three primary contributions in this work. The first is the development of 3D lidar tracking methods with improved ability to track closely-spaced targets and that can determine when target identities have become ambiguous. Secondly, this thesis defines appearance models suitable for the task of determining the identities of previously-observed targets, which may include the use of data from additional sensing modalities. The final contribution of this work is the combination of lidar tracking and appearance modelling, to enable the clarification of target identities in the presence of ambiguities caused by scene complexity. The algorithms presented in this work are validated on both carefully controlled and unconstrained datasets. The experiments show that in complex dynamic scenes with interacting targets, the proposed methods achieve significant improvements in tracking performance

Occlusion reasoning for multiple object visual tracking

Thesis (Ph.D.)--Boston UniversityOcclusion reasoning for visual object tracking in uncontrolled environments is a challenging problem. It becomes significantly more difficult when dense groups of indistinguishable objects are present in the scene that cause frequent inter-object interactions and occlusions. We present several practical solutions that tackle the inter-object occlusions for video surveillance applications. In particular, this thesis proposes three methods. First, we propose "reconstruction-tracking," an online multi-camera spatial-temporal data association method for tracking large groups of objects imaged with low resolution. As a variant of the well-known Multiple-Hypothesis-Tracker, our approach localizes the positions of objects in 3D space with possibly occluded observations from multiple camera views and performs temporal data association in 3D. Second, we develop "track linking," a class of offline batch processing algorithms for long-term occlusions, where the decision has to be made based on the observations from the entire tracking sequence. We construct a graph representation to characterize occlusion events and propose an efficient graph-based/combinatorial algorithm to resolve occlusions. Third, we propose a novel Bayesian framework where detection and data association are combined into a single module and solved jointly. Almost all traditional tracking systems address the detection and data association tasks separately in sequential order. Such a design implies that the output of the detector has to be reliable in order to make the data association work. Our framework takes advantage of the often complementary nature of the two subproblems, which not only avoids the error propagation issue from which traditional "detection-tracking approaches" suffer but also eschews common heuristics such as "nonmaximum suppression" of hypotheses by modeling the likelihood of the entire image. The thesis describes a substantial number of experiments, involving challenging, notably distinct simulated and real data, including infrared and visible-light data sets recorded ourselves or taken from data sets publicly available. In these videos, the number of objects ranges from a dozen to a hundred per frame in both monocular and multiple views. The experiments demonstrate that our approaches achieve results comparable to those of state-of-the-art approaches