Trajectory based video analysis in multi-camera setups

PhDThis thesis presents an automated framework for activity analysis in multi-camera setups. We start with the calibration of cameras particularly without overlapping views. An algorithm is presented that exploits trajectory observations in each view and works iteratively on camera pairs. First outliers are identified and removed from observations of each camera. Next, spatio-temporal information derived from the available trajectory is used to estimate unobserved trajectory segments in areas uncovered by the cameras. The unobserved trajectory estimates are used to estimate the relative position of each camera pair, whereas the exit-entrance direction of each object is used to estimate their relative orientation. The process continues and iteratively approximates the configuration of all cameras with respect to each other. Finally, we refi ne the initial configuration estimates with bundle adjustment, based on the observed and estimated trajectory segments. For cameras with overlapping views, state-of-the-art homography based approaches are used for calibration. Next we establish object correspondence across multiple views. Our algorithm consists of three steps, namely association, fusion and linkage. For association, local trajectory pairs corresponding to the same physical object are estimated using multiple spatio-temporal features on a common ground plane. To disambiguate spurious associations, we employ a hybrid approach that utilises the matching results on the image plane and ground plane. The trajectory segments after association are fused by adaptive averaging. Trajectory linkage then integrates segments and generates a single trajectory of an object across the entire observed area. Finally, for activities analysis clustering is applied on complete trajectories. Our clustering algorithm is based on four main steps, namely the extraction of a set of representative trajectory features, non-parametric clustering, cluster merging and information fusion for the identification of normal and rare object motion patterns. First we transform the trajectories into a set of feature spaces on which Meanshift identi es the modes and the corresponding clusters. Furthermore, a merging procedure is devised to re fine these results by combining similar adjacent clusters. The fi nal common patterns are estimated by fusing the clustering results across all feature spaces. Clusters corresponding to reoccurring trajectories are considered as normal, whereas sparse trajectories are associated to abnormal and rare events. The performance of the proposed framework is evaluated on standard data-sets and compared with state-of-the-art techniques. Experimental results show that the proposed framework outperforms state-of-the-art algorithms both in terms of accuracy and robustness

A molecular dynamics simulation based principal component analysis framework for computation of multi-scale modeling of protein and its interaction with solvent

This dissertation presents a new computational framework for calculating the normal modes and interactions of proteins, macromolecular assemblies and surrounding solvents. The framework employs a combination of molecular dynamics simulation (MD) and principal component analysis (PCA). It enables the capture and visualization of the molecules\u27 normal modes and interactions over time scales that are computationally challenging. It also provides a starting point for experimental and further computational studies of protein conformational changes. A protein\u27s function is sometimes linked to its conformational flexibility. Normal mode analysis (NMA) and various extensions of it have provided insights into the conformational fluctuations associated with individual protein structures. In traditional NMA, each protein requires a customized model for such analysis due to its mechanical complexity. The methodology presented here is applicable to any protein with known atomic coordinates. Because of its computational efficiency and scalability, it facilitates the study of slow protein conformational changes (on the order of milliseconds), such as protein folding. PCA reduces the dimensionality of MD atomic trajectory data and provides a concise way to visualize, analyze, and compare the motions observed over the course of a simulation. PCA involves diagonalization of the positional covariance matrix and identification of an orthogonal set of eigenvectors or modes describing the direction of maximum variation in the observed conformational distribution. Consequently, slow conformational changes can be identified by projecting these dominant modes back to original trajectory data. In this work, the new multiscale methodology was first applied to a relatively small mutant T4 phage lysozyme, establishing its equilibrium atomic thermal fluctuations and its inter-residue fluctuation correlations. These results were compared with published data obtained by NMA, by finite element methods, and by experiment. The eigenmodes captured are in quantitative agreement with previously published results. With this success on a small protein, the method was applied to the interaction of mutated hemoglobin molecules that cause sickle cell anemia and the atomic level details of which are unknown. The new methodology reveals slow motion processes of the hemoglobin-hemoglobin interaction. MD based PCA is computationally expensive. Thus, this dissertation work also includes a widely-applicable parallel programming implementation of the modeling framework to improve its performance

Tracking by Prediction: A Deep Generative Model for Mutli-Person localisation and Tracking

Current multi-person localisation and tracking systems have an over reliance on the use of appearance models for target re-identification and almost no approaches employ a complete deep learning solution for both objectives. We present a novel, complete deep learning framework for multi-person localisation and tracking. In this context we first introduce a light weight sequential Generative Adversarial Network architecture for person localisation, which overcomes issues related to occlusions and noisy detections, typically found in a multi person environment. In the proposed tracking framework we build upon recent advances in pedestrian trajectory prediction approaches and propose a novel data association scheme based on predicted trajectories. This removes the need for computationally expensive person re-identification systems based on appearance features and generates human like trajectories with minimal fragmentation. The proposed method is evaluated on multiple public benchmarks including both static and dynamic cameras and is capable of generating outstanding performance, especially among other recently proposed deep neural network based approaches.Comment: To appear in IEEE Winter Conference on Applications of Computer Vision (WACV), 201

Modelling shared space users via rule-based social force model

The promotion of space sharing in order to raise the quality of community living and safety of street surroundings is increasingly accepted feature of modern urban design. In this context, the development of a shared space simulation tool is essential in helping determine whether particular shared space schemes are suitable alternatives to traditional street layouts. A simulation tool that enables urban designers to visualise pedestrians and cars trajectories, extract flow and density relation in a new shared space design and achieve solutions for optimal design features before implementation. This paper presents a three-layered microscopic mathematical model which is capable of representing the behaviour of pedestrians and vehicles in shared space layouts and it is implemented in a traffic simulation tool. The top layer calculates route maps based on static obstacles in the environment. It plans the shortest path towards agents' respective destinations by generating one or more intermediate targets. In the second layer, the Social Force Model (SFM) is modified and extended for mixed traffic to produce feasible trajectories. Since vehicle movements are not as flexible as pedestrian movements, velocity angle constraints are included for vehicles. The conflicts described in the third layer are resolved by rule-based constraints for shared space users. An optimisation algorithm is applied to determine the interaction parameters of the force-based model for shared space users using empirical data. This new three-layer microscopic model can be used to simulate shared space environments and assess, for example, new street designs