Algorithms for trajectory integration in multiple views

PhDThis thesis addresses the problem of deriving a coherent and accurate localization of moving objects from partial visual information when data are generated by cameras placed in di erent view angles with respect to the scene. The framework is built around applications of scene monitoring with multiple cameras. Firstly, we demonstrate how a geometric-based solution exploits the relationships between corresponding feature points across views and improves accuracy in object location. Then, we improve the estimation of objects location with geometric transformations that account for lens distortions. Additionally, we study the integration of the partial visual information generated by each individual sensor and their combination into one single frame of observation that considers object association and data fusion. Our approach is fully image-based, only relies on 2D constructs and does not require any complex computation in 3D space. We exploit the continuity and coherence in objects' motion when crossing cameras' elds of view. Additionally, we work under the assumption of planar ground plane and wide baseline (i.e. cameras' viewpoints are far apart). The main contributions are: i) the development of a framework for distributed visual sensing that accounts for inaccuracies in the geometry of multiple views; ii) the reduction of trajectory mapping errors using a statistical-based homography estimation; iii) the integration of a polynomial method for correcting inaccuracies caused by the cameras' lens distortion; iv) a global trajectory reconstruction algorithm that associates and integrates fragments of trajectories generated by each camera

Three dimensional information estimation and tracking for moving objects detection using two cameras framework

Calibration, matching and tracking are major concerns to obtain 3D information consisting of depth, direction and velocity. In finding depth, camera parameters and matched points are two necessary inputs. Depth, direction and matched points can be achieved accurately if cameras are well calibrated using manual traditional calibration. However, most of the manual traditional calibration methods are inconvenient to use because markers or real size of an object in the real world must be provided or known. Self-calibration can solve the traditional calibration limitation, but not on depth and matched points. Other approaches attempted to match corresponding object using 2D visual information without calibration, but they suffer low matching accuracy under huge perspective distortion. This research focuses on achieving 3D information using self-calibrated tracking system. In this system, matching and tracking are done under self-calibrated condition. There are three contributions introduced in this research to achieve the objectives. Firstly, orientation correction is introduced to obtain better relationship matrices for matching purpose during tracking. Secondly, after having relationship matrices another post-processing method, which is status based matching, is introduced for improving object matching result. This proposed matching algorithm is able to achieve almost 90% of matching rate. Depth is estimated after the status based matching. Thirdly, tracking is done based on x-y coordinates and the estimated depth under self-calibrated condition. Results show that the proposed self-calibrated tracking system successfully differentiates the location of objects even under occlusion in the field of view, and is able to determine the direction and the velocity of multiple moving objects

Tracking people in crowds by a part matching approach

The major difficulty in human tracking is the problem raised by challenging occlusions where the target person is repeatedly and extensively occluded by either the background or another moving object. These types of occlusions may cause significant changes in the person's shape, appearance or motion, thus making the data association problem extremely difficult to solve. Unlike most of the existing methods for human tracking that handle occlusions by data association of the complete human body, in this paper we propose a method that tracks people under challenging spatial occlusions based on body part tracking. The human model we propose consists of five body parts with six degrees of freedom and each part is represented by a rich set of features. The tracking is solved using a layered data association approach, direct comparison between features (feature layer) and subsequently matching between parts of the same bodies (part layer) lead to a final decision for the global match (global layer). Experimental results have confirmed the effectiveness of the proposed method. © 2008 IEEE

Video object extraction in distributed surveillance systems

Recently, automated video surveillance and related video processing algorithms have received considerable attention from the research community. Challenges in video surveillance rise from noise, illumination changes, camera motion, splits and occlusions, complex human behavior, and how to manage extracted surveillance information for delivery, archiving, and retrieval: Many video surveillance systems focus on video object extraction, while few focus on both the system architecture and video object extraction. We focus on both and integrate them to produce an end-to-end system and study the challenges associated with building this system. We propose a scalable, distributed, and real-time video-surveillance system with a novel architecture, indexing, and retrieval. The system consists of three modules: video workstations for processing, control workstations for monitoring, and a server for management and archiving. The proposed system models object features as temporal Gaussians and produces: an 18 frames/second frame-rate for SIF video and static cameras, reduced network and storage usage, and precise retrieval results. It is more scalable and delivers more balanced distributed performance than recent architectures. The first stage of video processing is noise estimation. We propose a method for localizing homogeneity and estimating the additive white Gaussian noise variance, which uses spatially scattered initial seeds and utilizes particle filtering techniques to guide their spatial movement towards homogeneous locations from which the estimation is performed. The noise estimation method reduces the number of measurements required by block-based methods while achieving more accuracy. Next, we segment video objects using a background subtraction technique. We generate the background model online for static cameras using a mixture of Gaussians background maintenance approach. For moving cameras, we use a global motion estimation method offline to bring neighboring frames into the coordinate system of the current frame and we merge them to produce the background model. We track detected objects using a feature-based object tracking method with improved detection and correction of occlusion and split. We detect occlusion and split through the identification of sudden variations in the spatia-temporal features of objects. To detect splits, we analyze the temporal behavior of split objects to discriminate between errors in segmentation and real separation of objects. Both objective and subjective experimental results show the ability of the proposed algorithm to detect and correct both splits and occlusions of objects. For the last stage of video processing, we propose a novel method for the detection of vandalism events which is based on a proposed definition for vandal behaviors recorded on surveillance video sequences. We monitor changes inside a restricted site containing vandalism-prone objects and declare vandalism when an object is detected as leaving the site while there is temporally consistent and significant static changes representing damage, given that the site is normally unchanged after use. The proposed method is tested on sequences showing real and simulated vandal behaviors and it achieves a detection rate of 96%. It detects different forms of vandalism such as graffiti and theft. The proposed end-ta-end video surveillance system aims at realizing the potential of video object extraction in automated surveillance and retrieval by focusing on both video object extraction and the management, delivery, and utilization of the extracted informatio

An Efficient Multiple Object Vision Tracking System using Bipartite Graph Matching

For application domains like 11 vs. 11 robot soccer league, crowd surveillance and air traffic control, vision systems need to be able to identify and maintain information in real time about multiple objects as they move through an environment using video images. In this paper, we reduce the multi-object tracking problem to a bipartite graph matching and present efficient techniques that compute the optimal matching in real time. We demonstrate the robustness of our system on a task of tracking indistinguishable objects. One of the advantages of our tracking system is that it requires a much lower frame rate than standard tracking systems to reliably keep track of multiple objects

Intelligent video object tracking in large public environments

This Dissertation addresses the problem of video object tracking in large public environments, and was developed within the context of a partnership between ISCTE-IUL and THALES1 object. This partnership aimed at developing a new approach to video tracking, based on a simple tracking algorithm aided by object position estimations to deal with the harder cases of video object tracking. This proposed approach has been applied successfully in the TRAPLE2 project developed at THALES where the main focus is the real-time monitoring of public spaces and the tracking of moving objects (i.e., persons). The proposed low-processing tracking solution woks as follows: after the detection step, the various objects in the visual scene are tracked through their centres of mass (centroids) that, typically, exhibit little variations along close apart video frames. After this step, some heuristics are applied to the results to maintain coherent the identification of the video objects and estimate their positions in cases of uncertainties, e.g., occlusions, which is one of the major novelties proposed in this Dissertation. The proposed approach was tested with relevant test video sequences representing real video monitoring scenes and the obtained results showed that this approach is able to track multiple persons in real-time with reasonable computational power.Esta dissertação aborda o problema do seguimento de objectos vídeo em ambientes públicos de grande dimensão e foi desenvolvida no contexto de uma parceria entre o ISCTE-IUL e a THALES. Esta parceria visou o desenvolvimento de uma nova abordagem ao seguimento de objectos de vídeo baseada num processamento de vídeo simples em conjunto com a estimação da posição dos objectos nos casos mais difíceis de efectuar o seguimento. Esta abordagem foi aplicada com sucesso no âmbito do projecto TRAPLE desenvolvido pela THALES onde um dos principais enfoques é o seguimento de múltiplos objectos de vídeo em tempo real em espaços públicos, tendo como objectivo o seguimento de pessoas que se movam ao longo desse espaço. A solução de baixo nível de processamento proposta funciona do seguinte modo: após o passo de detecção de objectos, os diversos objectos detectados na cena são seguidos através dos seus centros de massa que, normalmente, apresentam poucas variações ao longo de imagens consecutivas de vídeo. Após este passo, algumas heurísticas são aplicadas aos resultados mantendo a identificação dos objectos de vídeo coerente e estimando as suas posições em casos de incertezas (e.g., oclusões) que é uma das principais novidades propostas nesta dissertação. A abordagem proposta foi testada com várias sequências de vídeo de teste representando cenas reais de videovigilância e os resultados obtidos mostraram que esta abordagem é capaz de seguir várias pessoas em tempo real com um nível de processamento moderado

Use of Microsoft Kinect in a dual camera setup for action recognition applications

Conventional human action recognition methods use a single light camera to extract all the necessary information needed to perform the recognition. However, the use of a single light camera poses limitations which can not be addressed without a hardware change. In this thesis, we propose a novel approach to the multi camera setup. Our approach utilizes the skeletal pose estimation capabilities of the Microsoft Kinect camera, and uses this estimated pose on the image of the non-depth camera. The approach aims at improving performance of image analysis of multiple camera, which would not be as easy in a typical multiple camera setup. The depth information sharing between the camera is in the form of pose projection, which depends on location awareness between them, where the locations can be found using chessboard pattern calibration techniques. Due to the limitations of pattern calibration, we propose a novel calibration refinement approach to increase the detection distance, and simplify the long calibration process. The two tests performed demonstrate that the pose projection process performs with good accuracy with a successful calibration and good Kinect pose estimation, however not so with a failed one. Three tests were performed to determine the calibration performance. Distance calculations were prone to error with a mean accuracy of 96% under 60cm difference, and dropping drastically beyond that, and a stable orientation calculation with mean accuracy of 97%. Last test also proves that our new refinement approach improves the outcome of the projection significantly with a failed pattern calibration, and allows for almost double the camera difference detection of about 120cm. While the orientation mean calculation accuracy achieved similar results to pattern calibration, the distance was less so at around 92%, however, it did maintain a stable standard deviation, while the pattern calibration increased as distance increased