Towards A Self-calibrating Video Camera Network For Content Analysis And Forensics

Due to growing security concerns, video surveillance and monitoring has received an immense attention from both federal agencies and private firms. The main concern is that a single camera, even if allowed to rotate or translate, is not sufficient to cover a large area for video surveillance. A more general solution with wide range of applications is to allow the deployed cameras to have a non-overlapping field of view (FoV) and to, if possible, allow these cameras to move freely in 3D space. This thesis addresses the issue of how cameras in such a network can be calibrated and how the network as a whole can be calibrated, such that each camera as a unit in the network is aware of its orientation with respect to all the other cameras in the network. Different types of cameras might be present in a multiple camera network and novel techniques are presented for efficient calibration of these cameras. Specifically: (i) For a stationary camera, we derive new constraints on the Image of the Absolute Conic (IAC). These new constraints are shown to be intrinsic to IAC; (ii) For a scene where object shadows are cast on a ground plane, we track the shadows on the ground plane cast by at least two unknown stationary points, and utilize the tracked shadow positions to compute the horizon line and hence compute the camera intrinsic and extrinsic parameters; (iii) A novel solution to a scenario where a camera is observing pedestrians is presented. The uniqueness of formulation lies in recognizing two harmonic homologies present in the geometry obtained by observing pedestrians; (iv) For a freely moving camera, a novel practical method is proposed for its self-calibration which even allows it to change its internal parameters by zooming; and (v) due to the increased application of the pan-tilt-zoom (PTZ) cameras, a technique is presented that uses only two images to estimate five camera parameters. For an automatically configurable multi-camera network, having non-overlapping field of view and possibly containing moving cameras, a practical framework is proposed that determines the geometry of such a dynamic camera network. It is shown that only one automatically computed vanishing point and a line lying on any plane orthogonal to the vertical direction is sufficient to infer the geometry of a dynamic network. Our method generalizes previous work which considers restricted camera motions. Using minimal assumptions, we are able to successfully demonstrate promising results on synthetic as well as on real data. Applications to path modeling, GPS coordinate estimation, and configuring mixed-reality environment are explored

Communication framework for distributed computer vision on stationary and mobile platforms

Recent advances in the complexity and manufacturability of digital video cameras coupled with the ubiquity of high speed computers and communication networks have led to burgeoning research in the fields of computer vision and image understanding. As the generated vision algorithms become increasingly complex, a need arises for robust communication between remote cameras on mobile units and their associated distributed vision algorithms. A communication framework would provide a basis for modularization and abstraction of a collection of computer vision algorithms; the resulting system would allow for straightforward image capture, simplified communication between algorithms, and easy replacement or upgrade of existing component algorithms. The objective of this thesis is to create such a communication framework and demonstrate its viability and applicability by implementing a relatively complex system of distributed computer vision algorithms. These multi-camera algorithms include body tracking, pose estimation and face recognition. Although a plethora of research exists documenting individual algorithms which may utilize multiple networked cameras, this thesis aims to develop a novel way of sharing information between cameras and algorithms in a distributed computation system. In addition, this thesis strives to extend such an approach to using both stationary and mobile cameras. For this application, a mobile computer vision platform was developed that integrates seamlessly with the aforementioned communication framework, extending both its functionality and robustness

Camera Planning and Fusion in a Heterogeneous Camera Network

Wide-area camera networks are becoming more and more common. They have widerange of commercial and military applications from video surveillance to smart home and from traffic monitoring to anti-terrorism. The design of such a camera network is a challenging problem due to the complexity of the environment, self and mutual occlusion of moving objects, diverse sensor properties and a myriad of performance metrics for different applications. In this dissertation, we consider two such challenges: camera planing and camera fusion. Camera planning is to determine the optimal number and placement of cameras for a target cost function. Camera fusion describes the task of combining images collected by heterogenous cameras in the network to extract information pertinent to a target application. I tackle the camera planning problem by developing a new unified framework based on binary integer programming (BIP) to relate the network design parameters and the performance goals of a variety of camera network tasks. Most of the BIP formulations are NP hard problems and various approximate algorithms have been proposed in the literature. In this dissertation, I develop a comprehensive framework in comparing the entire spectrum of approximation algorithms from Greedy, Markov Chain Monte Carlo (MCMC) to various relaxation techniques. The key contribution is to provide not only a generic formulation of the camera planning problem but also novel approaches to adapt the formulation to powerful approximation schemes including Simulated Annealing (SA) and Semi-Definite Program (SDP). The accuracy, efficiency and scalability of each technique are analyzed and compared in depth. Extensive experimental results are provided to illustrate the strength and weakness of each method. The second problem of heterogeneous camera fusion is a very complex problem. Information can be fused at different levels from pixel or voxel to semantic objects, with large variation in accuracy, communication and computation costs. My focus is on the geometric transformation of shapes between objects observed at different camera planes. This so-called the geometric fusion approach usually provides the most reliable fusion approach at the expense of high computation and communication costs. To tackle the complexity, a hierarchy of camera models with different levels of complexity was proposed to balance the effectiveness and efficiency of the camera network operation. Then different calibration and registration methods are proposed for each camera model. At last, I provide two specific examples to demonstrate the effectiveness of the model: 1)a fusion system to improve the segmentation of human body in a camera network consisted of thermal and regular visible light cameras and 2) a view dependent rendering system by combining the information from depth and regular cameras to collecting the scene information and generating new views in real time

이종 센서들을 이용한 지능형 공간의 운용

학위논문 (박사)-- 서울대학교 대학원 : 전기·컴퓨터공학부, 2014. 8. 이범희.A new approach of multi-sensor operation is presented in an intelligent space, which is based on heterogeneous multiple vision sensors and robots mounted with an infrared (IR) sensor. The intelligent space system is a system that exists in task space of robots, helps missions of the robots, and can self-control the robots in a particular situation. The conventional intelligent space consists of solely static cameras. However, the adoption of multiple heterogeneous sensors and an operation technique for the sensors are required in order to extend the ability of intelligent space. First, this dissertation presents the sub-systems for each sensor group in the proposed intelligent space. The vision sensors consist of two groups: static (fixed) cameras and dynamic (pan-tilt) cameras. Each sub-system can detect and track the robots. The sub-system using static cameras localize the robot within a high degree of accuracy. In this system, a handoff method is proposed using the world-to-pixel transformation in order to interwork among the multiple static cameras. The sub-system using dynamic cameras is designed to have various views without losing the robot in view. In this system, a handoff method is proposed using the predictive positions of the robot, relationship among cameras, and relationship between the robot and the camera in order to interwork among the multiple dynamic cameras. The robots system localizes itself using an IR sensor and IR tags. The IR sensor can localize the robot even if illumination of the environment is low. For robust tracking, a sensor selection method is proposed using the advantages of these sensors under environmental change of the task space. For the selection method, we define interface protocol among the sub-systems, sensor priority, and selection criteria. The proposed method is adequate for a real-time system, which has a low computational cost than sensor fusion methods. Performance of each sensor group is verified through various experiments. In addition, multi-sensor operation using the proposed sensor selection method is experimentally verified in the environment with an occlusion and low-illumination setting.Abstracts i Contents iii List of Figures vii List of Tables xv Chapter 1 Introduction 1 1.1 Background and Motivation 1 1.2 Related Work 4 1.3 Contributions 7 1.4 Organization 10 Chapter 2 Overview of Intelligent Space 11 2.1 Original Concept of Intelligent Space 11 2.2 Related Research 13 2.3 Problem Statement and Objective 16 Chpater 3 Architecture of a Proposed Intelligent Space 18 3.1 Hardware Architecture 19 3.2.1 Metallic Structure 20 3.2.2 Static Cameras 22 3.2.3 Dynamic Cameras 24 3.2.4 Infrared (IR) Camera and Passive IR Tags 27 3.2.5 Mobile Robots 28 3.2 Software Architecture 31 Chpater 4 Localization and Tracking of Mobile Robots in a Proposed Intelligent Space 36 4.1 Localization and Tracking with an IR Sensor Mounted on Robots 36 4.1.1 Deployment of IR Tags 36 4.1.2 Localization and Tracking Using an IR Sensor 38 4.2 Localization and Tracking with Multiple Dynamic Cameras 41 4.2.1 Localization and Tracking based on the Geometry between a Robot and a Single Dynamic Camera 41 4.2.2 Proposed Predictive Handoff among Dynamic Cameras 45 4.3 Localization and Tracking with Multiple Static Cameras 53 4.3.1 Preprocess for Static Cameras 53 4.3.2 Marker-based Localization and Tracking of Multiple Robots 58 4.3.3 Proposed Reprojection-based Handoff among Static Cameras 67 Chpater 5 Operation with Heterogeneous Sensor Groups 72 5.1 Interface Protocol among Sensor Groups 72 5.2 Sensor Selection for an Operation Using Heterogeneous Sensors 84 5.3 Proposed Operation with Static Cameras and Dynamic cameras 87 5.4 Proposed Operation with the iSpace and Robots 90 Chapter 6 Experimental Results 94 6.1 Experimental Setup 94 6.2 Experimental Results of Localization 95 6.2.1 Results using Static Cameras and Dynamic Cameras 95 6.2.2 Results using the IR Sensor 102 6.3 Experimental Results of Tracking 104 6.3.1 Results using Static and Dynamic Cameras 104 6.3.2 Results using the IR Sensor 108 6.4 Experimental Results using Heterogeneous Sensors 111 6.4.1 Results in Environment with Occlusion 111 6.4.2 Results in Low-illumination Environment 115 6.5 Discussion 118 Chapter 7 Conclusions 120 Bibliography 125Docto

Activity topology estimation for large networks of cameras

Copyright © 2006 IEEEEstimating the paths that moving objects can take through the fields of view of possibly non-overlapping cameras, also known as their activity topology, is an important step in the effective interpretation of surveillance video. Existing approaches to this problem involve tracking moving objects within cameras, and then attempting to link tracks across views. In contrast we propose an approach which begins by assuming all camera views are potentially linked, and successively eliminates camera topologies that are contradicted by observed motion. Over time, the true patterns of motion emerge as those which are not contradicted by the evidence. These patterns may then be used to initialise a finer level search using other approaches if required. This method thus represents an efficient and effective way to learn activity topology for a large network of cameras, particularly with a limited amount of data.van den Hengel, A.; Dick, A.; Hill, R

Object Association Across Multiple Moving Cameras In Planar Scenes

In this dissertation, we address the problem of object detection and object association across multiple cameras over large areas that are well modeled by planes. We present a unifying probabilistic framework that captures the underlying geometry of planar scenes, and present algorithms to estimate geometric relationships between different cameras, which are subsequently used for co-operative association of objects. We first present a local1 object detection scheme that has three fundamental innovations over existing approaches. First, the model of the intensities of image pixels as independent random variables is challenged and it is asserted that useful correlation exists in intensities of spatially proximal pixels. This correlation is exploited to sustain high levels of detection accuracy in the presence of dynamic scene behavior, nominal misalignments and motion due to parallax. By using a non-parametric density estimation method over a joint domain-range representation of image pixels, complex dependencies between the domain (location) and range (color) are directly modeled. We present a model of the background as a single probability density. Second, temporal persistence is introduced as a detection criterion. Unlike previous approaches to object detection that detect objects by building adaptive models of the background, the foreground is modeled to augment the detection of objects (without explicit tracking), since objects detected in the preceding frame contain substantial evidence for detection in the current frame. Finally, the background and foreground models are used competitively in a MAP-MRF decision framework, stressing spatial context as a condition of detecting interesting objects and the posterior function is maximized efficiently by finding the minimum cut of a capacitated graph. Experimental validation of the method is performed and presented on a diverse set of data. We then address the problem of associating objects across multiple cameras in planar scenes. Since cameras may be moving, there is a possibility of both spatial and temporal non-overlap in the fields of view of the camera. We first address the case where spatial and temporal overlap can be assumed. Since the cameras are moving and often widely separated, direct appearance-based or proximity-based constraints cannot be used. Instead, we exploit geometric constraints on the relationship between the motion of each object across cameras, to test multiple correspondence hypotheses, without assuming any prior calibration information. Here, there are three contributions. First, we present a statistically and geometrically meaningful means of evaluating a hypothesized correspondence between multiple objects in multiple cameras. Second, since multiple cameras exist, ensuring coherency in association, i.e. transitive closure is maintained between more than two cameras, is an essential requirement. To ensure such coherency we pose the problem of object associating across cameras as a k-dimensional matching and use an approximation to find the association. We show that, under appropriate conditions, re-entering objects can also be re-associated to their original labels. Third, we show that as a result of associating objects across the cameras, a concurrent visualization of multiple aerial video streams is possible. Results are shown on a number of real and controlled scenarios with multiple objects observed by multiple cameras, validating our qualitative models. Finally, we present a unifying framework for object association across multiple cameras and for estimating inter-camera homographies between (spatially and temporally) overlapping and non-overlapping cameras, whether they are moving or non-moving. By making use of explicit polynomial models for the kinematics of objects, we present algorithms to estimate inter-frame homographies. Under an appropriate measurement noise model, an EM algorithm is applied for the maximum likelihood estimation of the inter-camera homographies and kinematic parameters. Rather than fit curves locally (in each camera) and match them across views, we present an approach that simultaneously refines the estimates of inter-camera homographies and curve coefficients globally. We demonstrate the efficacy of the approach on a number of real sequences taken from aerial cameras, and report quantitative performance during simulations

POINTING, ACQUISITION, AND TRACKING FOR DIRECTIONAL WIRELESS COMMUNICATIONS NETWORKS

Directional wireless communications networks (DWNs) are expected to become a workhorse of the military, as they provide great network capacity in hostile areas where omnidirectional RF systems can put their users in harm's way. These networks will also be able to adapt to new missions, change topologies, use different communications technologies, yet still reliably serve all their terminal users. DWNs also have the potential to greatly expand the capacity of civilian and commercial wireless communication. The inherently narrow beams present in these types of systems require a means of steering them, acquiring the links, and tracking to maintain connectivity. This area of technological challenges encompasses all the issues of pointing, acquisition, and tracking (PAT). iii The two main technologies for DWNs are Free-Space Optical (FSO) and millimeter wave RF (mmW). FSO offers tremendous bandwidths, long ranges, and uses existing fiber-based technologies. However, it suffers from severe turbulence effects when passing through long (>kms) atmospheric paths, and can be severely affected by obscuration. MmW systems do not suffer from atmospheric effects nearly as much, use much more sensitive coherent receivers, and have wider beam divergences allowing for easier pointing. They do, however, suffer from a lack of available small-sized power amplifiers, complicated RF infrastructure that must be steered with a platform, and the requirement that all acquisition and tracking be done with the data beam, as opposed to FSO which uses a beacon laser for acquisition and a fast steering mirror for tracking. This thesis analyzes the many considerations required for designing and implementing a FSO PAT system, and extends this work to the rapidly expanding area of mmW DWN systems. Different types of beam acquisition methods are simulated and tested, and the tradeoffs between various design specifications are analyzed and simulated to give insight into how to best implement a transceiver platform. An experimental test-bed of six FSO platforms is also designed and constructed to test some of these concepts, along with the implementation of a three-node biconnected network. Finally, experiments have been conducted to assess the performance of fixed infrastructure routing hardware when operating with a physically reconfigurable RF network