Cognitive visual tracking and camera control

Cognitive visual tracking is the process of observing and understanding the behaviour of a moving person. This paper presents an efficient solution to extract, in real-time, high-level information from an observed scene, and generate the most appropriate commands for a set of pan-tilt-zoom (PTZ) cameras in a surveillance scenario. Such a high-level feedback control loop, which is the main novelty of our work, will serve to reduce uncertainties in the observed scene and to maximize the amount of information extracted from it. It is implemented with a distributed camera system using SQL tables as virtual communication channels, and Situation Graph Trees for knowledge representation, inference and high-level camera control. A set of experiments in a surveillance scenario show the effectiveness of our approach and its potential for real applications of cognitive vision

Development of an Active Vision System for the Remote Identification of Multiple Targets

This thesis introduces a centralized active vision system for the remote identification of multiple targets in applications where the targets may outnumber the active system resources. Design and implementation details of a modular active vision system are presented, from which a prototype has been constructed. The system employs two different, yet complimentary, camera technologies. Omnidirectional cameras are used to detect and track targets at a low resolution, while perspective cameras mounted to pan-tilt stages are used to acquire high resolution images suitable for identification. Five greedy-based scheduling policies have been developed and implemented to manage the active system resources in an attempt to achieve optimal target-to-camera assignments. System performance has been evaluated using both simulated and real-world experiments under different target and system configurations for all five scheduling policies. Parameters affecting performance that were considered include: target entry conditions, congestion levels, target to camera speeds, target trajectories, and number of active cameras. An overall trend in the relative performance of the scheduling algorithms was observed. The Least System Reconfiguration and Future Least System Reconfiguration scheduling policies performed the best for the majority of conditions investigated, while the Load Sharing and First Come First Serve policies performed the poorest. The performance of the Earliest Deadline First policy was seen to be highly dependent on target predictability

Development of an Active Vision System for the Remote Identification of Multiple Targets

This thesis introduces a centralized active vision system for the remote identification of multiple targets in applications where the targets may outnumber the active system resources. Design and implementation details of a modular active vision system are presented, from which a prototype has been constructed. The system employs two different, yet complimentary, camera technologies. Omnidirectional cameras are used to detect and track targets at a low resolution, while perspective cameras mounted to pan-tilt stages are used to acquire high resolution images suitable for identification. Five greedy-based scheduling policies have been developed and implemented to manage the active system resources in an attempt to achieve optimal target-to-camera assignments. System performance has been evaluated using both simulated and real-world experiments under different target and system configurations for all five scheduling policies. Parameters affecting performance that were considered include: target entry conditions, congestion levels, target to camera speeds, target trajectories, and number of active cameras. An overall trend in the relative performance of the scheduling algorithms was observed. The Least System Reconfiguration and Future Least System Reconfiguration scheduling policies performed the best for the majority of conditions investigated, while the Load Sharing and First Come First Serve policies performed the poorest. The performance of the Earliest Deadline First policy was seen to be highly dependent on target predictability

Spatio-temporal coverage optimization of sensor networks

Les réseaux de capteurs sont formés d’un ensemble de dispositifs capables de prendre individuellement des mesures d’un environnement particulier et d’échanger de l’information afin d’obtenir une représentation de haut niveau sur les activités en cours dans la zone d’intérêt. Une telle détection distribuée, avec de nombreux appareils situés à proximité des phénomènes d’intérêt, est pertinente dans des domaines tels que la surveillance, l’agriculture, l’observation environnementale, la surveillance industrielle, etc. Nous proposons dans cette thèse plusieurs approches pour effectuer l’optimisation des opérations spatio-temporelles de ces dispositifs, en déterminant où les placer dans l’environnement et comment les contrôler au fil du temps afin de détecter les cibles mobiles d’intérêt. La première nouveauté consiste en un modèle de détection réaliste représentant la couverture d’un réseau de capteurs dans son environnement. Nous proposons pour cela un modèle 3D probabiliste de la capacité de détection d’un capteur sur ses abords. Ce modèle inègre également de l’information sur l’environnement grâce à l’évaluation de la visibilité selon le champ de vision. À partir de ce modèle de détection, l’optimisation spatiale est effectuée par la recherche du meilleur emplacement et l’orientation de chaque capteur du réseau. Pour ce faire, nous proposons un nouvel algorithme basé sur la descente du gradient qui a été favorablement comparée avec d’autres méthodes génériques d’optimisation «boites noires» sous l’aspect de la couverture du terrain, tout en étant plus efficace en terme de calculs. Une fois que les capteurs placés dans l’environnement, l’optimisation temporelle consiste à bien couvrir un groupe de cibles mobiles dans l’environnement. D’abord, on effectue la prédiction de la position future des cibles mobiles détectées par les capteurs. La prédiction se fait soit à l’aide de l’historique des autres cibles qui ont traversé le même environnement (prédiction à long terme), ou seulement en utilisant les déplacements précédents de la même cible (prédiction à court terme). Nous proposons de nouveaux algorithmes dans chaque catégorie qui performent mieux ou produits des résultats comparables par rapport aux méthodes existantes. Une fois que les futurs emplacements de cibles sont prédits, les paramètres des capteurs sont optimisés afin que les cibles soient correctement couvertes pendant un certain temps, selon les prédictions. À cet effet, nous proposons une méthode heuristique pour faire un contrôle de capteurs, qui se base sur les prévisions probabilistes de trajectoire des cibles et également sur la couverture probabiliste des capteurs des cibles. Et pour terminer, les méthodes d’optimisation spatiales et temporelles proposées ont été intégrées et appliquées avec succès, ce qui démontre une approche complète et efficace pour l’optimisation spatio-temporelle des réseaux de capteurs.Sensor networks consist in a set of devices able to individually capture information on a given environment and to exchange information in order to obtain a higher level representation on the activities going on in the area of interest. Such a distributed sensing with many devices close to the phenomena of interest is of great interest in domains such as surveillance, agriculture, environmental monitoring, industrial monitoring, etc. We are proposing in this thesis several approaches to achieve spatiotemporal optimization of the operations of these devices, by determining where to place them in the environment and how to control them over time in order to sense the moving targets of interest. The first novelty consists in a realistic sensing model representing the coverage of a sensor network in its environment. We are proposing for that a probabilistic 3D model of sensing capacity of a sensor over its surrounding area. This model also includes information on the environment through the evaluation of line-of-sight visibility. From this sensing model, spatial optimization is conducted by searching for the best location and direction of each sensor making a network. For that purpose, we are proposing a new algorithm based on gradient descent, which has been favourably compared to other generic black box optimization methods in term of performance, while being more effective when considering processing requirements. Once the sensors are placed in the environment, the temporal optimization consists in covering well a group of moving targets in the environment. That starts by predicting the future location of the mobile targets detected by the sensors. The prediction is done either by using the history of other targets who traversed the same environment (long term prediction), or only by using the previous displacements of the same target (short term prediction). We are proposing new algorithms under each category which outperformed or produced comparable results when compared to existing methods. Once future locations of targets are predicted, the parameters of the sensors are optimized so that targets are properly covered in some future time according to the predictions. For that purpose, we are proposing a heuristics for making such sensor control, which deals with both the probabilistic targets trajectory predictions and probabilistic coverage of sensors over the targets. In the final stage, both spatial and temporal optimization method have been successfully integrated and applied, demonstrating a complete and effective pipeline for spatiotemporal optimization of sensor networks

WATCHING PEOPLE: ALGORITHMS TO STUDY HUMAN MOTION AND ACTIVITIES

Nowadays human motion analysis is one of the most active research topics in Computer Vision and it is receiving an increasing attention from both the industrial and scientific communities. The growing interest in human motion analysis is motivated by the increasing number of promising applications, ranging from surveillance, human–computer interaction, virtual reality to healthcare, sports, computer games and video conferencing, just to name a few. The aim of this thesis is to give an overview of the various tasks involved in visual motion analysis of the human body and to present the issues and possible solutions related to it. In this thesis, visual motion analysis is categorized into three major areas related to the interpretation of human motion: tracking of human motion using virtual pan-tilt-zoom (vPTZ) camera, recognition of human motions and human behaviors segmentation. In the field of human motion tracking, a virtual environment for PTZ cameras (vPTZ) is presented to overcame the mechanical limitations of PTZ cameras. The vPTZ is built on equirectangular images acquired by 360° cameras and it allows not only the development of pedestrian tracking algorithms but also the comparison of their performances. On the basis of this virtual environment, three novel pedestrian tracking algorithms for 360° cameras were developed, two of which adopt a tracking-by-detection approach while the last adopts a Bayesian approach. The action recognition problem is addressed by an algorithm that represents actions in terms of multinomial distributions of frequent sequential patterns of different length. Frequent sequential patterns are series of data descriptors that occur many times in the data. The proposed method learns a codebook of frequent sequential patterns by means of an apriori-like algorithm. An action is then represented with a Bag-of-Frequent-Sequential-Patterns approach. In the last part of this thesis a methodology to semi-automatically annotate behavioral data given a small set of manually annotated data is presented. The resulting methodology is not only effective in the semi-automated annotation task but can also be used in presence of abnormal behaviors, as demonstrated empirically by testing the system on data collected from children affected by neuro-developmental disorders

Design of a mobile robot’s control system for obstacle identification and avoidance using sensor fusion and model predictive control

The aim of this master thesis is to design a control system based on model predictive control (MPC) with sensor data fusion for obstacle avoidance. Since the amount of obtained data is larger due to multiple sensors, the required sampling time has to be larger enough in comparison with the calculation time of the optimal problem. Then it is proposed a simplification of the mobile robot model in order to reduce this optimization time. The sensor data fusion technique uses the range information of a laser scanner and the data of a mono-camera acquired from image processing techniques. In image processing different detection algorithms are proposed such as shape and color detection. Therefore an estimation of the obstacles dimension and distance is explained obtaining accurate results. Finally a data fusion for obstacle determination is developed in order to use this information in the optimization control problem as a path constraint. The obtained results show the mobile robot behavior in trajectories tracking and obstacle avoidance problems by comparing two different sampling times. It is concluded that the mobile robot reaches the final desired position while avoiding the detected obstacles along the trajectory.Ziel dieser Masterarbeit ist, einen Steuerungsentwurf auf Basis der modellprädiktiven Regelung (MPC) mit Sensordatenfusion und zur Hindernisvermeidung. Da die Menge der erhaltenen Daten aufgrund mehrerer Sensoren größer ist, muss die erforderliche Abtastzeit im Vergleich zur Rechenzeit des optimalen Problems größer sein. In der Arbeit wird eine Vereinfachung des mobilen Robotermodells vorgeschlagen, um diese Optimierungszeit zu reduzieren. Die Sensordaten-Fusionstechnik verwendet die Bereichsinformation eines Laserscanners und die Daten einer Monokamera, die durch Bildverarbeitungstechniken gewonnen werden. Bei der Bildverarbeitung werden verschiedene Erfassungsalgorithmen vorgeschlagen, wie z. B. Muster- und Farbdetektion. Eine Schätzung der Hindernisdimension und -distanz wird erklärt, um genaue Ergebnisse zu erzielen. Schließlich wird eine Datenfusion zur Hindernisbestimmung entwickelt, um diese Information im Optimalsteuerungsproblem als Pfadbeschränkung zu nutzen. Die erzielten Ergebnisse zeigen das Verhalten des mobilen Roboters bei Trajektorienverfolgungsund Hindernisvermeidungsproblemen, indem zwei verschiedene Abtastzeiten verglichen werden. Es wird gefolgert, dass der mobile Roboter die endgültige gewünschte Position erreicht, während die erkannten Hindernisse entlang der Trajektorie vermieden werden.Tesi

Human Detection and Segmentation via Multi-View Consensus

Self-supervised detection and segmentation of foreground objects aims for accuracy without annotated training data. However, existing approaches predominantly rely on restrictive assumptions on appearance and motion. For scenes with dynamic activities and camera motion, we propose a multi-camera framework in which geometric constraints are embedded in the form of multi-view consistency during training via coarse 3D localization in a voxel grid and fine-grained offset regression. In this manner, we learn a joint distribution of proposals over multiple views. At inference time, our method operates on single RGB images. We outperform state-of-the-art techniques both on images that visually depart from those of standard benchmarks and on those of the classical Human3.6M dataset

Information theoretic sensor management for multi-target tracking with a single pan-tilt-zoom camera

Automatic multiple target tracking with pan-tilt-zoom (PTZ) cameras is a hard task, with few approaches in the lit-erature, most of them proposing simplistic scenarios. In this paper, we present a PTZ camera management framework which lies on information theoretic principles: at each time step, the next camera pose (pan, tilt, focal length) is chosen, according to a policy which ensures maximum information gain. The formulation takes into account occlusions, phys-ical extension of targets, realistic pedestrian detectors and the mechanical constraints of the camera. Convincing com-parative results on synthetic data, realistic simulations and the implementation on a real video surveillance camera val-idate the effectiveness of the proposed method. 1