Space-variant picture coding

PhDSpace-variant picture coding techniques exploit the strong spatial non-uniformity of the human visual system in order to increase coding efficiency in terms of perceived quality per bit. This thesis extends space-variant coding research in two directions. The first of these directions is in foveated coding. Past foveated coding research has been dominated by the single-viewer, gaze-contingent scenario. However, for research into the multi-viewer and probability-based scenarios, this thesis presents a missing piece: an algorithm for computing an additive multi-viewer sensitivity function based on an established eye resolution model, and, from this, a blur map that is optimal in the sense of discarding frequencies in least-noticeable- rst order. Furthermore, for the application of a blur map, a novel algorithm is presented for the efficient computation of high-accuracy smoothly space-variant Gaussian blurring, using a specialised filter bank which approximates perfect space-variant Gaussian blurring to arbitrarily high accuracy and at greatly reduced cost compared to the brute force approach of employing a separate low-pass filter at each image location. The second direction is that of artifi cially increasing the depth-of- field of an image, an idea borrowed from photography with the advantage of allowing an image to be reduced in bitrate while retaining or increasing overall aesthetic quality. Two synthetic depth of field algorithms are presented herein, with the desirable properties of aiming to mimic occlusion eff ects as occur in natural blurring, and of handling any number of blurring and occlusion levels with the same level of computational complexity. The merits of this coding approach have been investigated by subjective experiments to compare it with single-viewer foveated image coding. The results found the depth-based preblurring to generally be significantly preferable to the same level of foveation blurring

Content-prioritised video coding for British Sign Language communication.

Video communication of British Sign Language (BSL) is important for remote interpersonal communication and for the equal provision of services for deaf people. However, the use of video telephony and video conferencing applications for BSL communication is limited by inadequate video quality. BSL is a highly structured, linguistically complete, natural language system that expresses vocabulary and grammar visually and spatially using a complex combination of facial expressions (such as eyebrow movements, eye blinks and mouth/lip shapes), hand gestures, body movements and finger-spelling that change in space and time. Accurate natural BSL communication places specific demands on visual media applications which must compress video image data for efficient transmission. Current video compression schemes apply methods to reduce statistical redundancy and perceptual irrelevance in video image data based on a general model of Human Visual System (HVS) sensitivities. This thesis presents novel video image coding methods developed to achieve the conflicting requirements for high image quality and efficient coding. Novel methods of prioritising visually important video image content for optimised video coding are developed to exploit the HVS spatial and temporal response mechanisms of BSL users (determined by Eye Movement Tracking) and the characteristics of BSL video image content. The methods implement an accurate model of HVS foveation, applied in the spatial and temporal domains, at the pre-processing stage of a current standard-based system (H.264). Comparison of the performance of the developed and standard coding systems, using methods of video quality evaluation developed for this thesis, demonstrates improved perceived quality at low bit rates. BSL users, broadcasters and service providers benefit from the perception of high quality video over a range of available transmission bandwidths. The research community benefits from a new approach to video coding optimisation and better understanding of the communication needs of deaf people

Perception-driven approaches to real-time remote immersive visualization

In remote immersive visualization systems, real-time 3D perception through RGB-D cameras, combined with modern Virtual Reality (VR) interfaces, enhances the user’s sense of presence in a remote scene through 3D reconstruction rendered in a remote immersive visualization system. Particularly, in situations when there is a need to visualize, explore and perform tasks in inaccessible environments, too hazardous or distant. However, a remote visualization system requires the entire pipeline from 3D data acquisition to VR rendering satisfies the speed, throughput, and high visual realism. Mainly when using point-cloud, there is a fundamental quality difference between the acquired data of the physical world and the displayed data because of network latency and throughput limitations that negatively impact the sense of presence and provoke cybersickness. This thesis presents state-of-the-art research to address these problems by taking the human visual system as inspiration, from sensor data acquisition to VR rendering. The human visual system does not have a uniform vision across the field of view; It has the sharpest visual acuity at the center of the field of view. The acuity falls off towards the periphery. The peripheral vision provides lower resolution to guide the eye movements so that the central vision visits all the interesting crucial parts. As a first contribution, the thesis developed remote visualization strategies that utilize the acuity fall-off to facilitate the processing, transmission, buffering, and rendering in VR of 3D reconstructed scenes while simultaneously reducing throughput requirements and latency. As a second contribution, the thesis looked into attentional mechanisms to select and draw user engagement to specific information from the dynamic spatio-temporal environment. It proposed a strategy to analyze the remote scene concerning the 3D structure of the scene, its layout, and the spatial, functional, and semantic relationships between objects in the scene. The strategy primarily focuses on analyzing the scene with models the human visual perception uses. It sets a more significant proportion of computational resources on objects of interest and creates a more realistic visualization. As a supplementary contribution, A new volumetric point-cloud density-based Peak Signal-to-Noise Ratio (PSNR) metric is proposed to evaluate the introduced techniques. An in-depth evaluation of the presented systems, comparative examination of the proposed point cloud metric, user studies, and experiments demonstrated that the methods introduced in this thesis are visually superior while significantly reducing latency and throughput

Methods and Apparatus for Autonomous Robotic Control

Sensory processing of visual, auditory, and other sensor information (e.g., visual imagery, LIDAR, RADAR) is conventionally based on "stovepiped," or isolated processing, with little interactions between modules. Biological systems, on the other hand, fuse multi-sensory information to identify nearby objects of interest more quickly, more efficiently, and with higher signal-to-noise ratios. Similarly, examples of the OpenSense technology disclosed herein use neurally inspired processing to identify and locate objects in a robot's environment. This enables the robot to navigate its environment more quickly and with lower computational and power requirements

Visual Attention in Dynamic Environments and its Application to Playing Online Games

Abstract In this thesis we present a prototype of Cognitive Programs (CPs) - an executive controller built on top of Selective Tuning (ST) model of attention. CPs enable top-down control of visual system and interaction between the low-level vision and higher-level task demands. Abstract We implement a subset of CPs for playing online video games in real time using only visual input. Two commercial closed-source games - Canabalt and Robot Unicorn Attack - are used for evaluation. Their simple gameplay and minimal controls put the emphasis on reaction speed and attention over planning. Abstract Our implementation of Cognitive Programs plays both games at human expert level, which experimentally proves the validity of the concept. Additionally we resolved multiple theoretical and engineering issues, e.g. extending the CPs to dynamic environments, finding suitable data structures for describing the task and information flow within the network and determining the correct timing for each process

Long Range Automated Persistent Surveillance

This dissertation addresses long range automated persistent surveillance with focus on three topics: sensor planning, size preserving tracking, and high magnification imaging. field of view should be reserved so that camera handoff can be executed successfully before the object of interest becomes unidentifiable or untraceable. We design a sensor planning algorithm that not only maximizes coverage but also ensures uniform and sufficient overlapped camera’s field of view for an optimal handoff success rate. This algorithm works for environments with multiple dynamic targets using different types of cameras. Significantly improved handoff success rates are illustrated via experiments using floor plans of various scales. Size preserving tracking automatically adjusts the camera’s zoom for a consistent view of the object of interest. Target scale estimation is carried out based on the paraperspective projection model which compensates for the center offset and considers system latency and tracking errors. A computationally efficient foreground segmentation strategy, 3D affine shapes, is proposed. The 3D affine shapes feature direct and real-time implementation and improved flexibility in accommodating the target’s 3D motion, including off-plane rotations. The effectiveness of the scale estimation and foreground segmentation algorithms is validated via both offline and real-time tracking of pedestrians at various resolution levels. Face image quality assessment and enhancement compensate for the performance degradations in face recognition rates caused by high system magnifications and long observation distances. A class of adaptive sharpness measures is proposed to evaluate and predict this degradation. A wavelet based enhancement algorithm with automated frame selection is developed and proves efficient by a considerably elevated face recognition rate for severely blurred long range face images

효율적 영상처리를 위한 주의집중 샘플링

학위논문 (박사)-- 서울대학교 대학원 : 전기·컴퓨터공학부, 2013. 2. 최진영.컴퓨터 비전 문제는 영상 획득 장치를 통해 픽셀 단위로 수치화된 데이터를 샘플링 하는 것으로부터 시작된다. 가장 기본이 되는 데이터인 픽셀 값들을 그대로 사용하는 경우도 있고, 이 픽셀 값들을 조합하여 새로운 의미를 가진 데이터들을 구성하고 샘플링 하여 사용하기도 한다. 좋은 성능을 얻기 위해서는 최대한 많은 수의 데이터를 샘플링 하는 것이 필요하지만 이럴 경우 필요로 하는 연산량이 급격히 증가하는 문제가 있다. 반대로 연산량 만을 고려해 최소한의 데이터만 샘플링 하여 사용하는 경우 좋은 성능을 기대하기 어렵다. 그러므로 효율적인 연산량으로 최적의 성능을 얻기 위해서는, 이미지가 바뀜에 따라 혹은 시간이 흐름에 따라 문제를 풀기에 충분한 최소한의 데이터만 찾아내어 샘플링 하는 능동 샘플링(active sampling) 개념이 필요하다. 이러한 능동 샘플링 개념을 현실화하기 위해서는 문제를 해결하는데 중요한 데이터들을 찾아내는 과정이 매우 중요하며, 찾아낸 데이터들을 어떻게 집중하여 샘플링 하는가가 중요해진다. 본 논문에서는 서로 다른 세 가지의 주의집중 샘플링(attentional sampling) 방법, 즉 구조적 주의집중 샘플링(structured attentional sampling), 경험적 주의집중 샘플링(empirical attentional sampling), 선택적 주의집중 샘플링(selective attentional sampling)을 제안하였다. 제안된 각각의 주의집중 샘플링 방법들은 주의집중이 필요한 중요 데이터들을 찾기 위해 문제의 특성에 대한 사전 지식(prior knowledge)을 적용하는 세가지 방법을 제안하고 있으며, 그에 따라 적응적으로 샘플링 하는 방법들이다. 제안된 주의집중 샘플링 방법들은 컴퓨터 비전 문제들에 성공적으로 적용되어 연산 효율뿐만 아니라 각 알고리즘의 성능을 크게 향상 시켰다. 첫 번째 구조적 주의집중 샘플링(structured attentional sampling)은 문제의 특성에 맞춰 미리 구조화된 샘플링 패턴에 따라 샘플링을 수행하는 방법이다. 이러한 구조적 주의집중 샘플링 방법을 사람 눈의 구조를 흉내 내어 물체 추적 실패를 탐지하는 데 적용하였다. 사람 눈 망막 위의 시신경 세포(ganglion cells)의 분포를 근사화한 log-polar 패턴 구조로 이미지 픽셀 샘플링을 수행하여 사람 눈의 유용한 특성을 흉내 내었다. Log-polar 패턴으로 샘플링 된 이미지는 회전(rotation) 변화에 의한 영향은 감소되어 나타나고, 좌우나 위아래로의 병진(translation) 변화는 증폭되어 나타나는 특성이 있다. 이러한 특성은 회전에 의해 나타나는 포즈 변화들로 인해 발생하는 추적 실패에 대한 거짓 경보(false alarm)들은 줄이고, 급격한 위치 변화로 인한 추적 실패에 대한 참 경보(true alarm)를 증가시킬 수 있다. 게다가 log-polar 구조의 특징인 중심와(fovea) 선명화 특성(predominant property)은 초점이 맞춰진 중심 부분(추적 물체의 중심 부분)의 선명도는 증가시키고 그 이외의 주변부(추적 물체 바깥 부분)는 흐릿하게 함으로써 추적 실패의 순간을 정확하게 탐지할 수 있도록 도와준다. 또한 망막 위의 시신경 세포 하나하나는 log-polar 변환 이미지의 각 픽셀에 대응시켜, 각 세포가 빛에 적응하는 방식과 유사하게 각 픽셀의 추적 물체의 색상에 대한 적응을 가우시안 혼합 모델(Gaussian mixture model)을 이용하여 모델링 하였다. 이러한 방식으로 제안된 추적 실패 탐지를 위한 구조적 주의집중 샘플링의 유용성은 다양한 실험을 통해 검증되었다. 두 번째 경험적 주의집중 샘플링(empirical attentional sampling)은 이전에 획득된 경험적 지식을 현재 단계 샘플링에 사용하는 방식이다. 경험적 지식은 경험 학습 과정을 통하여 확률 분포로 모델링 된다. 이러한 경험적 샘플링 개념은 움직이는 물체 탐지를 위해 일반적으로 사용되는 배경 제거 방법들에 픽셀 단위의 선택적 연산 마스크를 적용하여 연산 속도를 향상시키는 방식으로 적용되었다. 제안된 샘플링 방법은 전경 지역(foreground region)과 같이 주의집중을 필요로 하는 영역에 초점이 맞춰져 샘플링이 진행되도록 설계되었다. 주의집중 영역은 전경 확률 지도(foreground probability map)로 표현되고, 이 확률 지도는 이전 프레임에서의 탐지 결과를 이용하여 재귀적(recursive) 확률 업데이트 방식으로 추정된다. 전경 확률 지도는 전경 부분의 시간적(temporal), 공간적(spatial), 주파수적(frequency) 특성을 이용하여 생성되었다. 생성된 전경 확률 지도를 이용하여, 무작위 샘플링(randomly scattered sampling), 공간 확장 방식의 중요 샘플링(spatially expanding importance sampling), 놀람 픽셀 샘플링(surprise pixel sampling) 방법들이 순차적으로 진행되면서 주의집중 샘플링 마스크를 생성한다. 제안된 경험적 주의집중 샘플링 방법의 효율성은 다양한 실험을 통해 검증되었다. 제안된 방법은 기존의 픽셀 단위의 배경 제거 방법의 연산 속도를 탐지 성능 저하 없이 약 6.6배 향상 시켰다. 또한 기존의 배경 제거 알고리즘을 이용하여 full HD 영상(1920x1080)에서 실시간으로 움직이는 물체를 탐지할 수 있도록 하였다. 선택적 주의집중 샘플링(selective attentional sampling)은 주어진 데이터와 목적에 대한 사전 정보를 이용하여 문제의 해결을 위해 꼭 필요로 하는 중요 데이터만 미리 선택하여 문제 해결의 효율성을 높이는 방식이다. 본 논문에서는 이러한 선택적 샘플링 방식을 이용하여 일반인이 추는 유명 대중가요의 춤을 인식하는 방법을 제안하였다. 대중가요 춤은 일반적으로, 발레나 리듬 체조의 춤 동작과는 달리 하나하나를 따로 이름을 붙일 수 없는 짧고 복잡하며 다양한 행동의 연속으로 나타난다. 특히 춤에 대한 일정한 제약이 없다 보니, 동작의 정확성 보다는 추는 사람의 개성과 자유로움에 따라 동일한 춤도 다양하게 표현이 된다. 이러한 행동의 자유로움과 다양함, 그리고 시간적으로 긴 행동의 길이 때문에 기존의 행동 인식 알고리즘은 직접적으로 적용할 수 없다. 본 논문에서는 명확하게 구분할 수 없을 정도로 자유로운 행동의 흐름 특징을 효과적으로 표현하고 인식 알고리즘에 적용할 수 있도록 하기 위해 새로운 행동 특징 표현 방법을 제안하고, 이를 효과적으로 낮은 차원 데이터로 표현하는 방법을 제안하였다. 또한 효율적인 인식을 위해 특징적인 시공간적 행동의 변화 지점을 주의집중적 행동 지점(attentional motion spot)라 명명하고 이를 자동을 선택하는 방법을 제안하였다. 이 특징 점들의 시공간적 분포를 혼합 가우시안(Gaussian) 분포로 모델링하고, 이렇게 표현된 모델링 방법을 행동 악보(Action Chart)라고 명명하였다. 이 행동 악보는 시공간적인 행동의 흐름을 음악 악보처럼 중요 행동의 시간적 발생 지점과 종류, 지속 시간을 표현하고 있다. 이렇게 표현된 행동 악보를 이용하여 새롭게 제작된 대중 가요 춤 데이터 세트를 효율적이고 효과적으로 인식하였다. 제안된 방법을 검증하기 위하여 제안된 방법을 구성하는 세부 알고리즘 하나하나를 실험적으로 검증하여 각 부분의 필요성을 보였고, 현재 존재하는 길고 복잡한 행동을 인식하는 방법을 직접 구현하여 동일한 데이터 세트를 이용하여 제안된 방법이 인식 성능과 연산 시간측면에서 월등히 뛰어남을 검증하였다. 또한 더 나아가 행동 악보를 이용하면 긴 춤 동작을 사람이 하는 것과 거의 유사한 성능으로 요약 가능함을 보였다.In many practical computer vision scenarios it is possible to use information gleaned from the previous observations through the sampling process. In order to achieve a good performance with small computation, it is desirable that the samples cover the domain of target distribution with the small number of samples as possible via a concept of active or adaptive sampling. Based on the active sampling strategy, sampling could be concentrated on attentional portions, which can improve not only the sampling efficiency but also performances of algorithms. In this thesis, we define three different attentional sampling concepts, structured attentional sampling, empirical attentional sampling and selective attentional sampling. The proposed attentional sampling methods are successfully applied to computer vision problems, by achieving dramatic improvement in the sense of performance as well as computational load. The structured attentional sampling scheme uses an inherent structure to sample an interesting region densely instead of equally distributed sampling over the entire region. This sampling scheme is applied to a tracking failure detection method by imitating human visual system. In this scheme, we adopt a sampling structure based on Log-polar transformation simulating retina structure. Since the log-polar structure shows invariance against rotational changes and intensifies translational changes, it helps to reduce false alarms arising from rotational pose variations and increase true alarms in abrupt translational changes. In addition, foveal predominant property of log-polar structure helps to detect the tracking failing moment by amplifying the resolution around focus (tracking box center) and blurring the peripheries. Each ganglion cell corresponds to a pixel of log-polar image, and its adaptation is modeled as Gaussian mixture model. The validity of the structured attentional sampling method is illustrated through various experiments. The empirical attentional sampling scheme uses previously obtained empirical knowledge when sampling in current time. The empirical knowledge is modeled by a probability distribution function through an empirical learning process. This empirical sampling scheme is applied to mask generation to speed up conventional background subtraction algorithms for moving object detection. The proposed sampling strategy is designed to focus on attentional region such as foreground regions. The attentional region is estimated by using the detection results in the previous frame in a recursive probabilistic way. We generate a foreground probability map by using foreground properties of temporal, spatial, and frequency properties. Based on this foreground probability map, randomly scattered sampling, spatially expanding importance sampling and surprise pixel sampling are performed sequentially to make the attention sampling mask. The efficiency of the proposed empirical attention sampling method is shown through various experiments. The proposed masking method successfully speeds up pixel-wise background subtraction methods approximately 6.6 times without deteriorating detection performance. Also real-time detection with Full HD video is successfully achieved by various conventional background subtraction algorithms together with the proposed sampling scheme. The selective attentional sampling scheme does not use whole data but selects only important data enough to achieve a given classification objective. This selective sampling scheme is applied to the recognition of pop dances. Pop dances are action streams consisting of diverse actions which cannot be simply annotated. For such ``unannotatable'' action streams, conventional methods cannot be applied directly due to their complexity and longevity. In order to describe unannotatable action stream effectively, the proposed method employs a novel mid-level ``feature flow'' with low dimensional embedding. Also, for the purpose of recognition, ``attentional motion spots'' holding important information about the sequence are automatically selected. The feature values and the temporal locations of each attentional motion spot are modeled with Gaussian mixtures as ``Action Charts.'' The Action Chart describes the characteristics of an action stream in the spatio-temporal domain. Using the abstract information in the Action Charts, the proposed method efficiently recognizes pop dance sequences. In order to demonstrate the validity of the proposed method, we compare our method against the state-of-the-art methods with a newly built SNU Pop-Dance dataset containing long action streams composed of diverse actions.1 Introduction 1 1.1 Motivations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Contents of Research . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2.1 Structured Attentional Sampling . . . . . . . . . . . . . . . 3 1.2.2 Empirical Attentional Sampling . . . . . . . . . . . . . . . . 5 1.2.3 Selective Attentional Sampling . . . . . . . . . . . . . . . . 6 2 Structured Attentional Sampling for Tracking Failure Detection 8 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.2 Characteristics of Log-Polar Image and Tracking Failure . . . . . . 11 2.2.1 Properties of Log-Polar Image . . . . . . . . . . . . . . . . 11 2.2.2 Tracking Failure in Log-Polar Image . . . . . . . . . . . . . 12 2.3 Tracking Failure Detection Algorithm . . . . . . . . . . . . . . . . 14 2.3.1 Modeling of Ganglion Cell Adaptation . . . . . . . . . . . . 14 2.3.2 Initialization of GMM . . . . . . . . . . . . . . . . . . . . . 16 2.3.3 Tracking Failure Detection . . . . . . . . . . . . . . . . . . 16 2.4 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.4.1 Eectiveness of Log-Polar Transformation and Initial Color Model Generation . . . . . . . . . . . . . . . . . . . . . . . 19 2.4.2 Combining with various tracking algorithms . . . . . . . . . 20 2.5 Final Remarks and Discussion . . . . . . . . . . . . . . . . . . . . . 23 3 Empirical Attentional Sampling for Speed-up of Background Sub- traction 24 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.2 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.2.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.2.2 Overall Scheme of Proposed Algorithm . . . . . . . . . . . . 28 3.3 Foreground Probability Map Generation . . . . . . . . . . . . . . . 31 3.3.1 Estimation of Foreground Properties . . . . . . . . . . . . . 31 3.3.2 Foreground Probability Map: PFG . . . . . . . . . . . . . . 32 3.4 Active Sampling Mask Generation . . . . . . . . . . . . . . . . . . 33 3.4.1 Randomly Scattered Sampling . . . . . . . . . . . . . . . . 33 3.4.2 Spatially Expanding Importance Sampling . . . . . . . . . . 36 3.4.3 Surprise Pixel Sampling Mask . . . . . . . . . . . . . . . . . 39 3.5 Computational Eciency Boundary . . . . . . . . . . . . . . . . . 39 3.6 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . 40 3.6.1 Eciency of Active Attentional Sampling . . . . . . . . . . 42 3.6.2 Detection Performance Comparison . . . . . . . . . . . . . 43 3.6.3 Speed-up Performance Comparison . . . . . . . . . . . . . . 45 3.6.4 Real-time Detection in Full HD Video . . . . . . . . . . . . 49 3.7 Final Remarks and Discussion . . . . . . . . . . . . . . . . . . . . . 49 4 Selective Attentional Sampling for Recognition of Pop Dances 51 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 4.2 Action Chart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 4.2.1 Motion Feature Flow (MFF) . . . . . . . . . . . . . . . . . 58 4.2.2 Hierarchical Low Dimensional Embedding . . . . . . . . . . 62 4.2.3 Attentional Motion Spot Selection . . . . . . . . . . . . . . 64 4.2.4 Action Chart Generation and Recognition . . . . . . . . . . 65 4.3 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . 67 4.3.1 Pop-Dance Dataset . . . . . . . . . . . . . . . . . . . . . . . 67 4.3.2 Validation of Proposed Features . . . . . . . . . . . . . . . 71 4.3.3 Recognition Performance . . . . . . . . . . . . . . . . . . . 73 4.3.4 Automatic Action Abstraction . . . . . . . . . . . . . . . . 78 4.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 5 Concluding Remarks 82 Bibliography 85 A Derivation of Computational Eciency Boundaries 95 A.1 Denition of Notations . . . . . . . . . . . . . . . . . . . . . . . . . 95 A.2 Derivative of the eciency boundary . . . . . . . . . . . . . . . . . 96 Abstract in Korean 102Docto

BEYOND MULTI-TARGET TRACKING: STATISTICAL PATTERN ANALYSIS OF PEOPLE AND GROUPS

Ogni giorno milioni e milioni di videocamere monitorano la vita quotidiana delle persone, registrando e collezionando una grande quantit\ue0 di dati. Questi dati possono essere molto utili per scopi di video-sorveglianza: dalla rilevazione di comportamenti anomali all'analisi del traffico urbano nelle strade. Tuttavia i dati collezionati vengono usati raramente, in quanto non \ue8 pensabile che un operatore umano riesca a esaminare manualmente e prestare attenzione a una tale quantit\ue0 di dati simultaneamente. Per questo motivo, negli ultimi anni si \ue8 verificato un incremento della richiesta di strumenti per l'analisi automatica di dati acquisiti da sistemi di video-sorveglianza in modo da estrarre informazione di pi\uf9 alto livello (per esempio, John, Sam e Anne stanno camminando in gruppo al parco giochi vicino alla stazione) a partire dai dati a disposizione che sono solitamente a basso livello e ridondati (per esempio, una sequenza di immagini). L'obiettivo principale di questa tesi \ue8 quello di proporre soluzioni e algoritmi automatici che permettono di estrarre informazione ad alto livello da una zona di interesse che viene monitorata da telecamere. Cos\uec i dati sono rappresentati in modo da essere facilmente interpretabili e analizzabili da qualsiasi persona. In particolare, questo lavoro \ue8 focalizzato sull'analisi di persone e i loro comportamenti sociali collettivi. Il titolo della tesi, beyond multi-target tracking, evidenzia lo scopo del lavoro: tutti i metodi proposti in questa tesi che si andranno ad analizzare hanno come comune denominatore il target tracking. Inoltre andremo oltre le tecniche standard per arrivare a una rappresentazione del dato a pi\uf9 alto livello. Per prima cosa, analizzeremo il problema del target tracking in quanto \ue8 alle basi di questo lavoro. In pratica, target tracking significa stimare la posizione di ogni oggetto di interesse in un immagine e la sua traiettoria nel tempo. Analizzeremo il problema da due prospettive complementari: 1) il punto di vista ingegneristico, dove l'obiettivo \ue8 quello di creare algoritmi che ottengono i risultati migliori per il problema in esame. 2) Il punto di vista della neuroscienza: motivati dalle teorie che cercano di spiegare il funzionamento del sistema percettivo umano, proporremo in modello attenzionale per tracking e il riconoscimento di oggetti e persone. Il secondo problema che andremo a esplorare sar\ue0 l'estensione del tracking alla situazione dove pi\uf9 telecamere sono disponibili. L'obiettivo \ue8 quello di mantenere un identificatore univoco per ogni persona nell'intera rete di telecamere. In altre parole, si vuole riconoscere gli individui che vengono monitorati in posizioni e telecamere diverse considerando un database di candidati. Tale problema \ue8 chiamato in letteratura re-indetificazione di persone. In questa tesi, proporremo un modello standard di come affrontare il problema. In questo modello, presenteremo dei nuovi descrittori di aspetto degli individui, in quanto giocano un ruolo importante allo scopo di ottenere i risultati migliori. Infine raggiungeremo il livello pi\uf9 alto di rappresentazione dei dati che viene affrontato in questa tesi, che \ue8 l'analisi di interazioni sociali tra persone. In particolare, ci focalizzeremo in un tipo specifico di interazione: il raggruppamento di persone. Proporremo dei metodi di visione computazionale che sfruttano nozioni di psicologia sociale per rilevare gruppi di persone. Inoltre, analizzeremo due modelli probabilistici che affrontano il problema di tracking (congiunto) di gruppi e individui.Every day millions and millions of surveillance cameras monitor the world, recording and collecting huge amount of data. The collected data can be extremely useful: from the behavior analysis to prevent unpleasant events, to the analysis of the traffic. However, these valuable data is seldom used, because of the amount of information that the human operator has to manually attend and examine. It would be like looking for a needle in the haystack. The automatic analysis of data is becoming mandatory for extracting summarized high-level information (e.g., John, Sam and Anne are walking together in group at the playground near the station) from the available redundant low-level data (e.g., an image sequence). The main goal of this thesis is to propose solutions and automatic algorithms that perform high-level analysis of a camera-monitored environment. In this way, the data are summarized in a high-level representation for a better understanding. In particular, this work is focused on the analysis of moving people and their collective behaviors. The title of the thesis, beyond multi-target tracking, mirrors the purpose of the work: we will propose methods that have the target tracking as common denominator, and go beyond the standard techniques in order to provide a high-level description of the data. First, we investigate the target tracking problem as it is the basis of all the next work. Target tracking estimates the position of each target in the image and its trajectory over time. We analyze the problem from two complementary perspectives: 1) the engineering point of view, where we deal with problem in order to obtain the best results in terms of accuracy and performance. 2) The neuroscience point of view, where we propose an attentional model for tracking and recognition of objects and people, motivated by theories of the human perceptual system. Second, target tracking is extended to the camera network case, where the goal is to keep a unique identifier for each person in the whole network, i.e., to perform person re-identification. The goal is to recognize individuals in diverse locations over different non-overlapping camera views or also the same camera, considering a large set of candidates. In this context, we propose a pipeline and appearance-based descriptors that enable us to define in a proper way the problem and to reach the-state-of-the-art results. Finally, the higher level of description investigated in this thesis is the analysis (discovery and tracking) of social interaction between people. In particular, we focus on finding small groups of people. We introduce methods that embed notions of social psychology into computer vision algorithms. Then, we extend the detection of social interaction over time, proposing novel probabilistic models that deal with (joint) individual-group tracking

Enhancing Perception and Immersion in Pre-Captured Environments through Learning-Based Eye Height Adaptation

Pre-captured immersive environments using omnidirectional cameras provide a wide range of virtual reality applications. Previous research has shown that manipulating the eye height in egocentric virtual environments can significantly affect distance perception and immersion. However, the influence of eye height in pre-captured real environments has received less attention due to the difficulty of altering the perspective after finishing the capture process. To explore this influence, we first propose a pilot study that captures real environments with multiple eye heights and asks participants to judge the egocentric distances and immersion. If a significant influence is confirmed, an effective image-based approach to adapt pre-captured real-world environments to the user's eye height would be desirable. Motivated by the study, we propose a learning-based approach for synthesizing novel views for omnidirectional images with altered eye heights. This approach employs a multitask architecture that learns depth and semantic segmentation in two formats, and generates high-quality depth and semantic segmentation to facilitate the inpainting stage. With the improved omnidirectional-aware layered depth image, our approach synthesizes natural and realistic visuals for eye height adaptation. Quantitative and qualitative evaluation shows favorable results against state-of-the-art methods, and an extensive user study verifies improved perception and immersion for pre-captured real-world environments.Comment: 10 pages, 13 figures, 3 tables, submitted to ISMAR 202

マルチタスク学習を用いたシーン理解とデータ拡張による複合現実感の向上

早大学位記番号:新9140早稲田大