The objective of this work is to build virtual talking avatars of characters fully automatically from TV shows. From this unconstrained data, we show how to capture a character's style of speech, visual appearance and language in an e ort to construct an interactive avatar of the person and e ectively immortalize them in a computational model. We make three contributions (i) a complete framework for producing a generative model of the audiovisual and language of characters from TV shows; (ii) a novel method for aligning transcripts to video using the audio; and (iii) a fast audio segmentation system for silencing nonspoken audio from TV shows. Our framework is demonstrated using all 236 episodes from the TV series Friends [34] ( 97hrs of video) and shown to generate novel sentences as well as character specific speech and video

A Tejani

B Pepik

B Zheng

DB Fogel

E Brachmann

J Shotton

M Sun

MZ Zia

N Zhang

S Gupta

S Hinterstoisser

S Holzer

S Song

The objective of this work is to build virtual talking avatars of characters fully automatically from TV shows. From this unconstrained data, we show how to capture a character’s style of speech, visual appearance and language in an effort to construct an interactive avatar of the person and effectively immortalize them in a computational model. We make three contributions (i) a complete framework for producing a generative model of the audiovisual and language of characters from TV shows; (ii) a novel method for aligning transcripts to video using the audio; and (iii) a fast audio segmentation system for silencing non-spoken audio from TV shows. Our framework is demonstrated using all 236 episodes from the TV series Friends (≈ 97 h of video) and shown to generate novel sentences as well as character specific speech and video

Charles, J

Magee, D

Hogg, D

White Rose Research Online

Virtual Immortality: Reanimating Characters from TV Shows

PF Felzenszwalb

F Husain

V Kolmogorov

S Koo

C Liu

J Shi

D Tsai

Crossref

3D Point Cloud Video Segmentation Based on Interaction Analysis

Given the widespread availability of point cloud data from consumer depth sensors, 3D segmentation becomes a promising building block for high level applications such as scene understanding and interaction analysis. It benefits from the richer information contained in actual world 3D data compared to apparent (projected) data in 2D images. This also implies that the classical color segmentation challenges have recently shifted to RGBD data, whereas new emerging challenges are added as 3D information from depth measurements is usually noisy, sparse and unorganized. We present a novel segmentation approach for 3D point cloud video based on low level features and oriented to the analysis of object interactions. A hierarchical representation of the input point cloud is proposed to efficiently segment 3D data at the finer level, and to temporally establish the correspondence between segments, while dynamically managing the object split and merge at the coarser level. Experiments illustrate promising results and its potential application in object interaction analysis.Peer Reviewe

Lin, Xiao

Casas Pla, Josep Ramon

Pardàs Feliu, Montse

UPCommons

3D point cloud video segmentation based on interaction analysis

Given the widespread availability of point cloud data from consumer depth sensors, 3D segmentation becomes a promising building block for high level applications such as scene understanding and interaction analysis. It benefits from the richer information contained in actual world 3D data compared to apparent (projected) data in 2D images. This also implies that the classical color segmentation challenges have recently shifted to RGBD data, whereas new emerging challenges are added as 3D information from depth measurements is usually noisy, sparse and unorganized. We present a novel segmentation approach for 3D point cloud video based on low level features and oriented to the analysis of object interactions. A hierarchical representation of the input point cloud is proposed to efficiently segment 3D data at the finer level, and to temporally establish the correspondence between segments, while dynamically managing the object split and merge at the coarser level. Experiments illustrate promising results and its potential application in object interaction analysis.Postprint (author's final draft

UPCommons. Portal del coneixement obert de la UPC

English

This paper presents a novel method for single target tracking in RGB images under conditions of extreme clutter and camouflage, including frequent occlusions by objects with similar appearance as the target. In contrast to conventional single target trackers, which only maintain the estimated target status, we propose a multi-level clustering-based robust estimation for online detection and learning of multiple target-like regions, called distractors, when they appear near to the true target. To distinguish the target from these distractors, we exploit a global dynamic constraint (derived from the target and the distractors) in a feedback loop to improve single target tracking performance in situations where the target is camouflaged in highly cluttered scenes. Our proposed method successfully prevents the estimated target location from erroneously jumping to a distractor during occlusion or extreme camouflage interactions. To gain an insightful understanding of the evaluated trackers, we have augmented publicly available benchmark videos, by proposing a new set of clutter and camouflage sub-attributes, and annotating these sub-attributes for all frames in all sequences. Using this dataset, we first evaluate the effect of each key component of the tracker on the overall performance. Then, the proposed tracker is compared to other highly ranked single target tracking algorithms in the literature. The experimental results show that applying the proposed global dynamic constraint in a feedback loop can improve single target tracker performance, and demonstrate that the overall algorithm significantly outperforms other state-of-the-art single target trackers in highly cluttered scenes

Basevi, Hector

Leonardis, Ales

University of Birmingham Research Portal

Towards Categorization and Pose Estimation of Sets of Occluded Objects in Cluttered Scenes from Depth Data and Generic Object Models Using Joint Parsing

Given the widespread availability of point cloud data from consumer depth sensors, 3D segmentation becomes a promising building block for high level applications such as scene understanding and interaction analysis. It benefits from the richer information contained in actual world 3D data compared to apparent (projected) data in 2D images. This also implies that the classical color segmentation challenges have recently shifted to RGBD data, whereas new emerging challenges are added as 3D information from depth measurements is usually noisy, sparse and unorganized. We present a novel segmentation approach for 3D point cloud video based on low level features and oriented to the analysis of object interactions. A hierarchical representation of the input point cloud is proposed to efficiently segment 3D data at the finer level, and to temporally establish the correspondence between segments, while dynamically managing the object split and merge at the coarser level. Experiments illustrate promising results and its potential application in object interaction analysis

Given the widespread availability of point cloud data from consumer depth sensors, 3D segmentation becomes a promising building block for high level applications such as scene understanding and interaction analysis. It benefits from the richer information contained in actual world 3D data compared to apparent (projected) data in 2D images. This also implies that the classical color segmentation challenges have recently shifted to RGBD data, whereas new emerging challenges are added as 3D information from depth measurements is usually noisy, sparse and unorganized. We present a novel segmentation approach for 3D point cloud video based on low level features and oriented to the analysis of object interactions. A hierarchical representation of the input point cloud is proposed to efficiently segment 3D data at the finer level, and to temporally establish the correspondence between segments, while dynamically managing the object split and merge at the coarser level. Experiments illustrate promising results and its potential application in object interaction analysis.Peer ReviewedPostprint (author's final draft

Virtual Immortality: Reanimating Characters from TV Shows.

Abstract

Similar works

Full text

Available Versions

White Rose Research Online

Crossref

UPCommons

UPCommons. Portal del coneixement obert de la UPC

University of Birmingham Research Portal

UPCommons

UPCommons. Portal del coneixement obert de la UPC

University of Birmingham Research Portal

Crossref