With you – an experimental end-to-end telepresence system using video-based reconstruction

Abstract

We introduce withyou, our telepresence research platform. A systematic explanation of the theory brings together the linked nature of non-verbal communication and how it is influenced by technology. This leads to functional requirements for telepresence, in terms of the balance of visual, spatial and temporal qualities. The first end-to-end description of withyou describes all major processes and the display and capture environment. This includes two approaches to reconstructing the human form in 3D, from live video. An unprecedented characterization of our approach is given in terms of the above qualities, and influences of approach. This leads to non-functional requirements in terms of number and place of cameras and the avoidance of a resultant bottlekneck. Proposals are given for improved distribution of processes across networks, computers, and multi-core CPU and GPU. Simple conservative estimation shows that both approaches should meet our requirements. One is implemented and shown to meet minimum and come close to desirable requirements