Network streaming and compression for mixed reality tele-immersion

Bulterman, D.C.A. [Promotor]Cesar, P.S. [Copromotor

Enabling geometry-based 3-D tele-immersion with fast mesh compression and linear rateless coding

3-D tele-immersion (3DTI) enables participants in remote locations to share, in real time, an activity. It offers users interactive and immersive experiences, but it challenges current media-streaming solutions. Work in the past has mainly focused on the efficient delivery of image-based 3-D videos and on realistic rendering and reconstruction of geometry-based 3-D objects. The contribution of this paper is a real-time streaming component for 3DTI with dynamic reconstructed geometry. This component includes both a novel fast compression method and a rateless packet protection scheme specifically designed towards the requirements imposed by real time transmission of live-reconstructed mesh geometry. Tests on a large dataset show an encoding speed-up up to ten times at comparable compression ratio and quality, when compared with the high-end MPEG-4 SC3DMC mesh encoders. The implemented rateless code ensures complete packet loss protection of the triangle mesh object and a delivery delay within interactive bounds. Contrary to most linear fountain codes, the designed codec enables real-time progressive decoding allowing partial decoding each time a packet is received. This approach is compared with transmission over TCP in packet loss rates and latencies, typical in managed WAN and MAN networks, and heavily outperforms it in terms of end-to-end delay. The streaming component has been integrated into a larger 3DTI environment that includes state of the art 3-D reconstruction and rendering modules. This resulted in a prototype that can capture, compress transmit, and render triangle mesh geometry in real-time in realistic internet conditions as shown in experiments. Compared with alternative methods, lower interactive end-to-end delay and frame rates over three times higher are achieved

Multi-Stream Management for Supporting Multi-Party 3D Tele-Immersive Environments

Three-dimensional tele-immersive (3DTI) environments have great potential to promote collaborative work among geographically distributed participants. However, extensive application of 3DTI environments is still hindered by problems pertaining to scalability, manageability and reliance of special-purpose components. Thus, one critical question is how to organize the acquisition, transmission and display of large volume real-time 3D visual data over commercially available computing and networking infrastructures so that .everybody. would be able to install and enjoy 3DTI environments for high quality tele-collaboration. In the thesis, we explore the design space from the angle of multi-stream Quality-of-Service (QoS) management to support multi-party 3DTI communication. In 3DTI environments, multiple correlated 3D video streams are deployed to provide a comprehensive representation of the physical scene. Traditional QoS approach in 2D and single-stream scenario has become inadequate. On the other hand, the existence of multiple streams provides unique opportunity for QoS provisioning. We propose an innovative cross-layer hierarchical and distributed multi-stream management middleware framework for QoS provisioning to fully enable multi-party 3DTI communication over general delivery infrastructure. The major contributions are as follows. First, we introduce the view model for representing the user interest in the application layer. The design revolves around the concept of view-aware multi-stream coordination, which leverages the central role of view semantics in 3D video systems. Second, in the stream differentiation layer we present the design of view to stream mapping, where a subset of relevant streams are selected based on the relative importance of each stream to the current view. Conventional streaming controllers focus on a fixed set of streams specified by the application. Different from all the others, in our management framework the application layer only specifies the view information while the underlying controller dynamically determines the set of streams to be managed. Third, in the stream coordination layer we present two designs applicable in different situations. In the case of end-to-end 3DTI communication, a learning-based controller is embedded which provides bandwidth allocation for relevant streams. In the case of multi-party 3DTI communication, we propose a novel ViewCast protocol to coordinate the multi-stream content dissemination upon an end-system overlay network

07171 Abstracts Collection -- Visual Computing -- Convergence of Computer Graphics and Computer Vision

From 22.04. to 27.04.2007, the Dagstuhl Seminar 07171 ``Visual Computing - Convergence of Computer Graphics and Computer Vision\u27\u27 was held in the International Conference and Research Center (IBFI), Schloss Dagstuhl. During the seminar, several participants presented their current research, and ongoing work and open problems were discussed. Abstracts of the presentations given during the seminar as well as abstracts of seminar results and ideas are put together in this paper. The first section describes the seminar topics and goals in general. Links to extended abstracts or full papers are provided, if available

Source coding for transmission of reconstructed dynamic geometry: a rate-distortion-complexity analysis of different approaches

Live 3D reconstruction of a human as a 3D mesh with commodity electronics is becoming a reality. Immersive applications (i.e. cloud gaming, tele-presence) benefit from effective transmission of such content over a bandwidth limited link. In this paper we outline different approaches for compressing live reconstructed mesh geometry based on distributing mesh reconstruction functions between sender and receiver. We evaluate rate-performance-complexity of different configurations. First, we investigate 3D mesh compression methods (i.e. dynamic/static) from MPEG-4. Second, we evaluate the option of using octree based point cloud compression and receiver side surface reconstruction

Archiving and Delivery of 3DTI Rehabilitation Sessions

In this paper we present CyPhy: a cyber-physiotherapy system that brings daily rehabilitation to patient’s home with supervision from trained therapist. With its archiving and delivery features, CyPhy is able to 1) capture and record RGB-D and physiotherapy-related medical sensing data streams in home environment; 2) provide efficient storage for rehabilitation session recordings; 3) provide fast metadata analysis over stored sessions for review recommendation; 4) adaptively deliver rehabilitation session under different networking capabilities; 5) support smooth viewpoint changing during 3D video streaming with scene rendering schemes tailored for devices with different bandwidth and power limitations; and 6) provide platform-independent streaming client for various mobile and PC environments