Loss-resilient Coding of Texture and Depth for Free-viewpoint Video Conferencing

Free-viewpoint video conferencing allows a participant to observe the remote 3D scene from any freely chosen viewpoint. An intermediate virtual viewpoint image is commonly synthesized using two pairs of transmitted texture and depth maps from two neighboring captured viewpoints via depth-image-based rendering (DIBR). To maintain high quality of synthesized images, it is imperative to contain the adverse effects of network packet losses that may arise during texture and depth video transmission. Towards this end, we develop an integrated approach that exploits the representation redundancy inherent in the multiple streamed videos a voxel in the 3D scene visible to two captured views is sampled and coded twice in the two views. In particular, at the receiver we first develop an error concealment strategy that adaptively blends corresponding pixels in the two captured views during DIBR, so that pixels from the more reliable transmitted view are weighted more heavily. We then couple it with a sender-side optimization of reference picture selection (RPS) during real-time video coding, so that blocks containing samples of voxels that are visible in both views are more error-resiliently coded in one view only, given adaptive blending will erase errors in the other view. Further, synthesized view distortion sensitivities to texture versus depth errors are analyzed, so that relative importance of texture and depth code blocks can be computed for system-wide RPS optimization. Experimental results show that the proposed scheme can outperform the use of a traditional feedback channel by up to 0.82 dB on average at 8% packet loss rate, and by as much as 3 dB for particular frames

BIT-RATE ALLOCATION BETWEEN TEXTURE AND DEPTH: INFLUENCE OF DATA SEQUENCE CHARACTERISTICS

International audienceThis paper questions the existence of factors influencing the quality of the synthesized views, in the context of multi-view video plus depth coding (MVC). The issue of bit-rate allocation between texture and depth data in MVC is still open, despite the many efforts already raised for the development of optimization techniques. The originality of this study lies in the investigation of direct relationships between the best bit-rate allocation, in terms of objective quality of synthesized views, and the sequence characteristics (entropy of depth maps, depth complexity and camera baseline distance, background/foreground contrast areas). The results confirm our assumptions regarding the impact of the sequence features on the bit-rate allocation. The results and the limitations of the study are also discussed

Fast inter-mode decision in multi-view video plus depth coding

Motion and disparity estimations are employed in Multi-view Video Coding (MVC) to remove redundancies present between temporal and different viewpoint frames, respectively, in both the color and the depth multi-view videos. These constitute the major computational expensive tasks of the video encoder, as iterative search for the optimal mode and its appropriate compensation vectors is employed to reduce the Rate-Distortion Optimization (RDO) cost function. This paper proposes a solution to limit the number of modes that are tested for RDO to encode the inter-view predicted views. The decision is based on the encoded information obtained from the corresponding Macroblock in the Base view, identified accurately by using the multi-view geometry together with the depth data. Results show that this geometric technique manages to reduce about 70% of the estimation's computational time and can also be used with fast geometric estimations to reduce up to 95% of the original encoding time. These gains are obtained with little degradation on the multi-view video quality for both color and depth MVC.peer-reviewe

Depth coding using depth discontinuity prediction and in-loop boundary reconstruction filtering

This paper presents a depth coding strategy that employs K-means clustering to segment the sequence of depth images into K clusters. The resulting clusters are losslessly compressed and transmitted as supplemental enhancement information to aid the decoder in predicting macroblocks containing depth discontinuities. This method further employs an in-loop boundary reconstruction filter to reduce distortions at the edges. The proposed algorithm was integrated within both H.264/AVC and H.264/MVC video coding standards. Simulation results demonstrate that the proposed scheme outperforms the state of the art depth coding schemes, where rendered Peak Signal to Noise Ratio (PSNR) gains between 0.1 dB and 0.5 dB were observed.peer-reviewe

Multi-View Video Packet Scheduling

In multiview applications, multiple cameras acquire the same scene from different viewpoints and generally produce correlated video streams. This results in large amounts of highly redundant data. In order to save resources, it is critical to handle properly this correlation during encoding and transmission of the multiview data. In this work, we propose a correlation-aware packet scheduling algorithm for multi-camera networks, where information from all cameras are transmitted over a bottleneck channel to clients that reconstruct the multiview images. The scheduling algorithm relies on a new rate-distortion model that captures the importance of each view in the scene reconstruction. We propose a problem formulation for the optimization of the packet scheduling policies, which adapt to variations in the scene content. Then, we design a low complexity scheduling algorithm based on a trellis search that selects the subset of candidate packets to be transmitted towards effective multiview reconstruction at clients. Extensive simulation results confirm the gain of our scheduling algorithm when inter-source correlation information is used in the scheduler, compared to scheduling policies with no information about the correlation or non-adaptive scheduling policies. We finally show that increasing the optimization horizon in the packet scheduling algorithm improves the transmission performance, especially in scenarios where the level of correlation rapidly varies with time