Forchhammer, Søren

Zamarin, Marco

In this short paper a brief overview of the topic of coding and compression of depth images for multi-view image and video coding is provided. Depth images represent a convenient way to describe distances in the 3D scene, useful for 3D video processing purposes. Standard approaches for the compression of depth images are described and compared against some recent specialized algorithms able to achieve higher compression performances. Future research directions close the paper

Online Research Database In Technology

                       General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.  • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal   If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.     Downloaded from orbit.dtu.dk on: Dec 20, 2017Coding of Depth Images for 3DTVZamarin, Marco; Forchhammer, SørenPublication date:2012Link back to DTU OrbitCitation (APA):Zamarin, M., & Forchhammer, S. (2012). Coding of Depth Images for 3DTV. Paper presented at 3rd AnnualWorkshop on Photonic Technologies and Applications : 2nd Danish-Californian Photonics Workshop, Berkeley,California, United States.Coding of Depth Images for 3DTVMarco Zamarin, Søren ForchhammerCoding & Visual Communications Group, Dept. of Photonics EngineeringTechnical University of Denmark, Ørsteds Plads B.343, 2800 Kgs. Lyngby, Denmark{mzam,sofo}@fotonik.dtu.dkAbstract: In this short paper a brief overview of the topic of coding and compression ofdepth images for multi-view image and video coding is provided. Depth images represent aconvenient way to describe distances in the 3D scene, useful for 3D video processing purposes.Standard approaches for the compression of depth images are described and compared againstsome recent specialized algorithms able to achieve higher compression performances. Futureresearch directions close the paper.1. IntroductionA 3D scene can be represented for coding purposes through different data formats. As depicted in Fig. 1(a), theseformats can be grouped into two main categories: the image-based one (e.g. light-fields and multi-view videos) inwhich no 3D geometry is used, and the geometry-based one (e.g. 3D mesh models) in which explicit geometricmodels are exploited [1]. A very popular and flexible format combining the advantages of both the categories is ahybrid one, namely “Multi-View Video Plus Depth”. This representation includes two different types of data: colorand geometry. Color information is provided by a number of synchronized video streams (typically between 2 and 16)of the same scene from different viewpoints while geometric information is described by a number of depth streams.A depth image (or depth map) is a grey-scale image in which pixel values describe distances between the cameraplane and the 3D points in the 3D scene.Figure 2 shows an example of a color image with the associated depth map. Depth maps are typically used toproject views to different viewpoints (“3D warping” [2]) in order to obtain predicted views to be used for codingpurposes as done in [3], or synthesized views e.g. at the decoder side. The ISO/IEC Moving Picture Experts Groups(MPEG) is currently developing a standard for 3D video services and devices that enables advanced stereoscopicdisplay processing and improves support for auto-stereoscopic N-view displays [4] (see Fig. 1(b)). Since they allowfor high-quality view synthesis, depth images appear as a very useful side information in this scenario and have beenexploited in most of the responses to the MPEG Call for Proposals on 3D Video Coding Technology [5]. Therefore,Image-based Geometry-basedRay-spaceLight-field3D meshmodelLumigraph LayeredDepthvideoView-dependentvideo texture mapping3D point samplesVideo fragmentsVideo plus depthMulti-view videoplus depthData FormatData FormatConstrained Rate(based on distribution)LimitedCamera Inputs• Wide viewing angle• Large number of output viewsLeftRightAuto-stereoscopicN-view displaysStereoscopic displays• Variable stereo baseline• Adjust depth perception(a) (b)Fig. 1. (a): 3D scene representations for 3DV and FVV (reprinted from [1], copyright 2008Springer); (b) “Target of 3D Video format illustrating limited camera inputs and constrained ratetransmission according to a distribution environment. The 3DV data format aims to be capable ofrendering a large number of output views for auto-stereoscopic N-view displays and support ad-vanced stereoscopic processing”, reprinted from [4].Fig. 2. A color image (left) and the relative depth image (right) from the “Breakdancers” data set [6].Fig. 3. Compressed color data (left) and relative segmentation (right) used to predict shapes in thedepth image.the compression of depth images plays a crucial role in the development of a 3D video communication system andneeds to be taken into account carefully in order to allow for effective performances.2. Depth coding: standard approaches versus specialized approachesSince depth maps are gray-scale images, they can be compressed using classical coding algorithms adopted forvideos and still images - such as H.264/AVC and JPEG2000 - as done in [7] where a standard H.264/AVC Intracoder is used. However, standard approaches do not reach optimal performances since depth images have quitedifferent characteristics than natural images: they are usually made of smooth regions divided by sharp edges. Amajor requirement for depth coding is the preservation of important edge information in order to allow for accurateview synthesis rendering thus avoiding sample displacement artifacts. One approach that satisfies this requirementis the Platelet-based one proposed by Morvat et al. [8] in which smooth regions are modeled using piecewise-linearfunctions and sharp edges approximated by straight lines. As shown in [9], this approach permits obtaining sharperand more accurate synthesized views compared to the cases in which H.264/AVC and MVC are used for depthcompression.Depth images are usually transmitted together with their relative color views. Thanks to the correlation betweencolor views and corresponding depth images, color data can be effectively exploited to reduce the depth bit-rate.In [10] segmented color data are used to predict shapes in the depth image (see Fig. 3). Since within each segmentdepth data are expected to be very smooth, each depth segment is approximated with a parameterized plane. In case theapproximation is sufficiently accurate for the target bit-rate, the surface coefficients are compressed and transmitted.Otherwise, the region is coded using a standard H.264/AVC Intra coder. Note that since the color image being seg-mented is the compressed one (i.e. the same one available at the decoder), no segmentation data needs to be encodedand transmitted thus permitting a considerable saving in terms of bit-rate.Experimental results show that the proposed scheme outperforms the H.264/AVC Intra codec on depth data interms of PSNR - with average gains of 1 to 2 dB - and edge-preservation capabilities. Moreover, this approach canoutperform JPEG2000 and the Platelet-based method of [8] in terms of depth PSNR. Figure 4 compares the impactof coding artifacts on a depth image from the “ballet” dataset in case of a standard H.264/AVC Intra codec and theproposed scheme. While the H.264/AVC Intra codec introduces significant degradations of the main edges due to itsFig. 4. Details of reconstructed depth images from the “ballet” data set [6]: standard H.264/AVCIntra (left) and plane fitting exploiting color segmentation (right), both at 0.025 bpp.low-pass behavior, the proposed approach is able to preserve sharp edges thus permitting more accurate view synthesisoperations.3. Lossless coding of depth imagesAn interesting research direction considers lossless compression of depth images. Lossless coding permits avoidingrendering artifacts in synthesized views due to depth compression artifacts. In case of natural images, lossless codingtypically provides compression factors of about 2 to 3, which make lossless compression not suitable for many practicalapplications. As for depth images, preliminary results show that much higher compression factors can be achievedwhen specialized algorithms are applied: 15 to 30 in case of intra coding and 20 to 50 in case of stereo depth images.Some early works like [11] show that efficient lossless compression can be obtained also in case of depth video. Suchperformances encourage further investigation and can motivate for lossless (and near-lossless) transmission of depthdata, at least in specific scenarios.4. Conclusion and future workIn this short paper the topic of compression of depth images and its relevance in view synthesis operations is dis-cussed. Some standard and specialized coding approaches for depth data have been briefly presented, highlightingthe advantages of the latter over the former. Future developments include depth coding using graph-based transforms,near-lossless coding of depth data for higher compression efficiency, multi-view depth image coding exploiting inter-view depth redundancy, and efficient joint color and depth coding.References1. A. Smolic, P. Merkle, K. Muller, C. Fehn, P. Kauff, and T. Wiegand, Three-Dimensional Television, ch. Compression ofMulti-view Video and Associated Data, pp. 313350. Springer Heidelberg, 2008.2. R. I. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision. Cambridge University Press, second ed., 2004.3. M. Zamarin, S. Milani, P. Zanuttigh, and G. Cortelazzo, “A novel multi-view image coding scheme based on view-warpingand 3D-DCT,” Elsevier Journal of Visual Communication and Image Representation, vol. 21, no. 5-6, pp. 462473, 2010.4. Video and Requirements Group, “Vision on 3D Video,” ISO/IEC JTC1/SC29/WG11 N10357, Lausanne, CH, Feb. 2009.5. 3D Video AHG, “AHG Report on 3D Video Coding,” ISO/IEC JTC1/SC29/WG11 m21469, Geneva, CH, Nov. 2011.6. L. Zitnick, S. Kang, M. Uyttendaele, S. Winder, and R. Szeliski, “High-quality video view interpolation using a layeredrepresentation,” ACM Trans. Graph., vol. 23, no. 3, pp. 600608, 2004.7. M. Zamarin, P. Zanuttigh, S. Milani, G. M. Cortelazzo, and S. Forchhammer, “A joint multi-view plus depth image codingscheme based on 3D-warping,” in Proc. of 1st ACM Int. Workshop on 3D Video Processing (3DVP 2010), pp. 712, 2010.8. Y. Morvan, D. Farin, and P. de With, “Depth-image compression based on an R-D optimized quadtree decomposition forthe transmission of multiview images,” in Proc. of IEEE ICIP 2007, 2007.9. K. Mu¨ller, P. Merkle, and T. Wiegand, “3-D video representation using depth maps,” Proc. of the IEEE, vol. 99, pp. 643656,Apr. 2011.10. S. Milani, P. Zanuttigh, M. Zamarin, and S. Forchhammer, “Efficient depth map compression exploiting segmented colordata,” in 2011 IEEE Intl Conf. Multimedia and Expo, pp. 16, July 2011.11. K. Kim, G. Park, and D. Suh, “Bit-plane-based lossless depthmap coding,” Optical Engineering, vol. 49, June 2010.

Coding of Depth Images for 3DTV

http://orbit.dtu.dk/ws/files/53065348/Berkeley2012.pdf

Coding of Depth Images for 3DTV

Abstract

Similar works

Full text

Available Versions

Online Research Database In Technology