Radha, Hayder

Shukla, Rahul

Vetterli, Martin

Infoscience - École polytechnique fédérale de Lausanne

DISPARITY DEPENDENT SEGMENTATION BASED STEREO IMAGE CODINGRahul Shukla†, Hayder Radha§ and Martin Vetterli††Audio-Visual Communications LaboratorySwiss Federal Institute of Technology Lausanne (EPFL), Switzerland§Department of ECE, Michigan State University, East Lansing, USAemail: rahul.shukla@ep.ch, radha@egr.msu.edu, martin.vetterli@ep.chABSTRACTIn this paper, we propose a novel rate-distortion (R-D) op-timized disparity based coding scheme for stereo images.This new scheme efficiently integrates the coding of thedisparity field with the residual image obtained via dis-parity estimation/compensation (DE/DC) in an R-D frame-work. The scheme first performs a quadtree decomposi-tion of the target image and computes the disparity infor-mation along with the residual image for each node in thetree. An R-D based algorithm is then used for optimumbit allocation among the different quadtree nodes. The pro-posed scheme further jointly encodes the neighboring nodeswith similar disparity information to attain higher cod-ing gains. We present simulation results for the proposedscheme and compare these results with the performance ofa fixed block size DE/DC based JPEG2000 stereo imagecoder. Our simulations show that the proposed scheme out-performs the fixed block size based disparity compensatedJPEG2000 coder by more than 0.5 dB.1. INTRODUCTIONIt is well known that the binocular or disparity informationextracted from stereo images plays a crucial role in sev-eral fields like computer vision, remote sensing/handling,tele-presence style video conferencing, tele-medicine and3-D cinema. In particular, the majority of multi-view vi-sual coding schemes are based on disparity-field estima-tion and compensation. Moreover, the recent increase instereo visual applications translates into a growing demandfor efficient methods for the transmission and storage ofstereo image/video pairs. If the stereo images are com-pressed and transmitted independently using the standardcoding schemes, then the required bandwidth would needto be doubled. However, due to the binocular dependencybetween the stereo images, they contain significant redun-dant information in the form of inter-frame redundancy [7].This work was supported by the Swiss National Science Foundationgrant 20-63664.00.Thus, by exploiting this binocular redundancy in additionto the intra-frame redundancy present in the two constituentimages, we can achieve significant data compression with-out sacrificing the overall image quality.Coding of stereoscopic images is generally based on ex-ploiting the correlation between the left and right images.This is achieved by computing a disparity field between thestereo image pair [5, 7, 8]. The disparity field represents theamount of shift one needs to perform on the pixels withinone image (target) to find the corresponding pixels in theother image (reference). A popular approach for computingthe disparity field is to partition the target image into a set ofblocks and perform a block matching algorithm to find thebest match in the reference image. For coding applications,this approach is known as disparity estimation/disparitycompensation (DE/DC) due to its resemblance to motionestimation/motion compensation (ME/MC) methods, whichare popular for video coding. The key concept of stereo im-age compression based on DE/DC is to use one of the im-ages in the stereo pair as a reference and to predict the otherimage (the target).Disparity compensation based prediction for stereo im-ages was first introduced by Lukacs [5]. In [7], Perkins pro-posed the conditional coder/decoder structure for the stereoimage coding and analyzed the R-D behavior of this struc-ture. Aydinoglu et al. presented a coding scheme whichcombines disparity compensation and transform coding ofthe residual image in the single framework using an adap-tive incomplete transform [1]. Hierarchical block matchingalgorithms have been used to generate a more homogeneousdisparity fields that lead to better coding efficiency [10, 15].In [2], a novel scheme is presented for the efficient ex-ploitation of the zerotree algorithm in stereo image codingapplications. Disparity field estimation based on low levelfeatures, such as edges, and on model/object based meth-ods have also been used [13]. In [15], feature/object basedDE/DC scheme is proposed. This scheme determines a setof objects/features in both images and seeks correspondencebetween the two sets. Jiang [3] proposed a hybrid approachbetween block and object based schemes for the efficientcoding of stereo pairs. In [18], overlapped block disparitycompensation with adaptive search windows is presented.In [6], Moellenhoff et al. analyzed the properties of dis-parity compensated residual images and proposed transformdomain coding methods for improved coding of residual im-ages. Other efforts on stereo coding have focused on op-timal rate-distortion based algorithms for block dependentquantization [17] and motion/disparity field estimation [16].Our goal is to develop a compression algorithm, whichperforms the disparity dependent segmentation of stereoimages in the R-D framework. We employ a generalizedquadtree decomposition based DE/DC. This scheme furtherperforms the joint coding of the disparity information andthe residual image obtained by the disparity compensationin the R-D optimal sense.The rest of the paper is organized as follows: In Sec-tion 2, we describe the proposed disparity dependent seg-mentation based coding scheme for stereo images. We thenpresent some simulation results in Section 3. Finally, weconclude with a discussion of further research directions inSection 4.2. DISPARITY DEPENDENT SEGMENTATIONBASED STEREO IMAGE CODING ALGORITHMThe generic structure of a stereo image coding scheme is asfollows: First encode one of the images of the stereo pair asreference, then estimate the disparity information betweenblocks in the target and encoded reference images and codethe disparity information and the residual image obtained bysubtracting the disparity compensated target image from theoriginal image. The disparity compensated/predicted targetimage is obtained by using the encoded reference image andthe disparity map. Thus, our scheme employs the closedloop methodology for obtaining the predicted target image.The main novelty of our stereo image coding scheme isthat it performs the disparity dependent quadtree segmen-tation in the R-D framework.1 This disparity dependentsegmentation based stereo image coding scheme can be de-scribed as follows:Step 1: Disparity estimation and compensation1. Segment the target image using the quadtree decomposi-tion up to a tree depth J .2. Compute the disparity information for each node by us-ing an intensity based block matching approach.3. Predict the node using the associated disparity informa-tion and encoded reference image.4. Compute the residual image block by subtracting the pre-dicted target image block from the original image block.5. Generate the R-D curve for the residual block associ-ated with each node using a standard coding scheme like1Mean squared error (MSE) is used as a distortion measure.JPEG2000 [14].Step 2: The Lagrangian cost (L(λ) = D + λR) basedpruning6. For the given operating slope λ, R-D optimal pruning cri-terion is as follows: Prune the children if the sum of the La-grangian costs of the children is greater than or equal to theLagrangian cost of the parent. This parent-children pruningcriterion is used recursively to do fast pruning from the fulltree depth towards the root to find the optimal subtree for agiven λ [9]. This scheme provides a pruned tree. However,this independent pruning scheme fails to join neighboringblocks with similar disparity information, if they have dif-ferent parents. Thus, this coding scheme fails to exploit thecomplete dependency among neighbors. For correcting thissuboptimality, we introduce the following neighbor joiningstep proposed in [11, 12].Step 3: Neighbor joining7. Given the pruned tree obtained from Step 2, the neigh-bor joint coding is performed on the leaves of the prunedtree in the R-D optimal manner. That means, the two neigh-bors will be joined if the sum of the Lagrangian costs ofthe neighbors is greater than or equal to the Lagrangiancost of the joint block. This scheme essentially ensures thatthe neighbors with similar disparity information are jointlycoded for further improving the R-D performance. Thus,this leads to the disparity dependent segmentation of the tar-get image.8. By summing up the allocated rates to all the joint blocksalong with the costs of the segmentation map and disparitymap will provide the overall bit-rate R∗(λ). Similarly, bysumming up the associated distortions of all the joint blockswill give the net distortion D∗(λ). Since we have employedthe R-D optimization, D∗(λ) represents the minimum dis-tortion for the bit-rate constraint R∗(λ).Step 4: Search for the desired R-D operating slope9. The value for λ is determined iteratively until the bit-rateconstraint R0 is met as closely as possible. We employ thefast bisection search algorithm given in [9].Clearly, the above described algorithm provides the R-D optimal representation of the target image for a given bitbudget and encoded reference image. Due to neighbor join-ing (Step 3), it is obvious that the proposed scheme resultsinto the non regular partitioning of the target image (for ex-ample, see Figure 3). It is important to note that the highquality encoded reference image tends to allow for more ef-ficient encoding of the residual image due to the good qual-ity of the disparity predicted target image.Similar to the encoder, the structure of the decoder is asfollows: 1) Reconstruct the reference and residual images.2) Generate the predicted target image using the disparityinformation and reconstructed reference image. 3) By sum-ming the predicted target image and residual image, we ob-tain the desired target image.3. SIMULATION RESULTSSimulations are performed on the Arch stereo image pairshown in Figure 1.2 This stereo pair has the disparity pre-dominantly along the horizontal direction, which is usuallythe case in stereo image coding scenario due to the parallelaxis geometry. Therefore, for DE/DC, we utilize the searchwindow of size 64 in the horizontal direction whereas searchin the vertical direction is confined only to ±2 pixels.Figure 2 shows the disparity field obtained by thequadtree based variable block size decomposition of the tar-get image. As expected, the disparity map is smooth ev-erywhere except along the object boundaries and in the oc-cluded regions.A reconstructed target image along with the disparitydependent segmentation map is presented in Figure 3. It isevident from Figure 3 that the algorithm correctly performsfiner segmentation along the edges of objects to capture thedepth discontinuity precisely, whereas smooth regions withuniform disparity are represented by larger blocks. Thisresults into smaller disparity information overhead bits forsmooth regions. This figure also demonstrates that the pro-posed coding scheme is capable of achieving high com-pression ratios without sacrificing the reconstructed imagequality which we are measuring in terms of PSNR. Thesesimulations also indicate that our R-D optimized algorithmautomatically takes care of occluded regions either by per-forming finer segmentation in those regions or by allocatingmore bits to corresponding residual blocks.Figure 4 compares the R-D performance of our dispar-ity dependent segmentation based coding scheme with theindependent JPEG2000 coder and fixed block size (16×16)based disparity compensated JPEG2000 stereo image coder.This R-D performance comparison clearly shows the supe-riority of the proposed algorithm over the other two algo-rithms.(a) Reference image. (b) Target image.Fig. 1. Original Arch stereo pair.2These stereo images are obtained fromhttp://vasc.ri.cmu.edu//idb/html/stereo/arch/index.html.Fig. 2. Quadtree based disparity map.Fig. 3. Reconstructed target image along with the segmenta-tion map obtained by the disparity dependent segmentationbased coding scheme (PSNR=41.4 dB, Bit-rate=0.116 bpp).4. DISCUSSION AND FURTHER WORKWe have presented an R-D optimized disparity dependentsegmentation based stereo image coding scheme. This newscheme performs the joint coding of disparity informationand the residual image to achieve the improved R-D perfor-mance. Experimental results shown in Figure 4 also confirmthat the proposed scheme outperforms the disparity com-pensated JPEG2000 stereo coder by about 0.5 dB. Sinceour scheme employs a quadtree based decomposition, thisscheme is computationally efficient as well. Our ongoingresearch effort is to extend this stereo image coding schemeto multi-view image and stereo video coding scenarios. Wewould also like to investigate feature/object based DE/DCscheme in the framework of the proposed algorithm.0.08 0.1 0.12 0.14 0.16 0.18 0.2 0.22 0.243939.54040.54141.54242.54343.5R−D performance for the target image of the Stereo PairBit−Rate (bpp) PSNR  (dB)Independent JPEG2000 codingJPEG2000 with DE/DCDisparity Dependent Segmentation Based CodingFig. 4. R-D performance comparison of different codingschemes for the target image of the Arch stereo pair shownin Figure 1.Another interesting problem setup is the distributed cod-ing scenario for stereo images. For instance, consider anelectronic surveillance system where the two cameras areviewing the same scene. In this setup, the communicationbetween the two encoders is not allowed due to some con-straints such as the power constraint. We are planning todevelop a distributed stereo image coding algorithm for thecases where the communication cost between the encodersis much higher than the computation cost.5. REFERENCES[1] H. Aydinoglu and H. Hayes, “Stereo image coding: Aprojection approach,” IEEE Trans. Image Proc., vol. 7,pp. 506-516, Apr. 1998.[2] N. V. Boulgouris and M. G. Strintzis, “A family ofwavelet-based stereo image coders,” IEEE Trans. Cir-cuits Syst. Video Tech., vol. 12, no. 10, pp. 898-903,Oct. 2002.[3] J. Jiang and E. A. Edirisinghe, “A hybrid scheme forlow bit-rate coding of stereo images,” IEEE Trans. Im-age Proc., vol. 11, no. 2, pp. 123-134, Feb. 2002.[4] F. Labonte, C. T. Le Dinh, J. Faubert, and P. Cohen,“Spatiotemporal spectral coding of stereo image se-quences,” IEEE Trans. Circuits, Syst., Video Tech., vol.9, pp. 144-155, Feb. 1999.[5] M. E. Lukacs, “Predictive coding of multi-viewpointimage sets,” in Proc. ICASSP, Oct. 1986, pp. 521-524.[6] M. S. Moellenhoff and M.W. Maier, “Transform codingof stereo image residuals,” IEEE Trans. Image Proc.,vol. 7, pp. 804-812, June 1998.[7] M. G. Perkins, “Data compression of stereo pairs,”IEEE Trans. Comm., vol. 40, pp. 684-696, Apr. 1992.[8] A. Puri, R. V. Kollarits, and B. G. Haskell, “Basicsof stereoscopic video, new compression results withMPEG-2 and a proposal for MPEG-4,” Jour. SignalProc. Image Comm., vol. 10, pp. 201-234, 1997.[9] K. Ramchandran and M. Vetterli, “Best wavelet packetbases in a rate distortion sense,” IEEE Trans. ImageProc., vol. 2, no. 2, pp. 160-175, April 1993.[10] S. Sethuramam, M. W. Siegel, and A. G. Jordan, “Amulti-resolution framework for stereoscopic image se-quence compression,” in Proc. ICIP-95, vol. 2, pp. 361-365.[11] R. Shukla, P.L. Dragotti, M.N. Do and M. Vetterli,“Improved quadtree algorithm based on joint coding forpiecewise smooth image compression,” in Proc. IEEEInt. Conf. on Multimedia, Switzerland, Aug. 2002.[12] R. Shukla, P.L. Dragotti, M.N. Do and M. Vetterli,“Rate-distortion optimized tree structured compressionalgorithms for piecewise smooth images ,” submitted toIEEE Trans. Image Proc., Dec. 2002.[13] M. G. Strintzis and S. Malassiotis, “Object-based cod-ing of stereoscopic and 3D image sequences: A review,”IEEE Signal Proc. Mag., vol. 16, pp. 14-29, May 1999.[14] D. Taubman and M. Marcellin, JPEG2000: ImageCompression Fundamentals, Standards and Practice,Kluwer Academic Publishers, Dordrecht, 2001.[15] D. Tzovaras, N. Grammalidis, and M. G. Strintzis,“Object-based coding of stereo image sequences usingjoint 3D motion/disparity compensation,” IEEE Trans.Circuits Syst. Video Tech., vol. 7, pp. 312-327, Apr.1997.[16] D. Tzovaras and M. G. Strintzis, “Motion and dispar-ity field estimation using rate-distortion optimization,”IEEE Trans. Circuits Syst. Video Tech., vol. 8, pp. 171-180, Apr. 1998.[17] W. Woo and A. Ortega, “Optimal blockwise dependentquantization for stereo image coding,” IEEE Trans. Cir-cuits Syst. Video Tech., vol. 9, pp. 861-867, Sept. 1999.[18] W. Woo and A. Ortega, “Overlapped block disparitycompensation with adaptive windows for stereo imagecoding,” IEEE Trans. Circuits Syst. Video Tech., vol. 10,pp. 194-200, Mar. 2000.

Disparity Dependent Segmentation Based Stereo Image Coding

Abstract

Similar works

Full text

Available Versions

Infoscience - École polytechnique fédérale de Lausanne