Frossard, Pascal

Luo, Chong

Schenkel, Markus Beat

Wu, Feng

Infoscience - École polytechnique fédérale de Lausanne

JOINT DECODING OF STEREO JPEG IMAGE PAIRSMarkus B. Schenkel 1,2 , Chong Luo1 , Pascal Frossard2 and Feng Wu11Microsoft Research Asia, Beijing, 100190, China2 Signal Processing Laboratory (LTS4), EPFL, Lausanne, 1015, SwitzerlandABSTRACTThis paper addresses the problem of joint decoding of stereoJPEG image pairs. Such images typically contain a high de-gree of redundancy. Predictive coding could efﬁciently cap-ture this redundancy, but cameras would have to implementproprietary encoding solutions in this case as no such stan-dard technology is available. We propose to rather use thepopular JPEG compression tools in the cameras, and focus onthe joint decoding problem for quality enhancement. We for-mulate this as a constrained optimization problem and showhow regularization leads to more consistent results. It is sim-ilar to a distributed source coding framework, where the ex-ploitation of the correlation at the decoder permits to save onthe overall bandwidth. Experiments on natural stereo imagesshow an improvement in both visual quality and PSNR whencompared to separate decoding.1. INTRODUCTIONStereo perception and 3D rendering are usually achievedby simultaneously presenting two images of the same scenetaken from two slightly displaced positions to both eyes. Wewill denote these two images with left (L) and right (R).Because they both represent the same reality they are highlyredundant and accordingly we can expect signiﬁcant com-pression ratios to be achievable. The depth informationdirectly translates into the disparity that relates the two im-ages and in the case of perfectly rectiﬁed views the possibletransformations reduce to horizontal shifts only. Therefore apredictive coding scheme could be employed by intra codingthe ﬁrst image, followed by the disparity ﬁeld for a predictorand the residue.Such a traditional coding has previously been proposedby Perkins [1] for example. But the proprietary transmissionformat required by such an approach is unlikely to replaceestablished general purpose image formats and it is worthinvestigating how much we could still improve without de-parting from JPEG encoding. Indeed, today’s digital cam-eras are widely equipped with a JPEG encoder and stereo-scopic image pairs are encoded as separate JPEG images bymost applications, be it JPEG Stereo (JPS), Multi-Picture Ob-ject (MPO) [2] or others; thus they directly double the band-width requirements. Cameras also have a limited amount ofprocessing power when compared to the corresponding de-coders; this further motivates a distributed coding scheme.A joint decoding strategy is proposed in this paper, in or-der to enhance the quality of the reconstructed image pairsthat have been compressed independently with JPEG. Over-all, the joint decoder allows for a lower overall bitrate fora given quality for both views. We cast the reconstructionproblem as a regularized convex optimization problem thatis constrained by consistent reconstruction conditions. Weshow that regularization permits to increase the accuracy ofthe disparity estimation, hence to obtain better reconstructionquality.In particular, we consider an asymmetric coding schemewhere one of the images is encoded at high quality, the otherone at a reduced quality. Such an asymmetric scheme is ofparticular interest to applications where the second view isnot always required, notably if no stereo display is available.In addition, controlled asymmetry does not really penalize 3Dperception. Studies by Seuntiens et al. [3] and others indicatethat the human brain can tolerate a fair amount of asymmet-ric image quality for stereo viewing such that the perceivedquality lies between that of the two views.This work is related to the distributed coding framework,where joint decoding is used to reconstruct correlated signalsthat have been independently encoded. However, we do notwork here on the coding strategy, but rather rely on joint de-coding approaches of signals that have been encoded withclassical solutions. The joint reconstruction of compressedimages has been considered also in the compressed sensingcommunity, where different approaches have been proposedto represent images or parts of them as a sparse linear com-bination of other images assembled in a dictionary. If sucha representation exists, it can under some conditions also berecovered in a stable way from linear projections onto a set ofrandom vectors that reduces its dimensionality. This has ledto applications in video coding where the dictionary is com-posed of blocks from a previous frame [4], face recognitionwhere the candidate faces build the dictionary [5], or multi-view representations [6]. In our solution a reconstructed blockwill be based on local dictionaries of candidate blocks. Com-pressed sensing in combination with total variation minimiza-tion has also been used for the compression of depth maps [7].Alternatively, super-resolution reconstruction from image se-2633978-1-4244-7993-1/10/$26.00 ©2010 IEEE ICIP 2010Proceedings of 2010 IEEE 17th International Conference on Image Processing September 26-29, 2010, Hong Kongquences has a similar objective of quality enhancement withmultiple compressed images. It tries to either estimate a densedisplacement ﬁeld or to fuse different frames of a video to-gether to enhance its quality. This is often formulated as aninverse optimization problem with a smoothness constraint onthe displacement ﬁeld (e.g. [8] and [9]), but unlike here morethan two images are usually involved where reconstruction athigher resolution is the main target.2. PROPOSED SCHEMEJPEG is a block-based still image compression scheme thatcompacts the image energy in a small number of coefﬁcientsand introduces losses mostly at high frequencies where theyare visually more acceptable [10]. This is done by apply-ing the two-dimensional discrete cosine transform (DCT) de-noted by D, followed by scalar quantization with up to tentimes bigger step sizes at the highest frequencies than for thelower ones. The quantization step sizes are given by a tableq . If b is an image block and y = Db its representationin the transform domain, then quantization can be written asyi = qi [ yi/qi ] where [ · ] denotes rounding to the nearest in-teger. Because of the lossy nature of JPEG encoding we havea certain freedom to ﬁll in the coarsely quantized coefﬁcientsfrom the other image at higher quality.In the following we will always use the left image as theintra coded reference and the right one as the compressed im-age that we want to enhance. We limit our studies to lumi-nance images only, but this extends to the other color compo-nents as well, because they are coded in the same fashion.The compressed version of the right image deﬁnes a set ofpossible solutions for approximating the original image ver-sion, which are all consistent with respect to the compressedone, meaning that they would yield exactly the same JPEGbitstream after a recompression using the same quantizationmatrix. This is the range we will operate in to reconstructthe image. Although a midpoint or a centroid dequantizationfollowed by an inverse DCT will likely minimize the recon-struction error if no further information is present, they are notthe only choices within the aforementioned admissible region.Thus we will use the compressed image only to formulate aconstraint on the output image. Our scheme operates on theblocks of 8×8 pixels deﬁned by JPEG. We can formulate thiselementwise constraint in the transform domain as∣∣D(bˆ(i) − b(i))∣∣  12q , (1)where b(i) is the ith block in the pixel domain after JPEGcompression, bˆ(i) is the estimate of that block after enhance-ment and the element-wise inequality |xj | ≤ yj ∀j is writtenas |x |  y .Now we build a dictionary Ψ(i) composed of possiblecandidate blocks ψ(i)j from the reference image. Figure 1illustrates the origin of the dictionary for block i. They areReference (left)Output (right)bˆ(i)DictionaryΨ(i)JPEGCamerasbqq ′bFig. 1. Schematic view of the proposed scheme. Illustrat-ing the separate encoding (with a ﬁne q ′ and a coarse q ), thecomposition of the dictionary for a given block i from the ref-erence view and the reconstruction of that block.gathered from a range from 0 disparity up to the maximumdisparity Dh in horizontal direction around the location inquestion. We further extend this range to 2Dv + 1 shifts inthe vertical direction to accommodate slight misalignmentsof the two cameras. Each of the D = Dh(2Dv + 1) dic-tionary elements ψ(i)j now has an associated disparity vectord j ∈ {0, . . . , Dh}×{−Dv, . . . , Dv}. They will be used lateron to build the disparity ﬁelds vx and vy which associate toeach block a horizontal and vertical shift respectively.We now want to represent a block b(i) as a linear com-bination of dictionary elements with the coefﬁcient vectors(i) ∈ RD asb(i) ≈∑jψ(i)j s(i)j = Ψ(i)s(i).In general it is not possible to ﬁnd such a decomposition thatalso satisﬁes (1). Thus we introduce the slack variables bˆ(i)that are constrained as above and approximated by a linearcombination over {ψ(i)j }.However, this inverse problem is still ill posed and wecan regularize it in two ways. First, only a small number ofthe dictionary elements will contribute – ideally only a sin-gle one – and we can thus require s to be sparse. Althougha high sparsity is best described with a low 0 pseudo-norm‖s‖0 = |{sj |sj = 0}| this is not a convex function and wouldlead to a problem of combinatorial nature. Consequently weapproximate it by the convex 1 norm ‖s‖1 =∑j |sj | .Second, we can assume the disparity ﬁeld v to be piece-wise smooth because disparity discontinuities will occuronly at object boundaries. Furthermore, not every blockcontributes an equal amount of depth information. Hencewe can improve the reconstruction by enforcing a low to-tal variation ‖ · ‖TV = 1N∑j,k |(∇ · )j,k| of the dispar-ity ﬁeld v (i) = [v(i)x v(i)y ]T . We calculate v (i) for eachblock as the weighted sum of the contributing disparitiesv (i) =∑j d j s(i)j . This step makes our problem a global oneinvolving all blocks at once. In the following, all N blocks ofan image are concatenated such that bT = [b(1)T · · · b(N)T ]and the transform D becomes a block diagonal matrix.2634JPEGQ: 1029.0 dBEnhanced28.3 dBQ: 2031.1 dB31.5 dBQ: 5033.8 dB35.6 dBOriginalleftrightDetailJPEGEnhanced (TV)DetailOriginalEnhanced (no TV)Fig. 2. This experiment on a 288× 288 pixel image of Sekhmet taken with a commercial stereo camera illustrates our method.Results for three different JPEG quality settings with and without enhancement are followed by the original left and right viewand enlarged details from the second image pair. We can improve the visual quality even if PSNR results drop and ﬁnd lessblocking artifacts if the total variation of the disparity ﬁeld is minimized. Compare these results with Fig. 3(a).Putting it all together we obtain the objective functionL(bˆ, s) =∥∥∥bˆ −Ψs∥∥∥22+λv(‖vx‖TV + ‖vy‖TV)+λs ‖s‖1bˆ = argminbˆ,sL(bˆ, s) s.t.∣∣∣D(bˆ − b)∣∣∣  12q . (2)The parameter λv is the Lagrange multiplier weighting theimportance of a smooth disparity ﬁeld. The second parameterλs acts on the sparsity of the signal and trades off ﬁdelityversus sparsity. All three terms in (2) scale with the numberof blocks N , hence we can set the relative values of λs andλv independently of the image size.The choice of λs is crucial, because on one hand a bigvalue makes s approach 0 , while on the other hand a smallvalue can lead to a non-sparse s and a blurred result. Wechoose it such that the terms of the objective function are ofsimilar magnitude and verify empirically that λs = 5× 10−2leads to good results for the tested images and a dictionary ofsize D = 13 × 9 = 117. The choice of λv is less criticaland we set it to λv = 1 × 10−2. All results presented in thefollowing were obtained with this same set of parameters.In order to solve (2) we use the cvx package for Mat-lab [11, 12] which applies an interior point method to it.Larger images are further partitioned into independentlyreconstructed regions of smaller size to keep the memoryrequirements manageable. Solving this problem is quite in-volved, but it scales linearly with the image size. An accurateknowledge of the maximum disparity reduces the number ofvariables and thus the runtime of the optimization.3. EXPERIMENTAL RESULTSWe analyze in this section the performance of the joint de-coding algorithm. As we can see from Fig. 2 we are able toenhance an image with correctly positioned details even if theright image is highly compressed. The ubiquitous blockingartifacts introduced by JPEG disappear and texture is added.Despite the small improvement in peak signal-to-noise ratio(PSNR) at low bitrates, the visual quality of the images im-proves clearly and in a consistent way. The enlarged areasshow that even small details can be recovered that would sim-ply be blurred out by JPEG. For medium bitrates (JPEG qual-ity around 50) the PSNR improves by about 1 dB on averageand more as the rate-distortion comparison in Fig. 3 high-lights. In the optimal region we can accommodate a bitratesaving of 20% and above for the right view at a similar de-coding quality.Regions that are occluded in the reference view cannotpossibly be reconstructed by this method. In the middle range(Quality 50 – 75) only few blocks have a PSNR decrease andthey lie usually in such regions around disparity discontinu-ities as well as the very right border of the image. Never-theless, ghosting artifacts appear seldom. Even though the re-sults shown here were obtained with well aligned image pairs,additional experiments after multiple pixel shifts and a 2◦ ro-tation still exhibit good performance.Furthermore, we study the inﬂuence of the individualparts of the optimization. First, we can set λv = 0 to removethe regularization of the disparity ﬁeld with a total variationconstraint. We ﬁnd that at low bitrates the compressed image26350.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.228303234363840 10 15 20 25 50 75Rate [bpp]Distortion: PSNR [dB]JPEG decodingEnhanced decoding with TV regularizationEnhanced decoding with l1 minimization onlyJPEG quality settingleft right(a) The Sekhmet image from Fig. 2 (288× 288 pixels).0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2262830323436 10 15 20 25 50 75Rate [bpp]Distortion: PSNR [dB]JPEG decodingEnhanced decoding with TV regularizationEnhanced decoding with l1 minimization onlyJPEG quality settingleft right(b) The Tsukuba stereo test set (192× 192 pixels).Fig. 3. Rate-distortion comparison of JPEG ( ) and joint decoding ( ) for the right view of two image pairs. Alsoshown is the curve for unconstrained 1 minimization ( ) given by Eq. (3).might not contain enough details to reliably ﬁnd a corre-sponding block in the reference view and it is in this regionwhere the additional regularization of the disparity ﬁeld leadsto a further improvement. Although the increase in PSNR isonly little, less artifacts are visible. The last part of Fig. 2shows such a case. Second, we can also compare our methodwith the unconstrained 1 minimizationbˆ = Ψsˆ, sˆ = argmins∥∥b −Ψs∥∥22+ λs ‖s‖1 . (3)This gives an improvement at low rates as seen from Fig. 3,but also tends to saturate at some PSNR level. We can expectthis since the output is a sparse linear combination over thereference image which does not contain all the details. Withthe consistency constraint (1) we can overcome this gap.Finally, we should note that in these experiments the ref-erence image was always given at full quality. If the referenceitself is compressed the improvements will naturally decrease;however, reference images at a quality setting of 80 and 90could still be used successfully, as it has been conﬁrmed byadditional experiments.4. CONCLUSIONSWe have presented a joint decoding solution for stereo imagepairs. This method permits to reduce the bitrate of one viewof a stereo image pair that is based on two separately coded,standard compliant JPEG images, but produces visually muchbetter results than separate decoding. The only assumptionswe make about the image pair is a relatively good alignmentand a known maximum disparity. However both are only re-quired to reduce the runtime of the algorithm.We showed that it is possible to bring the quality of a sec-ond view closer to that of the reference image and simultane-ously mitigate the effect of JPEG compression artifacts. Thepresented scheme provides good results for a viewing appli-cation because the two images will be of comparable quality.On the other hand it might not be a good preprocessing stepfor vision applications although the coarse depth maps ob-tained as a side product indicate that a fair amount of disparityestimation can still be done after compression.5. REFERENCES[1] M.G. Perkins, “Data Compression of Stereopairs,” IEEE Transactionson Communications, vol. 40, no. 4, pp. 684–696, 1992.[2] “Multi-Picture Format,” Camera & Imaging Products AssociationStandardization Committee, 2009.[3] P. Seuntiens, L. Meesters, and W. Ijsselsteijn, “Perceived quality ofcompressed stereoscopic images: effects of symmetric and asymmet-ric JPEG coding and camera separation,” ACM Transactions on Ap-plied Perception, vol. V, 2006.[4] J. Prades-Nebot, Y. Ma, and T. Huang, “Distributed video coding us-ing compressive sampling,” Proceedings of the Picture Coding Sym-posium, 2009.[5] J. Wright, A.Y. Yang, A. Ganesh, S.S. Sastry, and Y. Ma, “Robust facerecognition via sparse representation.,” IEEE transactions on patternanalysis and machine intelligence, vol. 31, no. 2, pp. 210–27, 2009.[6] V. Thirumalai and P. Frossard, “Motion estimation from compressedlinear measurements,” Proceedings of ICASSP, 2010.[7] M. Sarkis and K. Diepold, “Depth map compression via compressedsensing,” in Proceedings ICIP, November 2009.[8] B.K.P. Horn and B.G. Schunck, “Determining Optical Flow,” Artiﬁ-cial Intelligence, vol. 17, 1980.[9] A. Wedel, T. Pock, C. Zach, H. Bischof, and D. Cremers, “An im-proved algorithm for TV-L1 optical ﬂow,” Statistical and Geomet-rical Approaches to Visual Motion Analysis: International DagstuhlSeminar, pp. 23–45, 2009.[10] W. Pennebaker and J. Mitchell, JPEG still image data compressionstandard, New York, 1993.[11] S. Boyd and M. Grant, “CVX: Matlab software for disciplined convexprogramming,” http://cvxr.com/cvx, 2009.[12] M. Grant and S. Boyd, Graph implementations for nonsmooth convexprograms, Recent Advances in Learning and Control (a tribute to M.Vidyasagar), pp. 95–110, Springer, 2008.2636

Joint decoding of stereo JPEG images pairs

http://infoscience.epfl.ch/record/161384/files/icip2010g.pdf

Joint decoding of stereo JPEG images pairs

Abstract

Similar works

Full text

Available Versions

Infoscience - École polytechnique fédérale de Lausanne