Transmitting texture and depth images of captured camera view(s) of a 3D scene enables a receiver to synthesize novel virtual viewpoint images via Depth-Image-Based Rendering (DIBR). However, a DIBR-synthesized image often contains disocclusion holes, which are spatial regions in the virtual view image that were occluded by foreground objects in the captured camera view(s). In this paper, we propose to complete these disocclusion holes by exploiting the self-similarity characteristic of natural images via nonlocal template-matching (TM). Specifically, we first define self-similarity as nonlocal recurrences of pixel patches within the same image across different scales--one characterization of self-similarity in a given image is the scale range in which these patch recurrences take place. Then, at encoder we segment an image into multiple depth layers using available per-pixel depth values, and characterize self-similarity in each layer with a scale range; scale ranges for all layers are transmitted as side information to the decoder. At decoder, disocclusion holes are completed via TM on a per-layer basis by searching for similar patches within the designated scale range. Experimental results show that our method improves the quality of rendered images over previous disocclusion hole-filling algorithms by up to 3.9dB in PSNR

Cheung, G.

Dooley, L. S.

Reel, S.

Wong, K. C. P.

English

Open Research Online

Open Research OnlineThe Open University’s repository of research publicationsand other research outputsDisocclusion Hole-Filling in DIBR-Synthesized Imagesusing Multi-Scale Template MatchingConference or Workshop ItemHow to cite:Reel, S.; Wong, K. C. P.; Cheung, G. and Dooley, L. S. (2014). Disocclusion Hole-Filling in DIBR-Synthesized Imagesusing Multi-Scale Template Matching. In: Visual Communications and Image Processing Conference, 2014 IEEE,IEEE, pp. 494–497.For guidance on citations see FAQs.c© 2014 IEEEVersion: ProofLink(s) to article on publisher’s website:http://dx.doi.org/doi:10.1109/VCIP.2014.7051614Copyright and Moral Rights for the articles on this site are retained by the individual authors and/or other copyrightowners. For more information on Open Research Online’s data policy on reuse of materials please consult the policiespage.oro.open.ac.ukDisocclusion Hole-Filling in DIBR-SynthesizedImages using Multi-Scale Template MatchingSmarti Reel1, K. C. P. Wong1, Gene Cheung2, Laurence S. Dooley11Department of Computing and Communications, The Open University, Milton Keynes, United Kingdom2National Institute of Informatics, Tokyo, JapanEmail: 1{smarti.reel, k.c.p.wong, laurence.dooley}@open.ac.uk, 2cheung@nii.ac.jpAbstract—Transmitting texture and depth images of capturedcamera view(s) of a 3D scene enables a receiver to synthesizenovel virtual viewpoint images via Depth-Image-Based Rendering(DIBR). However, a DIBR-synthesized image often containsdisocclusion holes, which are spatial regions in the virtual viewimage that were occluded by foreground objects in the capturedcamera view(s). In this paper, we propose to complete thesedisocclusion holes by exploiting the self-similarity characteristicof natural images via nonlocal template-matching (TM). Specif-ically, we first define self-similarity as nonlocal recurrences ofpixel patches within the same image across different scales–onecharacterization of self-similarity in a given image is the scalerange in which these patch recurrences take place. Then, atencoder we segment an image into multiple depth layers usingavailable per-pixel depth values, and characterize self-similarityin each layer with a scale range; scale ranges for all layersare transmitted as side information to the decoder. At decoder,disocclusion holes are completed via TM on a per-layer basisby searching for similar patches within the designated scalerange. Experimental results show that our method improves thequality of rendered images over previous disocclusion hole-fillingalgorithms by up to 3.9dB in PSNR.Index Terms—Free viewpoint video, depth-image-based ren-dering, image inpaintingI. INTRODUCTIONBy transmitting both texture maps (color images) and depthmaps (per-pixel distance between objects in the 3D sceneand the capturing camera) captured from one or more cameraview(s), free viewpoint video [1] enables a user the abilityto synthesize novel virtual view images via depth-image-based rendering (DIBR) [2]. In a nutshell, DIBR copies colorpixels in the camera-captured view(s) to corresponding pixellocations in the virtual view image, given 3D geometric in-formation provided by the depth map(s). A DIBR-synthesizedimage often contains disocclusion holes, however, which arespatial regions in the virtual view that were occluded byforeground (FG) objects in the camera-captured view(s) butbecame visible after the view-switch. Satisfactory filling ofdisocclusion holes is essential to the free viewpoint visualexperience. This paper addresses the disocclusion hole-fillingproblem.Coincidentally, the computer vision community has studieda related image inpainting problem over the last decade [3]–[5]: completion of missing pixel regions in a natural image.Broadly speaking, there are two categories of approaches.In the first category are schemes that locally extrapolatesignals based on partial differential equations [4], Fourieranalysis [5] etc. While intuitive, these local schemes do notperform well when the missing pixel regions are large. Inthe second category are nonlocal schemes such as Criminisi’stemplate-matching (TM) [3] algorithm, that fill in missingpixels in a target region by identifying similar pixel patternsin faraway known region, assuming the well recognized self-similarity characteristic commonly observed in natural images.Recently, TM has been adopted for disocclusion hole-fillingin DIBR-synthesized images as well [6], [7]. There remaintwo problems. First, there may not always exist self-similarpatches of the same scale in a given image for TM to properlyfill in missing pixels in disocclusion holes. Second, nonlocalTM schemes tend to be computationally complex due to theexhaustive search employed in the large known pixel region.In this paper we propose a new sender-guided disocclusionhole-filling scheme that addresses the two aforementionedproblems in previous nonlocal TM schemes. Specifically, wefirst redefine self-similarity in a multi-scale manner for naturalimages−a characterization of self-similarity for a given naturalimage is then how well target pixel patches will match withnonlocal patches of the same image resized by a specifiedrange of scaling factors. Next, we design a sender-guideddisocclusion hole-filling algorithm, where i) at encoder wedivide a camera-captured texture image into multiple depthlayers, characterize self-similarity in each layer and transmitthe characterization parameters to decoder as side information(SI); ii) at decoder we perform TM for disocclusion hole-filling only within suitable depth layers but across multiplescales as specified by the transmitted self-similarity parame-ters. Performing TM within a subset of layers means searchcomplexity for matching patches in known region is drasticallyreduced. Experimental results show that our disocclusion hole-filling algorithm outperforms previous schemes by up to 3.9dBin PSNR at comparable or smaller computation complexity.The outline of the paper is as follows. In Section II we for-978-1-4799-6139-9/14/$31.00 c©2014 IEEEMulti-view EncoderDepth TextureTransmission NetworkProposedEncoder Side Processing Depth segmentationScale range characterization (each depth layer)Multi-view DecoderDepth TextureView RenderingViewSynthesisProposed Decoder Side ProcessingDepth layer selectionand hole-filling Fig. 1. Proposed processing blocks in multi-view contextmally define our proposed notion of self-similarity in a multi-scale manner. Using this definition, we describe our sender-guided disocclusion hole-filling algorithm based on multi-scaleself-similarity in Section III. We present experimental resultsand conclusion in Section IV and V, respectively.II. CHARACTERIZATION OF SELF-SIMILARITYIt is observed that natural images are self-similar in general;i.e., a given pixel patch is likely to recur one or more timesin faraway (nonlocal) spatial regions in the same image.While existing works on inpainting using TM [3] assumethe recurrence take place in the same scale, in this paper wegeneralize the notion to assume that the recurrence of a pixelpatch can take place across multiple scales. More precisely, wecharacterize self-similarity in natural images as the scale range(SR) (parameterized by upper and lower bounds) over whicha given pixel patch is likely to recur within the same image.This multi-scale self-similarity is an intuitive generalization;for example, repeating textural patterns like wallpaper vary insize as the distance to the capturing camera changes.In practice, we compute the upper and lower bound thatcharacterize multi-scale self-similarity as follows. A referencetexture patch of size w × w pixels is first selected in a colorimage. Then each sliding window of size (w + n)× (w + n)pixels is rescaled to w×w pixels where n denotes the scalingfactor within a given candidate range. Using mean squarederror (MSE) as the distortion metric, for each n we identifythe number of best-matched patches at this scaling factor andcompare against a threshold T . The range of n values forwhich the number of best matched patches is higher than Tdefines the upper and lower bounds that characterize multi-scale self-similarity in this image.III. SENDER-GUIDED HOLE-FILLING ALGORITHMHaving defined our notion of multi-scale self-similarity innatural images, we now describe our disocclusion hole-fillingalgorithm that exploits this multi-scale self-similarity via TM.We first describe operations at the encoder to characterize self-similarity of camera-captured color images;characterizationparameters are then transmitted to the decoder as SI. We thendescribe the operations at the decoder to efficiently performTM guided by the received SI.A. Encoder Side ProcessingAt the encoder (see Fig.1), the objective is twofold:i) segment the camera-captured texture image into depthlayers−contiguous spatial areas with similar depth values, andii) define and transmit SR for each depth layer to the decoderfor sender-guided disocclusion hole-filling. We discuss themin order next.1) Depth Layer Segmentation: The goal of depth layersegmentation is to divide a camera-captured texture image intocontiguous spatial areas that roughly correspond to physicalobjects in the 3D scene. This is done so that multi-scale TMperformed at the decoder can be done per layer instead of perimage, reducing complexity. This is reasonable, since repeatedtextural patterns likely recur within the same physical object,contained in a depth layer. Let IT and ID be the texture anddepth maps of a camera-captured view, respectively. We firstdivide depth map ID into k layers by detecting peaks andvalleys in a constructed histogram of depth values (see Fig2(a) for an example) [8]. The depth cut-off values (based onvalleys) D = {di} correspond to the segmented depth layersY = {yi} where i = 1, 2, ..., k and dk = 255. The samesegmentation is then applied to the corresponding color imageIT as well.2) Scale Range Characterization: We now characterizemulti-scale self-similarity for each computed texture (color)layer. For target in TM, we consider only patches near theboundary of given layers. The reason is that disocclusionholes tend to appear near FG object boundaries [2]. In ourexperiment, we consider scale value n within range [−3, 3].As illustration, Fig 2(b) shows the number of best matches forvarious scale values within the range [−3, 3] for a depth layerin Middlebury′s Aloe image. Here, it is observed that thenumber of best matches exceeds T1 at n = 0 and n = 1, hencethe SR= [0, 1]. The chosen SR for each layer is transmittedas SI to the decoder for sender-guided disocclusion hole-filling. Note that the SI transmission accounts for only a verysmall signaling overhead (0.01%) compared to the size of thecamera-captured texture and depth maps.B. Decoder Side ProcessingThe decoder receives a pair of texture and depth mapsIT , ID and SR per depth layer, as shown in Fig. 1. Fordisocclusion hole-filling, a recent Joint Texture and DepthInpainting (JTDI) algorithm [7] fills texture and depth holepixels alternately: use available depth information to fill intextural pixel holes, then use inpainted textural informationto fill corresponding depth pixel holes. However, JTDI stillemploys full-image TM, which is computation-expensive. Incontrast, we perform joint texture and depth pixel filling ofdisocclusion holes as done in [7], but employ multi-scale1The value of T is chosen empirically as 75. The experimental results arenot particularly sensitive to this chosen value.Layer 1Layer 20246x103Number of pixels100 255Depth value(a)03 2 1 0 -1 -2 -350100150200Scale value (n)Number of best matches(b)Fig. 2. (a) illustrates depth map histogram, k = 2 (b) shows bar graphrepresenting number of best patches per scale value n for Aloe image.TM within suitable depth layers. Here, SR provides sideinformation on the suitable scaling values to resize candidatepatches for each depth layer during TM.The virtual texture view (VT ) and depth map (VD) are syn-thesized from IT and ID via DIBR. Before filling disocclusionholes, both VD and VT are segmented with the same cut-off values D as discussed earlier. The following sub-sectionexplains the depth layer selection method in our proposeddisocclusion hole-filling algorithm.1) Depth Layer Selection and Hole-filling: First, we selectthe target patch PT and the corresponding depth target patchDT . The order of selecting target patches for filling is veryimportant, but is outside the scope of this paper; we willsimply use the PT selection method in [7]. The known valuesof DT are used to determine the mean of depth values dmean.Since disocclusion holes are missing pixels from background(BG) region [9], dmean facilitates the selection of appropriateBG depth layer(s) Yb as follows:Yb = {yi ∈ Y | dmean ≤ di, di ∈D} where i = 1, 2, ..., k. (1)The SR corresponding to depth layer(s) Yb helps in gen-erating multi-scale candidate search space X by rescalingthe patches for given values in the range such that X ={x1, x2, ..., xq} where q represents number of patches in X .This search space is used for finding best candidate patch CPas follows:CP = min MSE(xj , PT ) where j = 1, 2, ..., q (2)The known pixels of selected CP corresponding to theunknown (holes) pixels of PT are then copied into PT . Thisprocess repeats until all the disocclusion holes are filled.IV. EXPERIMENTAL SETUP AND RESULTSThe proposed algorithm was tested and evaluated usingfour Middlebury datasets [10]: Aloe, Baby2, Books andCones. These datasets contain seven different captured viewsof the same static scene, as well as disparity maps for views#1 and #5. For each sequence, DIBR has been used togenerate the reference view #3 using texture and the disparitymap of view #1. To quantitatively and qualitatively evaluatethe performance of proposed algorithm, the generated view#3 was inpainted using a patch-size of 9 × 9 pixels andcompared with Criminisi [3] and JTDI [7] methods. Both thecomparators, Criminisi [3] and JTDI [7] employs single-scaleexhaustive TM to find CP for filling disocclusion holes. Fornumerical analysis, the original view #3 of image datasetswas used as the ground truth for all peak signal-to-noiseratio (PSNR) calculations, with the PSNR computed for boththe whole image and hole regions. All experiments wereperformed on an Ubuntu 12.04 64-bit with 3.10 GHz IntelQuadCore and 4GB RAM, with all algorithms implementedin MATLAB.A. Quantitative Result AnalysisTable I shows the PSNR results for three inpainting meth-ods, which reveal that proposed algorithm performs consis-tently better than [3] and [7] for all four datasets. In Booksdataset for example, the PSNR increased by up to 2dB and1.76dB for whole image as compared to Criminisi [3] andJTDI [7] respectively, while the corresponding results for onlythe hole regions, show an increase of 3.92dB and 3.44dB.This confirms that characterization parameters provided in SRenhances the probability of finding a better match by searchingCP at two scales, i.e. at scale n = 0 and n = −1. The PSNRincrease supports the fact that during the disocclusion fillingprocess, there are cases where patches at scale n = −1 providebetter matches than scale n = 0 (same scale). The computationcost due to multi-scale TM is compensated by layer basedsearch during inpainting and the overall computation timeremains either comparable or smaller than exhaustive TM.Similar observations can be made upon the results for Aloe,Baby2 and Cones images using proposed algorithm.B. Qualitative Result AnalysisFrom a perceptual quality perspective, Fig. 3 shows thequalitative comparison of proposed algorithm with [3] and [7]with example zoomed-in regions for Aloe, Baby2, Booksand Cones in each row respectively. The proposed algorithmagain provides improved visual quality and fewer artefacts bypreserving the FG object boundaries as shown in Fig. 3(d)in comparison to Criminisi [3] (Fig. 3(b)) and JTDI [7] (Fig.3(c)). For comparison of various techniques, the disocclusionholes and ground truth are shown in Fig. 3(a) and 3(e),respectively. Multi-scale TM reduces the artefacts and fills thedisocclusion holes with enhanced perceptual quality.TABLE IPSNR COMPARISON FOR TEXTURE IMAGE INPAINTING (in dB)Whole Image Hole RegionsDataset Criminisi JTDI Proposed Criminisi JTDI Proposed[3] [7] n [3] [7]Books 26.80 27.05 -1,0 28.81 15.14 15.62 19.06Baby2 30.87 30.92 1, 0, -1 31.80 19.24 19.36 21.75Aloe 26.83 27.83 1, 0 28.14 17.02 18.66 19.50Cones 22.48 23.17 -1, 1, 2 23.41 17.45 19.92 20.34(a) Disocclusion holes (b) Criminisi [3] (c) JTDI [7] (d) Proposed (e) Ground truthFig. 3. shows Aloe (row 1), Baby2 (row 2), Books (row 3) and Cones (row 4) with corresponding (a) disocclusion holes, filled by (b) Criminisi [3], (c)JTDI [7], (d) Proposed method and respective (e) Ground truth.V. CONCLUSIONWhile free viewpoint video enables the synthesis of novelvirtual view images at decoder via DIBR, the synthesizedimages often contain disocclusion holes that require properfilling. In this paper, we propose a new disocclusion hole-filling algorithm that exploits multi-scale self-similarity dur-ing TM. Though searching for nonlocal patches of differentscales entails a larger search space, we contain the resultingsearch complexity by performing TM only within designateddepth layers−subset of the image with similar depth values.Experimental results show that our proposed hole-filling algo-rithm can outperform previous proposals by up to 3.9dB atcomparable or smaller complexity.REFERENCES[1] K. Takeuchi, N. Fukushima, T. Yendo, M. Panahpour Tehrani, T. Fujii,and M. Tanimoto, “Free-viewpoint image generation from a videocaptured by a handheld camera,” vol. 7863, 2011, pp. 78 631N–78 631N–9.[2] D. Tian, P.-L. Lai, P. Lopez, and C. Gomila, “View synthesis techniquesfor 3d video,” Proc. SPIE, vol. 7443, pp. 74 430T–74 430T–11, 2009.[3] A. Criminisi, P. Perez, and K. Toyama, “Region filling and objectremoval by exemplar-based image inpainting,” Image Processing, IEEETransactions on, vol. 13, no. 9, pp. 1200–1212, Sept 2004.[4] D. Tschumperle and R. Deriche, “Vector-valued image regularizationwith pdes: a common framework for different applications,” PatternAnalysis and Machine Intelligence, IEEE Transactions on, vol. 27, no. 4,pp. 506–517, April 2005.[5] J.-F. Cai, R. Chan, L. Shen, and Z. Shen, “Simultaneously inpaintingin image and transformed domains,” vol. 112, no. 4. Springer-Verlag,2009, pp. 509–533.[6] I. Daribo and B. Pesquet-Popescu, “Depth-aided image inpainting fornovel view synthesis,” in Multimedia Signal Processing (MMSP), 2010IEEE International Workshop on, Oct 2010, pp. 167–170.[7] S. Reel, G. Cheung, P. Wong, and L. Dooley, “Joint texture-depth pixelinpainting of disocclusion holes in virtual view synthesis,” in Signaland Information Processing Association Annual Summit and Conference(APSIPA), 2013 Asia-Pacific, Oct 2013, pp. 1–7.[8] D. De Silva, W. Fernando, H. Kodikaraarachchi, S. Worrall, and A. Kon-doz, “Adaptive sharpening of depth maps for 3dtv,” Electronics Letters,vol. 46, no. 23, pp. 1546–1548, November 2010.[9] I. Ahn and C. Kim, “Depth-based disocclusion filling for virtual viewsynthesis,” in Multimedia and Expo (ICME), 2012 IEEE InternationalConference on, July 2012, pp. 109–114.[10] D. Scharstein and C. Pal, “Learning conditional random fields forstereo,” in Computer Vision and Pattern Recognition, 2007. CVPR ’07.IEEE Conference on, June 2007, pp. 1–8.

Disocclusion Hole-Filling in DIBR-Synthesized Images using Multi-Scale Template Matching

Transmitting texture and depth images of captured camera view(s) of a 3D scene enables a receiver to synthesize novel virtual viewpoint images via Depth-Image-Based Rendering (DIBR). However, a DIBR-synthesized image often contains disocclusion holes, which are spatial regions in the virtual view image that were occluded by foreground objects in the captured camera view(s). In this paper, we propose to complete these disocclusion holes by exploiting the self-similarity characteristic of natural images via nonlocal template-matching (TM). Specifically, we first define self-similarity as nonlocal recurrences of pixel patches within the same image across different scales-one characterization of self-similarity in a given image is the scale range in which these patch recurrences take place. Then, at encoder we segment an image into multiple depth layers using available per-pixel depth values, and characterize self-similarity in each layer with a scale range; scale ranges for all layers are transmitted as side information to the decoder. At decoder, disocclusion holes are completed via TM on a per-layer basis by searching for similar patches within the designated scale range. Experimental results show that our method improves the quality of rendered images over previous disocclusion hole-filling algorithms by up to 3.9dB in PSNR.</p

Reel, Smarti

Cheung, Gene

Dooley, Laurence S.

University of Dundee Online Publications

Disocclusion hole-filling in DIBR-synthesized images using multi-scale template matching

Disocclusion Hole-Filling in DIBR-Synthesized Images using Multi-Scale Template Matching

Abstract

Similar works

Full text

Available Versions

Open Research Online

University of Dundee Online Publications

University of Dundee Online Publications

Crossref