One of the main aims of emerging audio-visual (AV) applications is to provide interactive navigation within a captured event or scene. This paper presents a view synthesis algorithm that provides a scalable and flexible approach to virtual viewpoint synthesis in multiple camera environments. The multi-view synthesis (MVS) process consists of four different phases that are described in detail: surface identification, surface selection, surface boundary blending and surface reconstruction. MVS view synthesis identifies and selects only the best quality surface areas from the set of available reference images, thereby reducing perceptual errors in virtual view reconstruction. The approach is camera setup independent and scalable as virtual views can be created given 1 to N of the available video inputs. Thus, MVS provides interactive AV applications with a means to handle scenarios where camera inputs increase or decrease over time

Cooke, Eddie

O'Connor, Noel E.

English

O\u27Connor, Noel E.

Irish Universities

Scalable virtual viewpoint image synthesis for multiple camera environments

DCU Online Research Access Service

Scalable Virtual Viewpoint Image Synthesis for Multiple Camera EnvironmentsE. Cooke, N. O’ConnorCentre for Digital Video Processing, Dublin City University, Dublin, Irelandejcooke@eeng.dcu.ie, oconnorn@eeng.dcu.ieAbstractOne of the main aims of emerging audio-visual (AV) ap-plications is to provide interactive navigation within a cap-tured event or scene. This paper presents a view synthe-sis algorithm that provides a scalable and ﬂexible approachto virtual viewpoint synthesis in multiple camera environ-ments. The Multi-View Synthesis (MVS) process consists offour different phases that are described in detail: surfaceidentiﬁcation, surface selection, surface boundary blendingand surface reconstruction. MVS view synthesis identiﬁesand selects only the best quality surface areas from the setof available reference images, thereby reducing perceptualerrors in virtual view reconstruction. The approach is cam-era setup independent and scalable as virtual views can becreated given 1 to N of the available video inputs. Thus,MVS provides interactive AV applications with a means tohandle scenarios where camera inputs increase or decreaseover time.1. IntroductionOne of the main aims of emerging audio-visual (AV) ap-plications is to remove the passiveness of viewing a cap-tured event or scene [8]. Such systems encourage users toexplore and navigate AV scenes by allowing them to specifytheir own viewpoint and orientation. Recently, the MovingPicture Experts Group (MPEG) of ISO/IEC formed a work-ing group, called 3DAV, to investigate the need for standard-ization in this area [9].Theoretically, all the visual information associated witha 3D scene can be speciﬁed using the 7D plenoptic func-tion [1]. However, in practical terms it is not possible toentirely deﬁne this scene representation method. Instead,captured reference images of a scene can be regarded asa sparse set of samples of the complete plenoptic function.The more samples provided, the less additional scene geom-etry information required for image-based novel view cre-ation. Image-based rendering (IBR) techniques create vir-tual views from a set of scene images with associated pointcorrespondences [7]. They are most suited to an environ-ment where the scene interaction area is well deﬁned andcan be captured using an N camera setup.The process of identifying and combining informationfrom captured images to create novel views is called viewsynthesis. A number of restrictive approaches exist for se-lecting which camera in a stereo setup best captures a sur-face and how these surfaces should be combined [2]. How-ever, these techniques fail when migrated to an arbitrarymulti-camera environment. Existing multiple image viewsynthesis approaches avoid surface identiﬁcation and sim-ply combine all available surface information, usually viaview orientation weighting, in order to create virtual views[6]. This implies that although more original information isavailable, errors due to occlusions, depth mismatches etc;are incorporated into the ﬁnal virtual view. In this paper wepresent a new view synthesis approach that identiﬁes and se-lects only the best quality surface areas from available refer-ence images, therefore reducing perceptual errors in virtualview reconstruction. The approach is camera setup inde-pendent and scalable as virtual views can be created given1 to N of the available video inputs.The layout of the paper is as follows: a new scalable vir-tual view synthesis algorithm called Multi-View Synthesis(MVS) is described in Section 2. In Section 3 experimen-tal results are presented and veriﬁed by comparison withground truths. Finally, in Section 4 conclusions and futurework are described.2. Scalable Virtual Viewpoint SynthesisOf the AV application scenarios being investigated by3DAV the most challenging is that of Free Viewpoint Video(FVV) [9]. FVV applications are designed to allow unre-stricted scene navigation and are captured using a multiplecamera setup. Current view synthesis solutions in FVV arebased on using either images with no scene geometry infor-mation that are obtained via a dense sampling of the scene,implying large information redundancy [10], or reconstruct-ing complex 3D models of the scene from a sparse camerasetup [4].Proceedings of the Ninth International Conference on Information Visualisation (IV’05) 1550-6037/05 $20.00 © 2005 IEEE (a) (b) (c) (d) (e)(f) (g) (h) (i)Figure 1. BUSINESS-MAN (a)-(d) Reference images A, B, C and D. (e) Ground truth. (f)-(i) 2D SurfaceSampling after warping (a)-(d) respectively.The MVS algorithm provides a scalable and ﬂexible ap-proach to view synthesis for FVV. MVS is scalable as it’sdesigned to handle an arbitrary number of reference imagesand is therefore suitable for both sparse and dense camerasampling. At the core of MVS is a ﬂexible deﬁnition ofwhat constitutes a 3D scene surface. This ﬂexibility ensuresthat the best quality virtual view reconstruction is producedfrom the available views. TheMVS process is a virtual viewdependent approach implying that it examines the referenceimages at the virtual viewpoint before the surface identiﬁ-cation and selection process occurs. It requires the camerasetup to be calibrated and that each reference image has acorresponding disparity/depth map [3].Fig. 1(a)-(d) presents four different camera views takenwithin a test environment that will be used to illustrate theMVS approach. Fig. 1(e) is taken from a camera placedat the position of the ﬁnal virtual view and will be usedas a ground truth to indicate the correctness of the approach.2.1. Surface IdentiﬁcationThe ﬁrst step in MVS surface identiﬁcation is to warpthe reference images to the virtual view position. After im-age warping, surface holes arise due to both sampling gapsand surface disclosures [2]. The 2D displacement across thesurfaces is a measure of the extent of the sampling of the 3Dsurfaces within the reference image and can be used as cri-teria for virtual view surface quality recognition. Fig. 1(f)-(i) illustrates the 2D sampling of the surface points (everyﬁfth sample) at the virtual viewpoint for reference imagesFig. 1(a)-(d), respectively. Examining the 2D surface sam-pling we can determine the following:• Corresponding 3D scene points/surfaces visible acrossreference images are warped to the same position atthe virtual viewpoint.• These matching virtual view areas/surfaces in thewarped images have varying sampling densities.Therefore, each reference image deﬁnes its own representa-tion of the 3D surfaces required at the virtual view position.How we identify their surface quality is determined usingthe Sampling Density Map (SDM).Let q be a surface sample in the reference image that iswarped to position q′ in the virtual view. Let q1 − q8 be theset of reference image neighbours of q deﬁned by a 3 × 3block centred at q, we call these samples the local surfaceof q. The displacement between a sample q′ = [x′, y′] andany of its warped reference image neighbours q′i = [x′i, y′i]can be determined via Eq. (1):ρq′(q′i) =√(x′i − x′)2 + (y′i − y′)2 (1)A measure of the extent of the displacement of q′ with re-spect to its warped local surface is computed using the sam-pling density function:δ(q′) =∑Ni=1 ρq′(q′i)N(2)where N is the number of reference image sample neigh-bours of q in the local surface. A sampling density value iscomputed for every pixel sample in the warped referenceimage and stored in the SDM. The higher the value at asample the larger the displacement from its local surface inthe warped view. Implying the surface is either undersam-pled in the reference image or that the sample’s referenceimage neighbourhood lies on different sides of a depthdiscontinuity visible from the virtual viewpoint.Proceedings of the Ninth International Conference on Information Visualisation (IV’05) 1550-6037/05 $20.00 © 2005 IEEE (a) (b) (c) (d)(e) (f) (g) (h) (i)Figure 2. (a)-(d) Improvement in virtual viewpoint SDM as reference images added incrementally.Colour-bar indicates SDM values.(e) Surface Map for surface selection of (d). (f)-(i) Surface recon-struction corresponding to surface selections of (a)-(d).2.2. Surface SelectionThe MVS surface selection process determines the sur-faces across the available surface representations to be usedfor view synthesis. This selection process is based on a sam-pling density weighting scheme. We have previously notedthat corresponding 3D scene points or surfaces visible inmore than one reference image are warped to the same po-sition in the virtual view. Therefore, examining the SDMof two reference images A and C, we know that the surfacesample at position δA(x,y) the SDM of A and position δC(x,y)in the SDM of C indicate the reference image’s samplingdensity for the same virtual view position. Hence, we cancompute a sampling density weight function for the surfacesample Q in the SDM of image A, αˆδAQ , via Eq. (3):αˆδAQ = 1−δAQ∑Ni=1 δiQ(3)where N is the number of reference images. We then nor-malise the surface sample sampling density weight acrossthe SDMs to sum to unity via:αδAQ =αˆδAQ∑Ni=1 αˆδiQ(4)The approach to surface selection is to loop through the Navailable SDMs selecting a non-processed surface with thehighest αδ weight on each pass Eq. (5):ΛQ = Φ(αδkQ ) (5)where ΛQ represents the ﬁnal virtual view surface sampleat Q, the αδiQ weight is a measure of the sampling density atsurface sample Q based on reference view i, where i rangesfrom [1, N ], and Φ is a function which selects the surfacesample associated with the maximum weight. Although theapproach is sample based it dynamically groups local sam-ple neighbourhoods into surfaces of similar sampling den-sities. This ensures a virtual viewpoint related surface divi-sion as opposed to a strict depth or texture surface identiﬁ-cation.Fig. 2(a) illustrates a 3D representation of the SDMat the virtual viewpoint when surface samples are takenexclusively from the warped reference image A (Fig. 1(a)).Each SDM value is presented as a height, the higher thevalue the more sparse the sampling, Fig. 1(f). In Fig. 2(b)reference image B is added to the surface selection process.It can be determined that this new SDM is smoother, as thesampling density spikes have decreased, indicating that thesurface sampling is denser and therefore the virtual viewsurface quality has improved. Fig. 2(c) and (d) illustrate theadded improvement in surface quality as images C and D,respectively, are included in the surface selection process.A denser surface sampling implies an increase in surfacequality at the virtual view.2.3. Surface Boundary BlendingIntegrating only the best view of each required surfaceimplies that neighbouring surfaces in the ﬁnal virtualview may be supplied from different reference images.Proceedings of the Ninth International Conference on Information Visualisation (IV’05) 1550-6037/05 $20.00 © 2005 IEEE (a) (b) (c) (d) (e) (f) (g) (h) (i)Figure 3. RABBIT (a)-(d) Reference images A, B, C and D. (e) Ground truth. (f)-(i) Improvement inview synthesis results from incrementally adding images A, B, C and D respectively to the MVSprocess.Fig. 2(e) illustrates a Surface Map indicating the chosensurface samples from images A, B, C and D for the virtualview deﬁned in Fig. 2(d). A surface map is designed toindicate from which reference view the chosen virtualview surfaces originate. Here, the colour red representssurfaces from image A, blue from image B, green fromimage C and yellow from D. At these surface boundaries,specularity and other photometric differences across theimages can cause perceptual seams to appear in the virtualview. In order to lessen this effect a weighted blending isimplemented on an extended boundary region (e.g. ﬁvepixels) around the connected surfaces. We use a ﬁxedlinear ramp blending in these overlapping areas to computecolour values for the ﬁnal virtual view.2.4. Surface ReconstructionView synthesis surface reconstruction deals with two is-sues: those of surface visibility and hole ﬁlling during novelview creation at the virtual viewpoint.Resolving surface visibility occurs at two levels: withineach warped reference image during surface identiﬁcationand across the N images used during view synthesis. Tosolve the former issue we implement a back-to-front warporder [5] to avoid surface occlusion errors. During mul-tiple image view synthesis the surface selection weightingscheme, which identiﬁes only one reference image for a re-quired surface area, resolves any visibility issues.View synthesis hole ﬁlling involves identifying areaswithin warped surfaces where virtual view required surfaceinformation is missing. There are two different types ofholes. Smaller sampling gaps in continuous surfaces areﬁlled using interpolation. While surface disclosures, whicharise due to the movement of a foreground object withrespect to the background, are ﬁlled using the MVS surfaceselection approach. This approach ensures that all thesurfaces across the N reference images are considered forhole ﬁlling and therefore reduces perceptual errors duringsurface reconstruction.3. Experimental ResultsThe MVS algorithm provides a scalable and ﬂexible ap-proach to view synthesis. To indicate the correctness of theapproach we compare the view synthesis results from thetwo test sequences illustrated in Fig. 1 and Fig. 3 with theirrespective ground-truths. In order to demonstrate the scal-ability of the MVS view synthesis method we incremen-tally add reference images to the view synthesis process andcompare all the resulting virtual views. This allows a di-rect comparison of the improvements in virtual view recon-struction as more reference images are added to the MVSprocess.It also indicates the ﬂexibility of the view synthesisapproach as we can clearly determine that the approach cangracefully deal with both a decrease and increase in camerainputs. Fig. 4(a) presents the results of PSNR measuresbetween the MVS view synthesis and the ground-truth overthe 60 frame BUSINESS-MAN test sequence. The graphdetails how the PSNR improves as reference images areadded to the view synthesis process. This improvementis from an average of 29dB for the sequence when justone image is used to an average of 32dB when using allfour. A subjective result is provided in Fig. 2(f)-(i). Thesecond test sequence is the 20 frame RABBIT sequence.Fig. 3(a)-(d) presents a frame from the four camera in-puts, Fig. 3(e) is the ground truth. The PSNR measuresbetween the MVS view synthesis and the ground-truthare presented in Fig. 4(b). The graph details how thePSNR improves as reference images are added to the viewsynthesis process. This improvement is from an average of25dB, when just one image is used, to an average of 33dBusing all four. A subjective result is provided in Fig. 3(f)-(i).Proceedings of the Ninth International Conference on Information Visualisation (IV’05) 1550-6037/05 $20.00 © 2005 IEEE (a) (b)Figure 4. Graph of scalable view synthesis improvements (PSNR) in (a) BUSINESS-MAN (b) RABBITtest sequence.4. Conclusions and Future WorkIn this paper we discussed the shortcomings of currentmultiple image view synthesis approaches. We then pre-sented the MVS algorithm that provides a scalable and ﬂex-ible approach to view synthesis in multiple camera environ-ments.The process consists of four different phases that aredescribed in detail: surface identiﬁcation, surface selec-tion, surface boundary blending and surface reconstruction.MVS identiﬁes and selects only the best quality surface ar-eas from the set of available reference images, thereby re-ducing perceptual errors in virtual view reconstruction. Theapproach is camera setup independent and scalable as vir-tual views can be created given 1 to N of the availablevideo inputs. Thus, MVS provides interactive AV appli-cations with a means to handle scenarios where camera in-puts increase or decrease over time. Experimental resultswere presented and veriﬁed using both objective (PSNR)and subjective comparisons.Ideas for future work include: a pre-processing stepto identify from a very large number of available videoinput streams only those input streams that contain surfacesrequired for the current virtual viewpoint synthesis; andincorporating a system’s currently available bandwidth andprocessing power into the scalability process.5. AcknowledgmentsThe authors would like to acknowledge the support ofScience Foundation Ireland grant number 03/IN.3/I361.References[1] E. Adelson and J. Bergen. The plenoptic function and theelements of early vision. Computational Models of VisualProcessing, pages 3–20, 1991.[2] E. Cooke, P. Kauff, and O. Schreer. Image analysis and syn-thesis for multiple stereo camera teleconferencing systems.Proc WIAMIS, pages 405–410, 2003.[3] C. Fehn, E. Cooke, O. Schreer, and P. Kauff. 3d analysis andimage-based rendering for immersive tv applications. ImageCommunication Journal, 17(9):705–715, 2002.[4] A. Laurentini. The visual hull concept for silhouette basedimage understanding. IEEE PAMI, pages 150–162, 1994.[5] L. McMillan. An Image-Based Approach to Three-Dimensional Computer Graphics. PhD thesis, Universityof North Carolina at Chapel Hill, 1997. Technical ReportTR97-013.[6] K. Mueller, A. Smolic, M. Droese, P. Voigt, and T. Wiegand.Multi-texture modelling of 3d trafﬁc scenes. Proc ICME,2003.[7] H. Shum and S. Kang. A review of image-based renderingtechniques. Proc VCIP, pages 2–13, 2000.[8] A. Smolic and P. Kauff. Interactive 3d video representationand coding technologies. IEEE Special Issue on Advancesin Video Coding and Delivery, 2004.[9] A. Smolic and D. McCutchen. 3dav exploration of video-based rendering technology in mpeg. IEEE TCSVT, pages348–356, 2004.[10] M. Tanimoto. Free viewpoint television - ftv. Proc PCS,2004.Proceedings of the Ninth International Conference on Information Visualisation (IV’05) 1550-6037/05 $20.00 © 2005 IEEE 

3d analysis and image-based rendering for immersive tv applications.

3dav exploration of videobased rendering technology in mpeg.

A review of image-based rendering techniques.

An Image-Based Approach to ThreeDimensional Computer Graphics.

Free viewpoint television - ftv. Proc PCS,

Image analysis and synthesis for multiple stereo camera teleconferencing systems.

Scalable virtual viewpoint image synthesis for multiple camera environments

Abstract

Similar works

Full text

Available Versions

Irish Universities

DCU Online Research Access Service

Crossref

Name not available