Bertel, Tobias

Holzer, Stefan

Ortiz-Cayon, Rodrigo

Rao, Srinivas

Richardt, Christian

Tomoto, Yusuke

English

Hedman Peter

Ritschel Tobias

Crossref

Deferred Neural Rendering for View Extrapolation

Image-based rendering methods that support visually pleasing specular surface reflections require accurate surface geometry and a large number of input images. Recent advances in neural scene representations show excellent visual quality while requiring only imperfect mesh proxies or no surface-based proxies at all. While providing state-of-the-art visual quality, the inference time of learned models is usually too slow for interactive applications. While using a casually captured circular video sweep as input, we extend Deferred Neural Rendering to extrapolate smooth viewpoints around specular objects like a car.This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 66599

OPUS

        Citation for published version:Bertel, T, Tomoto, Y, Rao, S, Ortiz-Cayon, R, Holzer, S & Richardt, C 2020, Deferred Neural Rendering for ViewExtrapolation. in SA '20 Posters: SIGGRAPH Asia 2020 Posters., 39, Association for Computing Machinery, pp.1-2, ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia,Online, Korea, Republic of, 4/12/20. https://doi.org/10.1145/3415264.3425441DOI:10.1145/3415264.3425441Publication date:2020Document VersionPeer reviewed versionLink to publication© ACM, 2020. This is the author's version of the work. It is posted here by permission of ACM for your personaluse. Not for redistribution. The definitive version was published in SA'20 Posters: SIGGRAPH Asia 2020Posters.  http://doi.acm.org/10.1145/3415264.3425441University of BathAlternative formatsIf you require this document in an alternative format, please contact:openaccess@bath.ac.ukGeneral rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright ownersand it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.Take down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Download date: 18. Sep. 2021Deferred Neural Rendering for View ExtrapolationTobias BertelUniversity ofBathYusukeTomotoFyusion Inc.Srinivas RaoFyusion Inc.RodrigoOrtiz-CayonFyusion Inc.Stefan HolzerFyusion Inc.ChristianRichardtUniversity ofBathDatasetObjPosesMesh UV mapvCaptureShiny objectDeferred Neural Rendering [Thies et al. 2019]Generative Adversarial NetworkOptimizeSampled TextureOutput (fake)Discriminator InputGeneratorResultsExtrapolationInterpolationDNR OursFigure 1: We capture an input video with a consumer camera, estimate camera poses, reconstruct a mesh and uv-map it. Weextend Deferred Neural Rendering [Thies et al. 2019] (blue) to enable smooth extrapolation of novel viewpoints (orange).ABSTRACTImage-based rendering methods that support visually pleasing spec-ular surface reflections require accurate surface geometry and alarge number of input images. Recent advances in neural scenerepresentations show excellent visual quality while requiring onlyimperfect mesh proxies or no surface-based proxies at all. While pro-viding state-of-the-art visual quality, the inference time of learnedmodels is usually too slow for interactive applications. While usinga casually captured circular video sweep as input, we extend De-ferred Neural Rendering to extrapolate smooth viewpoints aroundspecular objects like a car.CCS CONCEPTS• Computer graphics→ Neural rendering.KEYWORDSNovel-view synthesis, surface light field, extrapolationACM Reference Format:Tobias Bertel, Yusuke Tomoto, Srinivas Rao, Rodrigo Ortiz-Cayon, StefanHolzer, and Christian Richardt. 2020. Deferred Neural Rendering for ViewExtrapolation. In SIGGRAPH Asia 2020 (SA ’20 Posters), December 04-13, 2020.ACM, New York, NY, USA, 2 pages. https://doi.org/10.1145/3415264.3425441Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).SA ’20 Posters, December 04-13, 2020, Virtual Event, Republic of Korea© 2020 Copyright held by the owner/author(s).ACM ISBN 978-1-4503-8113-0/20/11.https://doi.org/10.1145/3415264.34254411 INTRODUCTIONThe reconstruction quality of casually captured data and the cho-sen scene representation constrain the development of interactivevisual experiences, in particular in real-world environments. Tradi-tionally, view synthesis of real-world environments is performedvia image-based rendering (IBR), whose quality is mostly limited bythe accuracy of the available scene proxy geometry. If RGB-D videois available, Park et al. [2020] show how to estimate scene propertieslike material and environment map, and use a physically motivatedneural renderer for compositing Fresnel effects. We focus on neuralscene representations and rendering methods that do not rely onexplicit geometry at all, i.e. posed images only, or address the usageof an imperfect proxy geometry. Volumetric scene representations,e.g. NeRF [Mildenhall et al. 2020], register all input viewpointswithin a learned semi-transparent volume. While the results arestate-of-the-art visually, the inference time of the trained modelsis prohibitively slow for interactive applications. Liu et al. [2020]recently presented a sparse neural voxel representation suitablefor indoor environments that renders 1–2 fps. Image translationmethods used for image synthesis [Isola et al. 2017] show temporalartefacts since there is no global registration of the input viewsand they cannot be expected to work for non-diffuse scene objects.Hedman et al. [2018] use an imperfect proxy geometry and learnhow to blend multiple images to mitigate rendering artefacts.2 OUR APPROACHThe baseline of our system trains a generative adversarial network(GAN) to learn a mapping from view-dependent neural features toobject appearance [Thies et al. 2019]. The learned features extractedfrom the neural texture are interpreted by the generator as a learnedsurface light field. We present a set of baseline extensions whichlead to improved extrapolation performance. Note that the baseline,SA ’20 Posters, December 04-13, 2020, Virtual Event, Republic of Korea Bertel, Tomoto, Rao, Ortiz-Cayon, Holzer, RichardtMeshvGeneratorDiscriminator OptimizeNoiseGenerator inputBackgroundNeural FeaturesInputOutputTexture atlas Additional generator guidesuv-MapForward Backprop Tuning(1)(2)(3)(4) (5)Baseline ExtensionsPoses(6) (7)(8) (9)(10)Optical centres Positions Normals16-D per pixelNeural features,background,guidesInputs(0)Figure 2: Overview of our baseline approach (blue) and our proposed extensions (orange). See detailed description in the text.similar to most learning-based techniques, is designed to interpolatethe training corpus. Deferred neural rendering is motivated byadding learnable components to the deferred rendering pipeline[Ritschel et al. 2012]. An overview of the baseline and our extensionsis given in Figure 2. Mathematically, the goal is to find a combinationof neural texture T and neural renderer R that minimizes the imagere-rendering loss L over the training dataset D of𝑀 posed imagesD = {𝐼𝑘 , 𝑝𝑘 }𝑀𝑘=1 created from our capture. 𝐼𝑘 is the 𝑘-th imageof the training dataset and its corresponding camera pose 𝑝𝑘 , i.e.viewing direction v and optical centre c. The optimal neural textureT ∗ and renderer R∗ are obtained by solving:T ∗,R∗ = argminT,R∑𝑑∈DL(𝐴(𝑑) | 𝐹𝑑 (T ),𝐺𝑑 (R)). (1)The baseline is obtained by setting 𝐹𝑑 (·) = 𝐺𝑑 (·) = 𝑖𝑑 . The augmen-tation operator 𝐴(·) needs to return crops during training and itsinput during testing (i.e. identity). Our extensions address (i) aug-menting training samples 𝑑 ∈ D, (ii) adding inputs to the rendererR, and (iii) injecting noise into the view- and thus dataset-dependentfeature generation 𝐹 (·) or the generator guides 𝐺 (·).Forward pass. See (0–7) in Figure 2. Per data item 𝑑 ∈ D: (0)Rasterize a viewpoint and obtain uv-map (deferred rendering). (1)Use uv-map to look up texture atlasA. (2) Retrieve neural features 𝑓 ,16-D texels, from A. The viewing direction v ∈ 𝑝 ∈ 𝑑 is convertedto spherical harmonics. The first 2 bands are used, i.e. 9 coefficients,which are multiplied with the neural texel channels 4–12 of 𝑓𝑖 . (3)Create background by eroding the uv-map from the training image𝐼 . (4) Encode generator input and decode it to produce output image(5). Note that the generator interprets the feature encoding used in(2). (6) Feed output and target into discriminator. (7) Optimize.Backpropagation. See (8–10) in Figure 2. (8) Update atlasA, keepview-dependent (specular) texels on finer levels. Regress neuralfeature channels 𝑓 [0 : 2] with the training image 𝐼 . The motivationhere is to get an estimate of the diffuse color of the surface. (9)Update generator via loss between the generated output𝑂 and dataitem image 𝐼 . (10) Update discriminator.Our extensions. See orange annotations in Figure 2. The baselineis extended in three ways: (i) A guided augmentation procedure𝐴(·)is introduced that focuses on poorly inferred image regions whenfetching dataset items 𝑑 leading to more efficient training. SSIM isused to determine these image regions. (ii) Additional viewpointinformation is added to the generator input, specifically opticalcentres, positions and normals. (iii) Noise is added to viewing direc-tions v and newly added guidance signal of the generator input𝐺𝑑 .Note that the noise injected into the viewing directions v reducesextrapolation artefacts significantly.3 CONCLUSIONIndividual viewpoints can be extrapolated smoothly, but there isno guarantee for inter-frame coherency. Note that the resulting tem-poral flickering could be reduced by: (i) providing a more accurateproxy geometry, (ii) increasing the image density and (iii) stay-ing closer to the input viewpoints. While the baseline method isdesigned to interpolate the training corpus, as well as most otherpopular neural scene representations (e.g. NeRF), extrapolationcauses uncertainty and thus noise. Our extensions technically tradethis noise with blur. It seems promising to incorporate more objectand scene information into the process and think of inpainting neu-ral scene representations for view extrapolation tasks. It would begreat to generalise the current representation in a physically moti-vated manner and surround the whole system with a differentiablerendering system to learn the components needed for real-timephoto-real view synthesis. The current representation misses edit-ing capabilities which hinders its practical applications.ACKNOWLEDGMENTSWe want to thank Justus Thies for fruitful discussions and shar-ing the code for DNR. This work was supported by EU Horizon2020 MSCA grant FIRE (665992), the EPSRC Centre for DoctoralTraining in Digital Entertainment (EP/L016540/1), RCUK grantCAMERA (EP/M023281/1), an EPSRC-UKRI Innovation Fellowship(EP/S001050/1), and a Rabin Ezra Scholarship.REFERENCESPeter Hedman, Julien Philip, True Price, Jan-Michael Frahm, George Drettakis, andGabriel Brostow. 2018. Deep Blending for Free-viewpoint Image-based Rendering.ACM Trans. Graph. 37, 6 (2018), 257:1–15. https://doi.org/10.1145/3272127.3275084Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-imagetranslation with conditional adversarial networks. In CVPR. https://doi.org/10.1109/CVPR.2017.632Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. 2020.Neural Sparse Voxel Fields. arXiv:2007.11571Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ra-mamoorthi, and Ren Ng. 2020. NeRF: Representing Scenes as Neural RadianceFields for View Synthesis. In ECCV. https://doi.org/10.1007/978-3-030-58452-8_24Jeong Joon Park, Aleksander Holynski, and Steve Seitz. 2020. Seeing the World in aBag of Chips. In CVPR. https://doi.org/10.1109/CVPR42600.2020.00149Tobias Ritschel, Carsten Dachsbacher, Thorsten Grosch, and Jan Kautz. 2012. TheState of the Art in Interactive Global Illumination. Comput. Graph. Forum 31, 1 (Feb.2012), 160–188. https://doi.org/10.1111/j.1467-8659.2012.02093.xJustus Thies, Michael Zollhöfer, and Matthias Nießner. 2019. Deferred Neural Render-ing: Image Synthesis Using Neural Textures. ACM Trans. Graph. 38, 4 (July 2019),66:1–12. https://doi.org/10.1145/3306346.3323035

https://purehost.bath.ac.uk/ws/files/218007912/DNR4VE_BertelEtAl_SA2020_Poster_Abstract.pdf

Deferred Neural Rendering for View Extrapolation

Abstract

Similar works

Full text

Available Versions

Crossref

OPUS