12 research outputs found
Rich Intrinsic Image Decomposition of Outdoor Scenes from Multiple Views
International audienceIntrinsic images aim at separating an image into its reflectance and illumination components to facilitate further analysis or manipulation. This separation is severely ill-posed and the most successful methods rely on user indications or precise geometry to resolve the ambiguities inherent to this problem. In this paper we propose a method to estimate intrinsic images from multiple views of an outdoor scene without the need for precise geometry and with a few manual steps to calibrate the input. We use multiview stereo to automatically reconstruct a 3D point cloud of the scene. Although this point cloud is sparse and incomplete, we show that it provides the necessary information to compute plausible sky and indirect illumination at each 3D point. We then introduce an optimization method to estimate sun visibility over the point cloud. This algorithm compensates for the lack of accurate geometry and allows the extraction of precise shadows in the final image. We finally propagate the information computed over the sparse point cloud to every pixel in the photograph using image-guided propagation. Our propagation not only separates reflectance from illumination, but also decomposes the illumination into a sun, sky and indirect layer. This rich decomposition allows novel image manipulations as demonstrated by our results
NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination
We address the problem of recovering the shape and spatially-varying
reflectance of an object from multi-view images (and their camera poses) of an
object illuminated by one unknown lighting condition. This enables the
rendering of novel views of the object under arbitrary environment lighting and
editing of the object's material properties. The key to our approach, which we
call Neural Radiance Factorization (NeRFactor), is to distill the volumetric
geometry of a Neural Radiance Field (NeRF) [Mildenhall et al. 2020]
representation of the object into a surface representation and then jointly
refine the geometry while solving for the spatially-varying reflectance and
environment lighting. Specifically, NeRFactor recovers 3D neural fields of
surface normals, light visibility, albedo, and Bidirectional Reflectance
Distribution Functions (BRDFs) without any supervision, using only a
re-rendering loss, simple smoothness priors, and a data-driven BRDF prior
learned from real-world BRDF measurements. By explicitly modeling light
visibility, NeRFactor is able to separate shadows from albedo and synthesize
realistic soft or hard shadows under arbitrary lighting conditions. NeRFactor
is able to recover convincing 3D models for free-viewpoint relighting in this
challenging and underconstrained capture setup for both synthetic and real
scenes. Qualitative and quantitative experiments show that NeRFactor
outperforms classic and deep learning-based state of the art across various
tasks. Our videos, code, and data are available at
people.csail.mit.edu/xiuming/projects/nerfactor/.Comment: Camera-ready version for SIGGRAPH Asia 2021. Project Page:
https://people.csail.mit.edu/xiuming/projects/nerfactor
Rich intrinsic image decomposition of outdoor scenes from multiple views
Abstract—Intrinsic images aim at separating an image into its reflectance and illumination components to facilitate further analysis or manipulation. This separation is severely ill-posed and the most successful methods rely on user indications or precise geometry to resolve the ambiguities inherent to this problem. In this paper we propose a method to estimate intrinsic images from multiple views of an outdoor scene without the need for precise geometry and with a few manual steps to calibrate the input. We use multiview stereo to automatically reconstruct a 3D point cloud of the scene. Although this point cloud is sparse and incomplete, we show that it provides the necessary information to compute plausible sky and indirect illumination at each 3D point. We then introduce an optimization method to estimate sun visibility over the point cloud. This algorithm compensates for the lack of accurate geometry and allows the extraction of precise shadows in the final image. We finally propagate the information computed over the sparse point cloud to every pixel in the photograph using image-guided propagation. Our propagation not only separates reflectance from illumination, but also decomposes the illumination into a sun, sky and indirect layer. This rich decomposition allows novel image manipulations as demonstrated by our results. Index Terms—Intrinsic images, image-guided propagation, multi-view stereo, mean-shift algorithm