522 research outputs found
NeFII: Inverse Rendering for Reflectance Decomposition with Near-Field Indirect Illumination
Inverse rendering methods aim to estimate geometry, materials and
illumination from multi-view RGB images. In order to achieve better
decomposition, recent approaches attempt to model indirect illuminations
reflected from different materials via Spherical Gaussians (SG), which,
however, tends to blur the high-frequency reflection details. In this paper, we
propose an end-to-end inverse rendering pipeline that decomposes materials and
illumination from multi-view images, while considering near-field indirect
illumination. In a nutshell, we introduce the Monte Carlo sampling based path
tracing and cache the indirect illumination as neural radiance, enabling a
physics-faithful and easy-to-optimize inverse rendering method. To enhance
efficiency and practicality, we leverage SG to represent the smooth environment
illuminations and apply importance sampling techniques. To supervise indirect
illuminations from unobserved directions, we develop a novel radiance
consistency constraint between implicit neural radiance and path tracing
results of unobserved rays along with the joint optimization of materials and
illuminations, thus significantly improving the decomposition performance.
Extensive experiments demonstrate that our method outperforms the
state-of-the-art on multiple synthetic and real datasets, especially in terms
of inter-reflection decomposition.Comment: Accepted in CVPR 202
Differentiable Display Photometric Stereo
Photometric stereo leverages variations in illumination conditions to
reconstruct per-pixel surface normals. The concept of display photometric
stereo, which employs a conventional monitor as an illumination source, has the
potential to overcome limitations often encountered in bulky and
difficult-to-use conventional setups. In this paper, we introduce
Differentiable Display Photometric Stereo (DDPS), a method designed to achieve
high-fidelity normal reconstruction using an off-the-shelf monitor and camera.
DDPS addresses a critical yet often neglected challenge in photometric stereo:
the optimization of display patterns for enhanced normal reconstruction. We
present a differentiable framework that couples basis-illumination image
formation with a photometric-stereo reconstruction method. This facilitates the
learning of display patterns that leads to high-quality normal reconstruction
through automatic differentiation. Addressing the synthetic-real domain gap
inherent in end-to-end optimization, we propose the use of a real-world
photometric-stereo training dataset composed of 3D-printed objects. Moreover,
to reduce the ill-posed nature of photometric stereo, we exploit the linearly
polarized light emitted from the monitor to optically separate diffuse and
specular reflections in the captured images. We demonstrate that DDPS allows
for learning display patterns optimized for a target configuration and is
robust to initialization. We assess DDPS on 3D-printed objects with
ground-truth normals and diverse real-world objects, validating that DDPS
enables effective photometric-stereo reconstruction
{3D} Morphable Face Models -- Past, Present and Future
In this paper, we provide a detailed survey of 3D Morphable Face Models over the 20 years since they were first proposed. The challenges in building and applying these models, namely capture, modeling, image formation, and image analysis, are still active research topics, and we review the state-of-the-art in each of these areas. We also look ahead, identifying unsolved challenges, proposing directions for future research and highlighting the broad range of current and future applications
Multi-view Inverse Rendering for Large-scale Real-world Indoor Scenes
We present a multi-view inverse rendering method for large-scale real-world
indoor scenes that reconstructs global illumination and physically-reasonable
SVBRDFs. Unlike previous representations, where the global illumination of
large scenes is simplified as multiple environment maps, we propose a compact
representation called Texture-based Lighting (TBL). It consists of 3D meshs and
HDR textures, and efficiently models direct and infinite-bounce indirect
lighting of the entire large scene. Based on TBL, we further propose a hybrid
lighting representation with precomputed irradiance, which significantly
improves the efficiency and alleviate the rendering noise in the material
optimization. To physically disentangle the ambiguity between materials, we
propose a three-stage material optimization strategy based on the priors of
semantic segmentation and room segmentation. Extensive experiments show that
the proposed method outperforms the state-of-the-arts quantitatively and
qualitatively, and enables physically-reasonable mixed-reality applications
such as material editing, editable novel view synthesis and relighting. The
project page is at https://lzleejean.github.io/TexIR.Comment: The project page is at: https://lzleejean.github.io/TexI
Learning to Learn and Sample BRDFs
We propose a method to accelerate the joint process of physically acquiring
and learning neural Bi-directional Reflectance Distribution Function (BRDF)
models. While BRDF learning alone can be accelerated by meta-learning,
acquisition remains slow as it relies on a mechanical process. We show that
meta-learning can be extended to optimize the physical sampling pattern, too.
After our method has been meta-trained for a set of fully-sampled BRDFs, it is
able to quickly train on new BRDFs with up to five orders of magnitude fewer
physical acquisition samples at similar quality. Our approach also extends to
other linear and non-linear BRDF models, which we show in an extensive
evaluation
Estimating Neural Reflectance Field from Radiance Field using Tree Structures
We present a new method for estimating the Neural Reflectance Field (NReF) of
an object from a set of posed multi-view images under unknown lighting. NReF
represents 3D geometry and appearance of objects in a disentangled manner, and
are hard to be estimated from images only. Our method solves this problem by
exploiting the Neural Radiance Field (NeRF) as a proxy representation, from
which we perform further decomposition. A high-quality NeRF decomposition
relies on good geometry information extraction as well as good prior terms to
properly resolve ambiguities between different components. To extract
high-quality geometry information from radiance fields, we re-design a new
ray-casting based method for surface point extraction. To efficiently compute
and apply prior terms, we convert different prior terms into different type of
filter operations on the surface extracted from radiance field. We then employ
two type of auxiliary data structures, namely Gaussian KD-tree and octree, to
support fast querying of surface points and efficient computation of surface
filters during training. Based on this, we design a multi-stage decomposition
optimization pipeline for estimating neural reflectance field from neural
radiance fields. Extensive experiments show our method outperforms other
state-of-the-art methods on different data, and enable high-quality free-view
relighting as well as material editing tasks
Physics vs. Learned Priors: Rethinking Camera and Algorithm Design for Task-Specific Imaging
Cameras were originally designed using physics-based heuristics to capture
aesthetic images. In recent years, there has been a transformation in camera
design from being purely physics-driven to increasingly data-driven and
task-specific. In this paper, we present a framework to understand the building
blocks of this nascent field of end-to-end design of camera hardware and
algorithms. As part of this framework, we show how methods that exploit both
physics and data have become prevalent in imaging and computer vision,
underscoring a key trend that will continue to dominate the future of
task-specific camera design. Finally, we share current barriers to progress in
end-to-end design, and hypothesize how these barriers can be overcome
- …