1,125 research outputs found
Plateau-reduced Differentiable Path Tracing
Current differentiable renderers provide light transport gradients with
respect to arbitrary scene parameters. However, the mere existence of these
gradients does not guarantee useful update steps in an optimization. Instead,
inverse rendering might not converge due to inherent plateaus, i.e., regions of
zero gradient, in the objective function. We propose to alleviate this by
convolving the high-dimensional rendering function that maps scene parameters
to images with an additional kernel that blurs the parameter space. We describe
two Monte Carlo estimators to compute plateau-free gradients efficiently, i.e.,
with low variance, and show that these translate into net-gains in optimization
error and runtime performance. Our approach is a straightforward extension to
both black-box and differentiable renderers and enables optimization of
problems with intricate light transport, such as caustics or global
illumination, that existing differentiable renderers do not converge on.Comment: Accepted to CVPR 2023. Project page and interactive demos at
https://mfischer-ucl.github.io/prdpt
NeFII: Inverse Rendering for Reflectance Decomposition with Near-Field Indirect Illumination
Inverse rendering methods aim to estimate geometry, materials and
illumination from multi-view RGB images. In order to achieve better
decomposition, recent approaches attempt to model indirect illuminations
reflected from different materials via Spherical Gaussians (SG), which,
however, tends to blur the high-frequency reflection details. In this paper, we
propose an end-to-end inverse rendering pipeline that decomposes materials and
illumination from multi-view images, while considering near-field indirect
illumination. In a nutshell, we introduce the Monte Carlo sampling based path
tracing and cache the indirect illumination as neural radiance, enabling a
physics-faithful and easy-to-optimize inverse rendering method. To enhance
efficiency and practicality, we leverage SG to represent the smooth environment
illuminations and apply importance sampling techniques. To supervise indirect
illuminations from unobserved directions, we develop a novel radiance
consistency constraint between implicit neural radiance and path tracing
results of unobserved rays along with the joint optimization of materials and
illuminations, thus significantly improving the decomposition performance.
Extensive experiments demonstrate that our method outperforms the
state-of-the-art on multiple synthetic and real datasets, especially in terms
of inter-reflection decomposition.Comment: Accepted in CVPR 202
Inverse Global Illumination using a Neural Radiometric Prior
Inverse rendering methods that account for global illumination are becoming
more popular, but current methods require evaluating and automatically
differentiating millions of path integrals by tracing multiple light bounces,
which remains expensive and prone to noise. Instead, this paper proposes a
radiometric prior as a simple alternative to building complete path integrals
in a traditional differentiable path tracer, while still correctly accounting
for global illumination. Inspired by the Neural Radiosity technique, we use a
neural network as a radiance function, and we introduce a prior consisting of
the norm of the residual of the rendering equation in the inverse rendering
loss. We train our radiance network and optimize scene parameters
simultaneously using a loss consisting of both a photometric term between
renderings and the multi-view input images, and our radiometric prior (the
residual term). This residual term enforces a physical constraint on the
optimization that ensures that the radiance field accounts for global
illumination. We compare our method to a vanilla differentiable path tracer,
and more advanced techniques such as Path Replay Backpropagation. Despite the
simplicity of our approach, we can recover scene parameters with comparable and
in some cases better quality, at considerably lower computation times.Comment: Homepage: https://inverse-neural-radiosity.github.i
Physically-Based Editing of Indoor Scene Lighting from a Single Image
We present a method to edit complex indoor lighting from a single image with
its predicted depth and light source segmentation masks. This is an extremely
challenging problem that requires modeling complex light transport, and
disentangling HDR lighting from material and geometry with only a partial LDR
observation of the scene. We tackle this problem using two novel components: 1)
a holistic scene reconstruction method that estimates scene reflectance and
parametric 3D lighting, and 2) a neural rendering framework that re-renders the
scene from our predictions. We use physically-based indoor light
representations that allow for intuitive editing, and infer both visible and
invisible light sources. Our neural rendering framework combines
physically-based direct illumination and shadow rendering with deep networks to
approximate global illumination. It can capture challenging lighting effects,
such as soft shadows, directional lighting, specular materials, and
interreflections. Previous single image inverse rendering methods usually
entangle scene lighting and geometry and only support applications like object
insertion. Instead, by combining parametric 3D lighting estimation with neural
scene rendering, we demonstrate the first automatic method to achieve full
scene relighting, including light source insertion, removal, and replacement,
from a single image. All source code and data will be publicly released
Differentiable Transient Rendering
Recent differentiable rendering techniques have become key tools to tackle many inverse problems in graphics and vision. Existing models, however, assume steady-state light transport, i.e., infinite speed of light. While this is a safe assumption for many applications, recent advances in ultrafast imaging leverage the wealth of information that can be extracted from the exact time of flight of light. In this context, physically-based transient rendering allows to efficiently simulate and analyze light transport considering that the speed of light is indeed finite. In this paper, we introduce a novel differentiable transient rendering framework, to help bring the potential of differentiable approaches into the transient regime. To differentiate the transient path integral we need to take into account that scattering events at path vertices are no longer independent; instead, tracking the time of flight of light requires treating such scattering events at path vertices jointly as a multidimensional, evolving manifold. We thus turn to the generalized transport theorem, and introduce a novel correlated importance term, which links the time-integrated contribution of a path to its light throughput, and allows us to handle discontinuities in the light and sensor functions. Last, we present results in several challenging scenarios where the time of flight of light plays an important role such as optimizing indices of refraction, non-line-of-sight tracking with nonplanar relay walls, and non-line-of-sight tracking around two corners
Efficient Multi-View Inverse Rendering Using a Hybrid Differentiable Rendering Method
Recovering the shape and appearance of real-world objects from natural 2D
images is a long-standing and challenging inverse rendering problem. In this
paper, we introduce a novel hybrid differentiable rendering method to
efficiently reconstruct the 3D geometry and reflectance of a scene from
multi-view images captured by conventional hand-held cameras. Our method
follows an analysis-by-synthesis approach and consists of two phases. In the
initialization phase, we use traditional SfM and MVS methods to reconstruct a
virtual scene roughly matching the real scene. Then in the optimization phase,
we adopt a hybrid approach to refine the geometry and reflectance, where the
geometry is first optimized using an approximate differentiable rendering
method, and the reflectance is optimized afterward using a physically-based
differentiable rendering method. Our hybrid approach combines the efficiency of
approximate methods with the high-quality results of physically-based methods.
Extensive experiments on synthetic and real data demonstrate that our method
can produce reconstructions with similar or higher quality than
state-of-the-art methods while being more efficient.Comment: IJCAI202
The Differentiable Lens: Compound Lens Search over Glass Surfaces and Materials for Object Detection
Most camera lens systems are designed in isolation, separately from
downstream computer vision methods. Recently, joint optimization approaches
that design lenses alongside other components of the image acquisition and
processing pipeline -- notably, downstream neural networks -- have achieved
improved imaging quality or better performance on vision tasks. However, these
existing methods optimize only a subset of lens parameters and cannot optimize
glass materials given their categorical nature. In this work, we develop a
differentiable spherical lens simulation model that accurately captures
geometrical aberrations. We propose an optimization strategy to address the
challenges of lens design -- notorious for non-convex loss function landscapes
and many manufacturing constraints -- that are exacerbated in joint
optimization tasks. Specifically, we introduce quantized continuous glass
variables to facilitate the optimization and selection of glass materials in an
end-to-end design context, and couple this with carefully designed constraints
to support manufacturability. In automotive object detection, we report
improved detection performance over existing designs even when simplifying
designs to two- or three-element lenses, despite significantly degrading the
image quality.Comment: 15 pages, 12 figures, to appear in CVPR 2023 proceedings, updated to
reflect camera-ready submissio
Importance Sampling BRDF Derivatives
We propose a set of techniques to efficiently importance sample the
derivatives of several BRDF models. In differentiable rendering, BRDFs are
replaced by their differential BRDF counterparts which are real-valued and can
have negative values. This leads to a new source of variance arising from their
change in sign. Real-valued functions cannot be perfectly importance sampled by
a positive-valued PDF and the direct application of BRDF sampling leads to high
variance. Previous attempts at antithetic sampling only addressed the
derivative with the roughness parameter of isotropic microfacet BRDFs. Our work
generalizes BRDF derivative sampling to anisotropic microfacet models, mixture
BRDFs, Oren-Nayar, Hanrahan-Krueger, among other analytic BRDFs.
Our method first decomposes the real-valued differential BRDF into a sum of
single-signed functions, eliminating variance from a change in sign. Next, we
importance sample each of the resulting single-signed functions separately. The
first decomposition, positivization, partitions the real-valued function based
on its sign, and is effective at variance reduction when applicable. However,
it requires analytic knowledge of the roots of the differential BRDF, and for
it to be analytically integrable too. Our key insight is that the single-signed
functions can have overlapping support, which significantly broadens the ways
we can decompose a real-valued function. Our product and mixture decompositions
exploit this property, and they allow us to support several BRDF derivatives
that positivization could not handle. For a wide variety of BRDF derivatives,
our method significantly reduces the variance (up to 58x in some cases) at
equal computation cost and enables better recovery of spatially varying
textures through gradient-descent-based inverse rendering
3D Object Positioning Using Differentiable Multimodal Learning
This article describes a multi-modal method using simulated Lidar data via
ray tracing and image pixel loss with differentiable rendering to optimize an
object's position with respect to an observer or some referential objects in a
computer graphics scene. Object position optimization is completed using
gradient descent with the loss function being influenced by both modalities.
Typical object placement optimization is done using image pixel loss with
differentiable rendering only, this work shows the use of a second modality
(Lidar) leads to faster convergence. This method of fusing sensor input
presents a potential usefulness for autonomous vehicles, as these methods can
be used to establish the locations of multiple actors in a scene. This article
also presents a method for the simulation of multiple types of data to be used
in the training of autonomous vehicles.Comment: 7 pages, 8 figure
- …