247 research outputs found
What Is Around The Camera?
How much does a single image reveal about the environment it was taken in? In
this paper, we investigate how much of that information can be retrieved from a
foreground object, combined with the background (i.e. the visible part of the
environment). Assuming it is not perfectly diffuse, the foreground object acts
as a complexly shaped and far-from-perfect mirror. An additional challenge is
that its appearance confounds the light coming from the environment with the
unknown materials it is made of. We propose a learning-based approach to
predict the environment from multiple reflectance maps that are computed from
approximate surface normals. The proposed method allows us to jointly model the
statistics of environments and material properties. We train our system from
synthesized training data, but demonstrate its applicability to real-world
data. Interestingly, our analysis shows that the information obtained from
objects made out of multiple materials often is complementary and leads to
better performance.Comment: Accepted to ICCV. Project:
http://homes.esat.kuleuven.be/~sgeorgou/multinatillum
Intrinsic Harmonization for Illumination-Aware Compositing
Despite significant advancements in network-based image harmonization
techniques, there still exists a domain disparity between typical training
pairs and real-world composites encountered during inference. Most existing
methods are trained to reverse global edits made on segmented image regions,
which fail to accurately capture the lighting inconsistencies between the
foreground and background found in composited images. In this work, we
introduce a self-supervised illumination harmonization approach formulated in
the intrinsic image domain. First, we estimate a simple global lighting model
from mid-level vision representations to generate a rough shading for the
foreground region. A network then refines this inferred shading to generate a
harmonious re-shading that aligns with the background scene. In order to match
the color appearance of the foreground and background, we utilize ideas from
prior harmonization approaches to perform parameterized image edits in the
albedo domain. To validate the effectiveness of our approach, we present
results from challenging real-world composites and conduct a user study to
objectively measure the enhanced realism achieved compared to state-of-the-art
harmonization methods.Comment: 10 pages, 8 figures. Accepted to SIGGRAPH Asia 2023 (Conference
Track). Project page: https://yaksoy.github.io/intrinsicCompositing
OutCast: Outdoor Single-image Relighting with Cast Shadows
We propose a relighting method for outdoor images. Our method mainly focuses
on predicting cast shadows in arbitrary novel lighting directions from a single
image while also accounting for shading and global effects such the sun light
color and clouds. Previous solutions for this problem rely on reconstructing
occluder geometry, e.g. using multi-view stereo, which requires many images of
the scene. Instead, in this work we make use of a noisy off-the-shelf
single-image depth map estimation as a source of geometry. Whilst this can be a
good guide for some lighting effects, the resulting depth map quality is
insufficient for directly ray-tracing the shadows. Addressing this, we propose
a learned image space ray-marching layer that converts the approximate depth
map into a deep 3D representation that is fused into occlusion queries using a
learned traversal. Our proposed method achieves, for the first time,
state-of-the-art relighting results, with only a single image as input. For
supplementary material visit our project page at:
https://dgriffiths.uk/outcast.Comment: Eurographics 2022 - Accepte
Static scene illumination estimation from video with applications
We present a system that automatically recovers scene geometry and illumination from a video, providing a basis for various applications. Previous image based illumination estimation methods require either user interaction or external information in the form of a database. We adopt structure-from-motion and multi-view stereo for initial scene reconstruction, and then estimate an environment map represented by spherical harmonics (as these perform better than other bases). We also demonstrate several video editing applications that exploit the recovered geometry and illumination, including object insertion (e.g., for augmented reality), shadow detection, and video relighting
NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images
We present a neural rendering-based method called NeRO for reconstructing the
geometry and the BRDF of reflective objects from multiview images captured in
an unknown environment. Multiview reconstruction of reflective objects is
extremely challenging because specular reflections are view-dependent and thus
violate the multiview consistency, which is the cornerstone for most multiview
reconstruction methods. Recent neural rendering techniques can model the
interaction between environment lights and the object surfaces to fit the
view-dependent reflections, thus making it possible to reconstruct reflective
objects from multiview images. However, accurately modeling environment lights
in the neural rendering is intractable, especially when the geometry is
unknown. Most existing neural rendering methods, which can model environment
lights, only consider direct lights and rely on object masks to reconstruct
objects with weak specular reflections. Therefore, these methods fail to
reconstruct reflective objects, especially when the object mask is not
available and the object is illuminated by indirect lights. We propose a
two-step approach to tackle this problem. First, by applying the split-sum
approximation and the integrated directional encoding to approximate the
shading effects of both direct and indirect lights, we are able to accurately
reconstruct the geometry of reflective objects without any object masks. Then,
with the object geometry fixed, we use more accurate sampling to recover the
environment lights and the BRDF of the object. Extensive experiments
demonstrate that our method is capable of accurately reconstructing the
geometry and the BRDF of reflective objects from only posed RGB images without
knowing the environment lights and the object masks. Codes and datasets are
available at https://github.com/liuyuan-pal/NeRO.Comment: Accepted to SIGGRAPH 2023. Project page:
https://liuyuan-pal.github.io/NeRO/ Codes:
https://github.com/liuyuan-pal/NeR
NeuS-PIR: Learning Relightable Neural Surface using Pre-Integrated Rendering
Recent advances in neural implicit fields enables rapidly reconstructing 3D
geometry from multi-view images. Beyond that, recovering physical properties
such as material and illumination is essential for enabling more applications.
This paper presents a new method that effectively learns relightable neural
surface using pre-intergrated rendering, which simultaneously learns geometry,
material and illumination within the neural implicit field. The key insight of
our work is that these properties are closely related to each other, and
optimizing them in a collaborative manner would lead to consistent
improvements. Specifically, we propose NeuS-PIR, a method that factorizes the
radiance field into a spatially varying material field and a differentiable
environment cubemap, and jointly learns it with geometry represented by neural
surface. Our experiments demonstrate that the proposed method outperforms the
state-of-the-art method in both synthetic and real datasets
Scene relighting and editing for improved object insertion
Abstract. The goal of this thesis is to develop a scene relighting and object insertion pipeline using Neural Radiance Fields (NeRF) to incorporate one or more objects into an outdoor environment scene. The output is a 3D mesh that embodies decomposed bidirectional reflectance distribution function (BRDF) characteristics, which interact with varying light source positions and strengths. To achieve this objective, the thesis is divided into two sub-tasks.
The first sub-task involves extracting visual information about the outdoor environment from a sparse set of corresponding images. A neural representation is constructed, providing a comprehensive understanding of the constituent elements, such as materials, geometry, illumination, and shadows. The second sub-task involves generating a neural representation of the inserted object using either real-world images or synthetic data.
To accomplish these objectives, the thesis draws on existing literature in computer vision and computer graphics. Different approaches are assessed to identify their advantages and disadvantages, with detailed descriptions of the chosen techniques provided, highlighting their functioning to produce the ultimate outcome.
Overall, this thesis aims to provide a framework for compositing and relighting that is grounded in NeRF and allows for the seamless integration of objects into outdoor environments. The outcome of this work has potential applications in various domains, such as visual effects, gaming, and virtual reality
- …