319 research outputs found
Neural Free-Viewpoint Relighting for Glossy Indirect Illumination
Precomputed Radiance Transfer (PRT) remains an attractive solution for
real-time rendering of complex light transport effects such as glossy global
illumination. After precomputation, we can relight the scene with new
environment maps while changing viewpoint in real-time. However, practical PRT
methods are usually limited to low-frequency spherical harmonic lighting.
All-frequency techniques using wavelets are promising but have so far had
little practical impact. The curse of dimensionality and much higher data
requirements have typically limited them to relighting with fixed view or only
direct lighting with triple product integrals. In this paper, we demonstrate a
hybrid neural-wavelet PRT solution to high-frequency indirect illumination,
including glossy reflection, for relighting with changing view. Specifically,
we seek to represent the light transport function in the Haar wavelet basis.
For global illumination, we learn the wavelet transport using a small
multi-layer perceptron (MLP) applied to a feature field as a function of
spatial location and wavelet index, with reflected direction and material
parameters being other MLP inputs. We optimize/learn the feature field
(compactly represented by a tensor decomposition) and MLP parameters from
multiple images of the scene under different lighting and viewing conditions.
We demonstrate real-time (512 x 512 at 24 FPS, 800 x 600 at 13 FPS) precomputed
rendering of challenging scenes involving view-dependent reflections and even
caustics.Comment: 13 pages, 9 figures, to appear in cgf proceedings of egsr 202
Learning Object-Centric Neural Scattering Functions for Free-viewpoint Relighting and Scene Composition
Photorealistic object appearance modeling from 2D images is a constant topic
in vision and graphics. While neural implicit methods (such as Neural Radiance
Fields) have shown high-fidelity view synthesis results, they cannot relight
the captured objects. More recent neural inverse rendering approaches have
enabled object relighting, but they represent surface properties as simple
BRDFs, and therefore cannot handle translucent objects. We propose
Object-Centric Neural Scattering Functions (OSFs) for learning to reconstruct
object appearance from only images. OSFs not only support free-viewpoint object
relighting, but also can model both opaque and translucent objects. While
accurately modeling subsurface light transport for translucent objects can be
highly complex and even intractable for neural methods, OSFs learn to
approximate the radiance transfer from a distant light to an outgoing direction
at any spatial location. This approximation avoids explicitly modeling complex
subsurface scattering, making learning a neural implicit model tractable.
Experiments on real and synthetic data show that OSFs accurately reconstruct
appearances for both opaque and translucent objects, allowing faithful
free-viewpoint relighting as well as scene composition. Project website:
https://kovenyu.com/osf/Comment: Project website: https://kovenyu.com/osf/ Journal extension of
arXiv:2012.08503. The first two authors contributed equally to this wor
On-the-Go Reflectance Transformation Imaging with Ordinary Smartphones
Reflectance Transformation Imaging (RTI) is a popular technique that allows the recovery of per-pixel reflectance information by capturing an object under different light conditions. This can be later used to reveal surface details and interactively relight the subject. Such process, however, typically requires dedicated hardware setups to recover the light direction from multiple locations, making the process tedious when performed outside the lab.
We propose a novel RTI method that can be carried out by recording videos with two ordinary smartphones. The flash led-light of one device is used to illuminate the subject while the other captures the reflectance. Since the led is mounted close to the camera lenses, we can infer the light direction for thousands of images by freely moving the illuminating device while observing a fiducial marker surrounding the subject. To deal with such amount of data, we propose a neural relighting model that reconstructs object appearance for arbitrary light directions from extremely compact reflectance distribution data compressed via Principal Components Analysis (PCA). Experiments shows that the proposed technique can be easily performed on the field with a resulting RTI model that can outperform state-of-the-art approaches involving dedicated hardware setups
A Dataset of Multi-Illumination Images in the Wild
Collections of images under a single, uncontrolled illumination have enabled
the rapid advancement of core computer vision tasks like classification,
detection, and segmentation. But even with modern learning techniques, many
inverse problems involving lighting and material understanding remain too
severely ill-posed to be solved with single-illumination datasets. To fill this
gap, we introduce a new multi-illumination dataset of more than 1000 real
scenes, each captured under 25 lighting conditions. We demonstrate the richness
of this dataset by training state-of-the-art models for three challenging
applications: single-image illumination estimation, image relighting, and
mixed-illuminant white balance.Comment: ICCV 201
NeFII: Inverse Rendering for Reflectance Decomposition with Near-Field Indirect Illumination
Inverse rendering methods aim to estimate geometry, materials and
illumination from multi-view RGB images. In order to achieve better
decomposition, recent approaches attempt to model indirect illuminations
reflected from different materials via Spherical Gaussians (SG), which,
however, tends to blur the high-frequency reflection details. In this paper, we
propose an end-to-end inverse rendering pipeline that decomposes materials and
illumination from multi-view images, while considering near-field indirect
illumination. In a nutshell, we introduce the Monte Carlo sampling based path
tracing and cache the indirect illumination as neural radiance, enabling a
physics-faithful and easy-to-optimize inverse rendering method. To enhance
efficiency and practicality, we leverage SG to represent the smooth environment
illuminations and apply importance sampling techniques. To supervise indirect
illuminations from unobserved directions, we develop a novel radiance
consistency constraint between implicit neural radiance and path tracing
results of unobserved rays along with the joint optimization of materials and
illuminations, thus significantly improving the decomposition performance.
Extensive experiments demonstrate that our method outperforms the
state-of-the-art on multiple synthetic and real datasets, especially in terms
of inter-reflection decomposition.Comment: Accepted in CVPR 202
BiDi screen: a thin, depth-sensing LCD for 3D interaction using light fields
We transform an LCD into a display that supports both 2D multi-touch and unencumbered 3D gestures. Our BiDirectional (BiDi) screen, capable of both image capture and display, is inspired by emerging LCDs that use embedded optical sensors to detect multiple points of contact. Our key contribution is to exploit the spatial light modulation capability of LCDs to allow lensless imaging without interfering with display functionality. We switch between a display mode showing traditional graphics and a capture mode in which the backlight is disabled and the LCD displays a pinhole array or an equivalent tiled-broadband code. A large-format image sensor is placed slightly behind the liquid crystal layer. Together, the image sensor and LCD form a mask-based light field camera, capturing an array of images equivalent to that produced by a camera array spanning the display surface. The recovered multi-view orthographic imagery is used to passively estimate the depth of scene points. Two motivating applications are described: a hybrid touch plus gesture interaction and a light-gun mode for interacting with external light-emitting widgets. We show a working prototype that simulates the image sensor with a camera and diffuser, allowing interaction up to 50 cm in front of a modified 20.1 inch LCD.National Science Foundation (U.S.) (Grant CCF-0729126)Alfred P. Sloan Foundatio
Real-time Cinematic Design Of Visual Aspects In Computer-generated Images
Creation of visually-pleasing images has always been one of the main goals of computer graphics. Two important components are necessary to achieve this goal --- artists who design visual aspects of an image (such as materials or lighting) and sophisticated algorithms that render the image. Traditionally, rendering has been of greater interest to researchers, while the design part has always been deemed as secondary. This has led to many inefficiencies, as artists, in order to create a stunning image, are often forced to resort to the traditional, creativity-baring, pipelines consisting of repeated rendering and parameter tweaking. Our work shifts the attention away from the rendering problem and focuses on the design. We propose to combine non-physical editing with real-time feedback and provide artists with efficient ways of designing complex visual aspects such as global illumination or all-frequency shadows. We conform to existing pipelines by inserting our editing components into existing stages, hereby making editing of visual aspects an inherent part of the design process. Many of the examples showed in this work have been, until now, extremely hard to achieve. The non-physical aspect of our work enables artists to express themselves in more creative ways, not limited by the physical parameters of current renderers. Real-time feedback allows artists to immediately see the effects of applied modifications and compatibility with existing workflows enables easy integration of our algorithms into production pipelines
- …