26 research outputs found
Burst Denoising with Kernel Prediction Networks
We present a technique for jointly denoising bursts of images taken from a
handheld camera. In particular, we propose a convolutional neural network
architecture for predicting spatially varying kernels that can both align and
denoise frames, a synthetic data generation approach based on a realistic noise
formation model, and an optimization guided by an annealed loss function to
avoid undesirable local minima. Our model matches or outperforms the
state-of-the-art across a wide range of noise levels on both real and synthetic
data.Comment: To appear in CVPR 2018 (spotlight). Project page:
http://people.eecs.berkeley.edu/~bmild/kpn
Zero-Shot Text-Guided Object Generation with Dream Fields
We combine neural rendering with multi-modal image and text representations
to synthesize diverse 3D objects solely from natural language descriptions. Our
method, Dream Fields, can generate the geometry and color of a wide range of
objects without 3D supervision. Due to the scarcity of diverse, captioned 3D
data, prior methods only generate objects from a handful of categories, such as
ShapeNet. Instead, we guide generation with image-text models pre-trained on
large datasets of captioned images from the web. Our method optimizes a Neural
Radiance Field from many camera views so that rendered images score highly with
a target caption according to a pre-trained CLIP model. To improve fidelity and
visual quality, we introduce simple geometric priors, including
sparsity-inducing transmittance regularization, scene bounds, and new MLP
architectures. In experiments, Dream Fields produce realistic, multi-view
consistent object geometry and color from a variety of natural language
captions.Comment: CVPR 2022. 13 pages. Website: https://ajayj.com/dreamfield
Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields
Neural Radiance Field training can be accelerated through the use of
grid-based representations in NeRF's learned mapping from spatial coordinates
to colors and volumetric density. However, these grid-based approaches lack an
explicit understanding of scale and therefore often introduce aliasing, usually
in the form of jaggies or missing scene content. Anti-aliasing has previously
been addressed by mip-NeRF 360, which reasons about sub-volumes along a cone
rather than points along a ray, but this approach is not natively compatible
with current grid-based techniques. We show how ideas from rendering and signal
processing can be used to construct a technique that combines mip-NeRF 360 and
grid-based models such as Instant NGP to yield error rates that are 8% - 76%
lower than either prior technique, and that trains 22x faster than mip-NeRF
360.Comment: Project page: https://jonbarron.info/zipnerf
CamP: Camera Preconditioning for Neural Radiance Fields
Neural Radiance Fields (NeRF) can be optimized to obtain high-fidelity 3D
scene reconstructions of objects and large-scale scenes. However, NeRFs require
accurate camera parameters as input -- inaccurate camera parameters result in
blurry renderings. Extrinsic and intrinsic camera parameters are usually
estimated using Structure-from-Motion (SfM) methods as a pre-processing step to
NeRF, but these techniques rarely yield perfect estimates. Thus, prior works
have proposed jointly optimizing camera parameters alongside a NeRF, but these
methods are prone to local minima in challenging settings. In this work, we
analyze how different camera parameterizations affect this joint optimization
problem, and observe that standard parameterizations exhibit large differences
in magnitude with respect to small perturbations, which can lead to an
ill-conditioned optimization problem. We propose using a proxy problem to
compute a whitening transform that eliminates the correlation between camera
parameters and normalizes their effects, and we propose to use this transform
as a preconditioner for the camera parameters during joint optimization. Our
preconditioned camera optimization significantly improves reconstruction
quality on scenes from the Mip-NeRF 360 dataset: we reduce error rates (RMSE)
by 67% compared to state-of-the-art NeRF approaches that do not optimize for
cameras like Zip-NeRF, and by 29% relative to state-of-the-art joint
optimization approaches using the camera parameterization of SCNeRF. Our
approach is easy to implement, does not significantly increase runtime, can be
applied to a wide variety of camera parameterizations, and can
straightforwardly be incorporated into other NeRF-like models.Comment: SIGGRAPH Asia 2023, Project page: https://camp-nerf.github.i
Eclipse: Disambiguating Illumination and Materials using Unintended Shadows
Decomposing an object's appearance into representations of its materials and
the surrounding illumination is difficult, even when the object's 3D shape is
known beforehand. This problem is ill-conditioned because diffuse materials
severely blur incoming light, and is ill-posed because diffuse materials under
high-frequency lighting can be indistinguishable from shiny materials under
low-frequency lighting. We show that it is possible to recover precise
materials and illumination -- even from diffuse objects -- by exploiting
unintended shadows, like the ones cast onto an object by the photographer who
moves around it. These shadows are a nuisance in most previous inverse
rendering pipelines, but here we exploit them as signals that improve
conditioning and help resolve material-lighting ambiguities. We present a
method based on differentiable Monte Carlo ray tracing that uses images of an
object to jointly recover its spatially-varying materials, the surrounding
illumination environment, and the shapes of the unseen light occluders who
inadvertently cast shadows upon it.Comment: Project page: https://dorverbin.github.io/eclipse