4 research outputs found
ViP-NeRF: Visibility Prior for Sparse Input Neural Radiance Fields
Neural radiance fields (NeRF) have achieved impressive performances in view
synthesis by encoding neural representations of a scene. However, NeRFs require
hundreds of images per scene to synthesize photo-realistic novel views.
Training them on sparse input views leads to overfitting and incorrect scene
depth estimation resulting in artifacts in the rendered novel views. Sparse
input NeRFs were recently regularized by providing dense depth estimated from
pre-trained networks as supervision, to achieve improved performance over
sparse depth constraints. However, we find that such depth priors may be
inaccurate due to generalization issues. Instead, we hypothesize that the
visibility of pixels in different input views can be more reliably estimated to
provide dense supervision. In this regard, we compute a visibility prior
through the use of plane sweep volumes, which does not require any
pre-training. By regularizing the NeRF training with the visibility prior, we
successfully train the NeRF with few input views. We reformulate the NeRF to
also directly output the visibility of a 3D point from a given viewpoint to
reduce the training time with the visibility constraint. On multiple datasets,
our model outperforms the competing sparse input NeRF models including those
that use learned priors. The source code for our model can be found on our
project page:
https://nagabhushansn95.github.io/publications/2023/ViP-NeRF.html.Comment: SIGGRAPH 202
SimpleNeRF: Regularizing Sparse Input Neural Radiance Fields with Simpler Solutions
Neural Radiance Fields (NeRF) show impressive performance for the
photorealistic free-view rendering of scenes. However, NeRFs require dense
sampling of images in the given scene, and their performance degrades
significantly when only a sparse set of views are available. Researchers have
found that supervising the depth estimated by the NeRF helps train it
effectively with fewer views. The depth supervision is obtained either using
classical approaches or neural networks pre-trained on a large dataset. While
the former may provide only sparse supervision, the latter may suffer from
generalization issues. As opposed to the earlier approaches, we seek to learn
the depth supervision by designing augmented models and training them along
with the NeRF. We design augmented models that encourage simpler solutions by
exploring the role of positional encoding and view-dependent radiance in
training the few-shot NeRF. The depth estimated by these simpler models is used
to supervise the NeRF depth estimates. Since the augmented models can be
inaccurate in certain regions, we design a mechanism to choose only reliable
depth estimates for supervision. Finally, we add a consistency loss between the
coarse and fine multi-layer perceptrons of the NeRF to ensure better
utilization of hierarchical sampling. We achieve state-of-the-art
view-synthesis performance on two popular datasets by employing the above
regularizations. The source code for our model can be found on our project
page: https://nagabhushansn95.github.io/publications/2023/SimpleNeRF.htmlComment: SIGGRAPH Asia 202
Temporal View Synthesis of Dynamic Scenes through 3D Object Motion Estimation with Multi-Plane Images
The challenge of graphically rendering high frame-rate videos on low compute
devices can be addressed through periodic prediction of future frames to
enhance the user experience in virtual reality applications. This is studied
through the problem of temporal view synthesis (TVS), where the goal is to
predict the next frames of a video given the previous frames and the head poses
of the previous and the next frames. In this work, we consider the TVS of
dynamic scenes in which both the user and objects are moving. We design a
framework that decouples the motion into user and object motion to effectively
use the available user motion while predicting the next frames. We predict the
motion of objects by isolating and estimating the 3D object motion in the past
frames and then extrapolating it. We employ multi-plane images (MPI) as a 3D
representation of the scenes and model the object motion as the 3D displacement
between the corresponding points in the MPI representation. In order to handle
the sparsity in MPIs while estimating the motion, we incorporate partial
convolutions and masked correlation layers to estimate corresponding points.
The predicted object motion is then integrated with the given user or camera
motion to generate the next frame. Using a disocclusion infilling module, we
synthesize the regions uncovered due to the camera and object motion. We
develop a new synthetic dataset for TVS of dynamic scenes consisting of 800
videos at full HD resolution. We show through experiments on our dataset and
the MPI Sintel dataset that our model outperforms all the competing methods in
the literature.Comment: To appear in ISMAR 2022; Project website:
https://nagabhushansn95.github.io/publications/2022/DeCOMPnet.htm
Understanding the Perceived Quality of Video Predictions
The study of video prediction models is believed to be a fundamental approach
to representation learning for videos. While a plethora of generative models
for predicting the future frame pixel values given the past few frames exist,
the quantitative evaluation of the predicted frames has been found to be
extremely challenging. In this context, we study the problem of quality
assessment of predicted videos. We create the Indian Institute of Science
Predicted Videos Quality Assessment (IISc PVQA) Database consisting of 300
videos, obtained by applying different prediction models on different datasets,
and accompanying human opinion scores. We collected subjective ratings of
quality from 50 human participants for these videos. Our subjective study
reveals that human observers were highly consistent in their judgments of
quality of predicted videos. We benchmark several popularly used measures for
evaluating video prediction and show that they do not adequately correlate with
these subjective scores. We introduce two new features to effectively capture
the quality of predicted videos, motion-compensated cosine similarities of deep
features of predicted frames with past frames, and deep features extracted from
rescaled frame differences. We show that our feature design leads to state of
the art quality prediction in accordance with human judgments on our IISc PVQA
Database. The database and code are publicly available on our project website:
https://nagabhushansn95.github.io/publications/2020/pvqaComment: Project website:
https://nagabhushansn95.github.io/publications/2020/pvqa.htm