22 research outputs found
NeRFVS: Neural Radiance Fields for Free View Synthesis via Geometry Scaffolds
We present NeRFVS, a novel neural radiance fields (NeRF) based method to
enable free navigation in a room. NeRF achieves impressive performance in
rendering images for novel views similar to the input views while suffering for
novel views that are significantly different from the training views. To
address this issue, we utilize the holistic priors, including pseudo depth maps
and view coverage information, from neural reconstruction to guide the learning
of implicit neural representations of 3D indoor scenes. Concretely, an
off-the-shelf neural reconstruction method is leveraged to generate a geometry
scaffold. Then, two loss functions based on the holistic priors are proposed to
improve the learning of NeRF: 1) A robust depth loss that can tolerate the
error of the pseudo depth map to guide the geometry learning of NeRF; 2) A
variance loss to regularize the variance of implicit neural representations to
reduce the geometry and color ambiguity in the learning procedure. These two
loss functions are modulated during NeRF optimization according to the view
coverage information to reduce the negative influence brought by the view
coverage imbalance. Extensive results demonstrate that our NeRFVS outperforms
state-of-the-art view synthesis methods quantitatively and qualitatively on
indoor scenes, achieving high-fidelity free navigation results.Comment: 10 pages, 7 figure
Exploring Effective Mask Sampling Modeling for Neural Image Compression
Image compression aims to reduce the information redundancy in images. Most
existing neural image compression methods rely on side information from
hyperprior or context models to eliminate spatial redundancy, but rarely
address the channel redundancy. Inspired by the mask sampling modeling in
recent self-supervised learning methods for natural language processing and
high-level vision, we propose a novel pretraining strategy for neural image
compression. Specifically, Cube Mask Sampling Module (CMSM) is proposed to
apply both spatial and channel mask sampling modeling to image compression in
the pre-training stage. Moreover, to further reduce channel redundancy, we
propose the Learnable Channel Mask Module (LCMM) and the Learnable Channel
Completion Module (LCCM). Our plug-and-play CMSM, LCMM, LCCM modules can apply
to both CNN-based and Transformer-based architectures, significantly reduce the
computational cost, and improve the quality of images. Experiments on the
public Kodak and Tecnick datasets demonstrate that our method achieves
competitive performance with lower computational complexity compared to
state-of-the-art image compression methods.Comment: 10 page