2,583 research outputs found
Navigation domain representation for interactive multiview imaging
Enabling users to interactively navigate through different viewpoints of a
static scene is a new interesting functionality in 3D streaming systems. While
it opens exciting perspectives towards rich multimedia applications, it
requires the design of novel representations and coding techniques in order to
solve the new challenges imposed by interactive navigation. Interactivity
clearly brings new design constraints: the encoder is unaware of the exact
decoding process, while the decoder has to reconstruct information from
incomplete subsets of data since the server can generally not transmit images
for all possible viewpoints due to resource constrains. In this paper, we
propose a novel multiview data representation that permits to satisfy bandwidth
and storage constraints in an interactive multiview streaming system. In
particular, we partition the multiview navigation domain into segments, each of
which is described by a reference image and some auxiliary information. The
auxiliary information enables the client to recreate any viewpoint in the
navigation segment via view synthesis. The decoder is then able to navigate
freely in the segment without further data request to the server; it requests
additional data only when it moves to a different segment. We discuss the
benefits of this novel representation in interactive navigation systems and
further propose a method to optimize the partitioning of the navigation domain
into independent segments, under bandwidth and storage constraints.
Experimental results confirm the potential of the proposed representation;
namely, our system leads to similar compression performance as classical
inter-view coding, while it provides the high level of flexibility that is
required for interactive streaming. Hence, our new framework represents a
promising solution for 3D data representation in novel interactive multimedia
services
SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fields
Neural Radiance Fields (NeRFs) have emerged as a popular approach for novel
view synthesis. While NeRFs are quickly being adapted for a wider set of
applications, intuitively editing NeRF scenes is still an open challenge. One
important editing task is the removal of unwanted objects from a 3D scene, such
that the replaced region is visually plausible and consistent with its context.
We refer to this task as 3D inpainting. In 3D, solutions must be both
consistent across multiple views and geometrically valid. In this paper, we
propose a novel 3D inpainting method that addresses these challenges. Given a
small set of posed images and sparse annotations in a single input image, our
framework first rapidly obtains a 3D segmentation mask for a target object.
Using the mask, a perceptual optimizationbased approach is then introduced that
leverages learned 2D image inpainters, distilling their information into 3D
space, while ensuring view consistency. We also address the lack of a diverse
benchmark for evaluating 3D scene inpainting methods by introducing a dataset
comprised of challenging real-world scenes. In particular, our dataset contains
views of the same scene with and without a target object, enabling more
principled benchmarking of the 3D inpainting task. We first demonstrate the
superiority of our approach on multiview segmentation, comparing to NeRFbased
methods and 2D segmentation approaches. We then evaluate on the task of 3D
inpainting, establishing state-ofthe-art performance against other NeRF
manipulation algorithms, as well as a strong 2D image inpainter baselineComment: Project Page: https://spinnerf3d.github.i
- …