4 research outputs found
SANeRF-HQ: Segment Anything for NeRF in High Quality
Recently, the Segment Anything Model (SAM) has showcased remarkable
capabilities of zero-shot segmentation, while NeRF (Neural Radiance Fields) has
gained popularity as a method for various 3D problems beyond novel view
synthesis. Though there exist initial attempts to incorporate these two methods
into 3D segmentation, they face the challenge of accurately and consistently
segmenting objects in complex scenarios. In this paper, we introduce the
Segment Anything for NeRF in High Quality (SANeRF-HQ) to achieve high-quality
3D segmentation of any target object in a given scene. SANeRF-HQ utilizes SAM
for open-world object segmentation guided by user-supplied prompts, while
leveraging NeRF to aggregate information from different viewpoints. To overcome
the aforementioned challenges, we employ density field and RGB similarity to
enhance the accuracy of segmentation boundary during the aggregation.
Emphasizing on segmentation accuracy, we evaluate our method on multiple NeRF
datasets where high-quality ground-truths are available or manually annotated.
SANeRF-HQ shows a significant quality improvement over state-of-the-art methods
in NeRF object segmentation, provides higher flexibility for object
localization, and enables more consistent object segmentation across multiple
views. Results and code are available at the project site:
https://lyclyc52.github.io/SANeRF-HQ/.Comment: Accepted to CVPR 202
NeRF-RPN: A general framework for object detection in NeRFs
This paper presents the first significant object detection framework,
NeRF-RPN, which directly operates on NeRF. Given a pre-trained NeRF model,
NeRF-RPN aims to detect all bounding boxes of objects in a scene. By exploiting
a novel voxel representation that incorporates multi-scale 3D neural volumetric
features, we demonstrate it is possible to regress the 3D bounding boxes of
objects in NeRF directly without rendering the NeRF at any viewpoint. NeRF-RPN
is a general framework and can be applied to detect objects without class
labels. We experimented the NeRF-RPN with various backbone architectures, RPN
head designs and loss functions. All of them can be trained in an end-to-end
manner to estimate high quality 3D bounding boxes. To facilitate future
research in object detection for NeRF, we built a new benchmark dataset which
consists of both synthetic and real-world data with careful labeling and clean
up. Please click https://youtu.be/M8_4Ih1CJjE for visualizing the 3D region
proposals by our NeRF-RPN. Code and dataset will be made available
Instance Neural Radiance Field
This paper presents one of the first learning-based NeRF 3D instance
segmentation pipelines, dubbed as Instance Neural Radiance Field, or Instance
NeRF. Taking a NeRF pretrained from multi-view RGB images as input, Instance
NeRF can learn 3D instance segmentation of a given scene, represented as an
instance field component of the NeRF model. To this end, we adopt a 3D
proposal-based mask prediction network on the sampled volumetric features from
NeRF, which generates discrete 3D instance masks. The coarse 3D mask prediction
is then projected to image space to match 2D segmentation masks from different
views generated by existing panoptic segmentation models, which are used to
supervise the training of the instance field. Notably, beyond generating
consistent 2D segmentation maps from novel views, Instance NeRF can query
instance information at any 3D point, which greatly enhances NeRF object
segmentation and manipulation. Our method is also one of the first to achieve
such results without ground-truth instance information during inference.
Experimented on synthetic and real-world NeRF datasets with complex indoor
scenes, Instance NeRF surpasses previous NeRF segmentation works and
competitive 2D segmentation methods in segmentation performance on unseen
views. See the demo video at https://youtu.be/wW9Bme73coI