14,866 research outputs found
Instance Neural Radiance Field
This paper presents one of the first learning-based NeRF 3D instance
segmentation pipelines, dubbed as Instance Neural Radiance Field, or Instance
NeRF. Taking a NeRF pretrained from multi-view RGB images as input, Instance
NeRF can learn 3D instance segmentation of a given scene, represented as an
instance field component of the NeRF model. To this end, we adopt a 3D
proposal-based mask prediction network on the sampled volumetric features from
NeRF, which generates discrete 3D instance masks. The coarse 3D mask prediction
is then projected to image space to match 2D segmentation masks from different
views generated by existing panoptic segmentation models, which are used to
supervise the training of the instance field. Notably, beyond generating
consistent 2D segmentation maps from novel views, Instance NeRF can query
instance information at any 3D point, which greatly enhances NeRF object
segmentation and manipulation. Our method is also one of the first to achieve
such results without ground-truth instance information during inference.
Experimented on synthetic and real-world NeRF datasets with complex indoor
scenes, Instance NeRF surpasses previous NeRF segmentation works and
competitive 2D segmentation methods in segmentation performance on unseen
views. See the demo video at https://youtu.be/wW9Bme73coI
A LiDAR Point Cloud Generator: from a Virtual World to Autonomous Driving
3D LiDAR scanners are playing an increasingly important role in autonomous
driving as they can generate depth information of the environment. However,
creating large 3D LiDAR point cloud datasets with point-level labels requires a
significant amount of manual annotation. This jeopardizes the efficient
development of supervised deep learning algorithms which are often data-hungry.
We present a framework to rapidly create point clouds with accurate point-level
labels from a computer game. The framework supports data collection from both
auto-driving scenes and user-configured scenes. Point clouds from auto-driving
scenes can be used as training data for deep learning algorithms, while point
clouds from user-configured scenes can be used to systematically test the
vulnerability of a neural network, and use the falsifying examples to make the
neural network more robust through retraining. In addition, the scene images
can be captured simultaneously in order for sensor fusion tasks, with a method
proposed to do automatic calibration between the point clouds and captured
scene images. We show a significant improvement in accuracy (+9%) in point
cloud segmentation by augmenting the training dataset with the generated
synthesized data. Our experiments also show by testing and retraining the
network using point clouds from user-configured scenes, the weakness/blind
spots of the neural network can be fixed
RGB-D datasets using microsoft kinect or similar sensors: a survey
RGB-D data has turned out to be a very useful representation of an indoor scene for solving fundamental computer vision problems. It takes the advantages of the color image that provides appearance information of an object and also the depth image that is immune to the variations in color, illumination, rotation angle and scale. With the invention of the low-cost Microsoft Kinect sensor, which was initially used for gaming and later became a popular device for computer vision, high quality RGB-D data can be acquired easily. In recent years, more and more RGB-D image/video datasets dedicated to various applications have become available, which are of great importance to benchmark the state-of-the-art. In this paper, we systematically survey popular RGB-D datasets for different applications including object recognition, scene classification, hand gesture recognition, 3D-simultaneous localization and mapping, and pose estimation. We provide the insights into the characteristics of each important dataset, and compare the popularity and the difficulty of those datasets. Overall, the main goal of this survey is to give a comprehensive description about the available RGB-D datasets and thus to guide researchers in the selection of suitable datasets for evaluating their algorithms
- …