50 research outputs found
Text to 3D Scene Generation with Rich Lexical Grounding
The ability to map descriptions of scenes to 3D geometric representations has
many applications in areas such as art, education, and robotics. However, prior
work on the text to 3D scene generation task has used manually specified object
categories and language that identifies them. We introduce a dataset of 3D
scenes annotated with natural language descriptions and learn from this data
how to ground textual descriptions to physical objects. Our method successfully
grounds a variety of lexical terms to concrete referents, and we show
quantitatively that our method improves 3D scene generation over previous work
using purely rule-based methods. We evaluate the fidelity and plausibility of
3D scenes generated with our grounding approach through human judgments. To
ease evaluation on this task, we also introduce an automated metric that
strongly correlates with human judgments.Comment: 10 pages, 7 figures, 3 tables. To appear in ACL-IJCNLP 201
Im2Pano3D: Extrapolating 360 Structure and Semantics Beyond the Field of View
We present Im2Pano3D, a convolutional neural network that generates a dense
prediction of 3D structure and a probability distribution of semantic labels
for a full 360 panoramic view of an indoor scene when given only a partial
observation (<= 50%) in the form of an RGB-D image. To make this possible,
Im2Pano3D leverages strong contextual priors learned from large-scale synthetic
and real-world indoor scenes. To ease the prediction of 3D structure, we
propose to parameterize 3D surfaces with their plane equations and train the
model to predict these parameters directly. To provide meaningful training
supervision, we use multiple loss functions that consider both pixel level
accuracy and global context consistency. Experiments demon- strate that
Im2Pano3D is able to predict the semantics and 3D structure of the unobserved
scene with more than 56% pixel accuracy and less than 0.52m average distance
error, which is significantly better than alternative approaches.Comment: Video summary: https://youtu.be/Au3GmktK-S
PARIS: Part-level Reconstruction and Motion Analysis for Articulated Objects
We address the task of simultaneous part-level reconstruction and motion
parameter estimation for articulated objects. Given two sets of multi-view
images of an object in two static articulation states, we decouple the movable
part from the static part and reconstruct shape and appearance while predicting
the motion parameters. To tackle this problem, we present PARIS: a
self-supervised, end-to-end architecture that learns part-level implicit shape
and appearance models and optimizes motion parameters jointly without any 3D
supervision, motion, or semantic annotation. Our experiments show that our
method generalizes better across object categories, and outperforms baselines
and prior work that are given 3D point clouds as input. Our approach improves
reconstruction relative to state-of-the-art baselines with a Chamfer-L1
distance reduction of 3.94 (45.2%) for objects and 26.79 (84.5%) for parts, and
achieves 5% error rate for motion estimation across 10 object categories.
Video summary at: https://youtu.be/tDSrROPCgUcComment: Presented at ICCV 2023. Project website:
https://3dlg-hcvc.github.io/paris
LeTFuser: Light-weight End-to-end Transformer-Based Sensor Fusion for Autonomous Driving with Multi-Task Learning
In end-to-end autonomous driving, the utilization of existing sensor fusion
techniques and navigational control methods for imitation learning proves
inadequate in challenging situations that involve numerous dynamic agents. To
address this issue, we introduce LeTFuser, a lightweight transformer-based
algorithm for fusing multiple RGB-D camera representations. To perform
perception and control tasks simultaneously, we utilize multi-task learning.
Our model comprises of two modules, the first being the perception module that
is responsible for encoding the observation data obtained from the RGB-D
cameras. Our approach employs the Convolutional vision Transformer (CvT)
\cite{wu2021cvt} to better extract and fuse features from multiple RGB cameras
due to local and global feature extraction capability of convolution and
transformer modules, respectively. Encoded features combined with static and
dynamic environments are later employed by our control module to predict
waypoints and vehicular controls (e.g. steering, throttle, and brake). We use
two methods to generate the vehicular controls levels. The first method uses a
PID algorithm to follow the waypoints on the fly, whereas the second one
directly predicts the control policy using the measurement features and
environmental state. We evaluate the model and conduct a comparative analysis
with recent models on the CARLA simulator using various scenarios, ranging from
normal to adversarial conditions, to simulate real-world scenarios. Our method
demonstrated better or comparable results with respect to our baselines in term
of driving abilities. The code is available at
\url{https://github.com/pagand/e2etransfuser/tree/cvpr-w} to facilitate future
studies.Comment: 11 pages, 2 figures, 3 tables. CVPR Workshops (VCAD). 202
Evaluating 3D Shape Analysis Methods for Robustness to Rotation Invariance
This paper analyzes the robustness of recent 3D shape descriptors to SO(3)
rotations, something that is fundamental to shape modeling. Specifically, we
formulate the task of rotated 3D object instance detection. To do so, we
consider a database of 3D indoor scenes, where objects occur in different
orientations. We benchmark different methods for feature extraction and
classification in the context of this task. We systematically contrast
different choices in a variety of experimental settings investigating the
impact on the performance of different rotation distributions, different
degrees of partial observations on the object, and the different levels of
difficulty of negative pairs. Our study, on a synthetic dataset of 3D scenes
where objects instances occur in different orientations, reveals that deep
learning-based rotation invariant methods are effective for relatively easy
settings with easy-to-distinguish pairs. However, their performance decreases
significantly when the difference in rotations on the input pair is large, or
when the degree of observation of input objects is reduced, or the difficulty
level of input pair is increased. Finally, we connect feature encodings
designed for rotation-invariant methods to 3D geometry that enable them to
acquire the property of rotation invariance.Comment: 20th Conference on Robots and Vision (CRV) 202