11 research outputs found
Cross3DVG: Baseline and Dataset for Cross-Dataset 3D Visual Grounding on Different RGB-D Scans
We present Cross3DVG, a novel task for cross-dataset visual grounding in 3D
scenes, revealing the limitations of existing 3D visual grounding models using
restricted 3D resources and thus easily overfit to a specific 3D dataset. To
facilitate Cross3DVG, we have created a large-scale 3D visual grounding dataset
containing more than 63k diverse descriptions of 3D objects within 1,380 indoor
RGB-D scans from 3RScan with human annotations, paired with the existing 52k
descriptions on ScanRefer. We perform Cross3DVG by training a model on the
source 3D visual grounding dataset and then evaluating it on the target dataset
constructed in different ways (e.g., different sensors, 3D reconstruction
methods, and language annotators) without using target labels. We conduct
comprehensive experiments using established visual grounding models, as well as
a CLIP-based 2D-3D integration method, designed to bridge the gaps between 3D
datasets. By performing Cross3DVG tasks, we found that (i) cross-dataset 3D
visual grounding has significantly lower performance than learning and
evaluation with a single dataset, suggesting much room for improvement in
cross-dataset generalization of 3D visual grounding, (ii) better detectors and
transformer-based localization modules for 3D grounding are beneficial for
enhancing 3D grounding performance and (iii) fusing 2D-3D data using CLIP
demonstrates further performance improvements. Our Cross3DVG task will provide
a benchmark for developing robust 3D visual grounding models capable of
handling diverse 3D scenes while leveraging deep language understanding
CityRefer: Geography-aware 3D Visual Grounding Dataset on City-scale Point Cloud Data
City-scale 3D point cloud is a promising way to express detailed and
complicated outdoor structures. It encompasses both the appearance and geometry
features of segmented city components, including cars, streets, and buildings,
that can be utilized for attractive applications such as user-interactive
navigation of autonomous vehicles and drones. However, compared to the
extensive text annotations available for images and indoor scenes, the scarcity
of text annotations for outdoor scenes poses a significant challenge for
achieving these applications. To tackle this problem, we introduce the
CityRefer dataset for city-level visual grounding. The dataset consists of 35k
natural language descriptions of 3D objects appearing in SensatUrban city
scenes and 5k landmarks labels synchronizing with OpenStreetMap. To ensure the
quality and accuracy of the dataset, all descriptions and labels in the
CityRefer dataset are manually verified. We also have developed a baseline
system that can learn encoded language descriptions, 3D object instances, and
geographical information about the city's landmarks to perform visual grounding
on the CityRefer dataset. To the best of our knowledge, the CityRefer dataset
is the largest city-level visual grounding dataset for localizing specific 3D
objects.Comment: NeurIPS D&B 2023. The first two authors are equally contribute
Time-Aware Latent Concept Expansion for Microblog Search
Incorporating the temporal property of words into query expansion methods based on relevance feedback has been shown to have a significant positive effect on microblog search.In contrast to such word-based query expansion methods, we propose a concept-based query expansion method based on a temporal relevance model that uses the temporal variation of concepts (e.g., terms and phrases) on microblogs. Our model naturally extends an extremely effective existing concept-based relevance model by tracking the concept frequency over time.Moreover, the proposed model produces important concepts that are frequently used within a particular time periodassociated with a given topic, which better discriminate between relevant and non-relevant microblog documents than words.Our experiments using a corpus of microblog data (Tweets2011 corpus) show that the proposed concept-based query expansion method improves search performance significantly, especially for highly relevant documents
ScanQA: 3D Question Answering for Spatial Scene Understanding
We propose a new 3D spatial understanding task of 3D Question Answering
(3D-QA). In the 3D-QA task, models receive visual information from the entire
3D scene of the rich RGB-D indoor scan and answer the given textual questions
about the 3D scene. Unlike the 2D-question answering of VQA, the conventional
2D-QA models suffer from problems with spatial understanding of object
alignment and directions and fail the object identification from the textual
questions in 3D-QA. We propose a baseline model for 3D-QA, named ScanQA model,
where the model learns a fused descriptor from 3D object proposals and encoded
sentence embeddings. This learned descriptor correlates the language
expressions with the underlying geometric features of the 3D scan and
facilitates the regression of 3D bounding boxes to determine described objects
in textual questions and outputs correct answers. We collected human-edited
question-answer pairs with free-form answers that are grounded to 3D objects in
each 3D scene. Our new ScanQA dataset contains over 40K question-answer pairs
from the 800 indoor scenes drawn from the ScanNet dataset. To the best of our
knowledge, the proposed 3D-QA task is the first large-scale effort to perform
object-grounded question-answering in 3D environments.Comment: CVPR2022. The first three authors are equally contributed. Project
page: https://github.com/ATR-DBI/ScanQ
Generating an Event Timeline About Daily Activities From a Semantic Concept Stream
Recognizing activities of daily living (ADLs) in the real world is an important task for understanding everyday human life. However, even though our life events consist of chronological ADLs with the corresponding places and objects (e.g., drinking coffee in the living room after making coffee in the kitchen and walking to the living room), most existing works focus on predicting individual activity labels from sensor data. In this paper, we introduce a novel framework that produces an event timeline of ADLs in a home environment. The proposed method combines semantic concepts such as action, object, and place detected by sensors for generating stereotypical event sequences with the following three real-world properties. First, we use temporal interactions among concepts to remove objects and places unrelated to each action. Second, we use commonsense knowledge mined from a language resource to find a possible combination of concepts in the real world. Third, we use temporal variations of events to filter repetitive events, since our daily life changes over time. We use cross-place validation to evaluate our proposed method on a daily-activities dataset with manually labeled event descriptions. The empirical evaluation demonstrates that our method using real-world properties improves the performance of generating an event timeline over diverse environments
Egocentric Video Search via Physical Interactions
Retrieving past egocentric videos about personal daily life is important to support and augment human memory. Most previous retrieval approaches have ignored the crucial feature of human-physical world interactions, which is greatly related to our memory and experience of daily activities. In this paper, we propose a gesture-based egocentric video retrieval framework, which retrieves past visual experience using body gestures as non-verbal queries. We use a probabilistic framework based on a canonical correlation analysis that models physical interactions through a latent space and uses them for egocentric video retrieval and re-ranking search results. By incorporating physical interactions into the retrieval models, we address the problems resulting from the variability of human motions. We evaluate our proposed method on motion and egocentric video datasets about daily activities in household settings and demonstrate that our egocentric video retrieval framework robustly improves retrieval performance when retrieving past videos from personal and even other persons' video archives