18 research outputs found
Investigating the role of linguistic knowledge in vision and language tasks
Artificial Intelligence (AI) has transformed the way we interact with technology e.g., chatbots, voice-based assistants, smart devices, and so on. One particular area that has gained tremendous attention and importance is learning through multimodal data sources within AI systems. By incorporating multimodal learning into AI systems, we can bridge the gap between human and machine communication, enabling more intuitive and natural interactions. Multimodal learning is the integration of multiple sensory modalities, such as text, images, speech, and gestures, to enable machines to understand and interpret humans and the world around us more comprehensively. In this thesis we develop strategies to exploit multimodal data (specifically text and images) along with linguistic knowledge, making multimodal systems more reliable and accurate for various vision and language tasks. In the first part of the thesis, we focus on developing AI systems that can understand the visual world around us and respond in a more natural and human-like manner. This task is popularly known as image captioning. Despite the significant progress in this task, the image captions generated by the models are extremely generic and template-like for visually similar images. We address this limitation and generate detailed and image-specific captions by exploiting prior and implicit linguistic knowledge, without the need for more labeled data or computational overhead. Unlike previous work, our proposed method generates captions that reflect the image in detail. To further allow AI models to better understand and interpret context, in the second part of the thesis we leverage information from multiple modalities to gather a more comprehensive understanding of the visual data by generating scene graphs. Unlike image captioning that provides a high-level interpretation of the scene, in this setting a key question is â how do different objects/entities in the scene interact with each other? Collecting large amounts of labeled data that can capture every possible interaction is very expensive and infeasible. Hence, we propose an efficient training strategy that generates complete and informative scene graphs from incomplete and missing labels using the knowledge of label informativeness from linguistics.
In the third part of the thesis, we study the narrative descriptions of images generated from human speech i.e., natural language, to enable natural interaction between humans and machines. One fundamental and challenging problem when dealing with natural language is the task of coreference resolution. For example, in the sentence âJohn saw a dog. He petted it,â coreference resolution determines that âheâ refers to âJohnâ and âitâ refers to the âdog.â While coreference resolution may seem straightforward to humans, it poses several significant challenges for AI systems. Without proper coreference resolution, models will struggle to derive the correct meaning and produce coherent outputs. To address this important and complex problem, we propose a novel benchmark dataset for multimodal coreference resolution to evaluate coreference resolution in text and narrative grounding in images. We also propose a weakly supervised method with rule-based linguistic knowledge to address multimodal coreference resolution without a large supervised training dataset. Finally, we address the limitations of the weakly supervised learning setup in multimodal coreference resolution by proposing a semi-supervised learning strategy. By using a small labeled and a large unlabeled dataset with robust self-supervised and pseudo-labeled loss functions, we achieve strong performance gains for coreference resolution and narrative grounding in a data-efficient way. Our work addresses important aspects in vision and language and paves the way for interesting future avenues. In the last part of the thesis, we discuss in more detail directions for the future that are important for advancing the field and unlocking its full potential. Hence, continued research is needed to push the boundaries of multimodal learning
Who are you referring to?:Coreference resolution in image narrations
Coreference resolution aims to identify words and phrases which refer to same entity in a text, a core task in natural language processing. In this paper, we extend this task to resolving coreferences in long-form narrations of visual scenes. First we introduce a new dataset with annotated coreference chains and their bounding boxes, as most existing image-text datasets only contain short sentences without coreferring expressions or labeled chains. We propose a new technique that learns to identify coreference chains using weak supervision, only from image-text pairs and a regularization using prior linguistic knowledge. Our model yields large performance gains over several strong baselines in resolving coreferences. We also show that coreference resolution helps improving grounding narratives in images
Not All Relations are Equal: Mining Informative Labels for Scene Graph Generation
Scene graph generation (SGG) aims to capture a wide variety of interactions
between pairs of objects, which is essential for full scene understanding.
Existing SGG methods trained on the entire set of relations fail to acquire
complex reasoning about visual and textual correlations due to various biases
in training data. Learning on trivial relations that indicate generic spatial
configuration like 'on' instead of informative relations such as 'parked on'
does not enforce this complex reasoning, harming generalization. To address
this problem, we propose a novel framework for SGG training that exploits
relation labels based on their informativeness. Our model-agnostic training
procedure imputes missing informative relations for less informative samples in
the training data and trains a SGG model on the imputed labels along with
existing annotations. We show that this approach can successfully be used in
conjunction with state-of-the-art SGG methods and improves their performance
significantly in multiple metrics on the standard Visual Genome benchmark.
Furthermore, we obtain considerable improvements for unseen triplets in a more
challenging zero-shot setting.Comment: 16 page
Language-guided Robot Grasping: CLIP-based Referring Grasp Synthesis in Clutter
Robots operating in human-centric environments require the integration of
visual grounding and grasping capabilities to effectively manipulate objects
based on user instructions. This work focuses on the task of referring grasp
synthesis, which predicts a grasp pose for an object referred through natural
language in cluttered scenes. Existing approaches often employ multi-stage
pipelines that first segment the referred object and then propose a suitable
grasp, and are evaluated in private datasets or simulators that do not capture
the complexity of natural indoor scenes. To address these limitations, we
develop a challenging benchmark based on cluttered indoor scenes from OCID
dataset, for which we generate referring expressions and connect them with
4-DoF grasp poses. Further, we propose a novel end-to-end model (CROG) that
leverages the visual grounding capabilities of CLIP to learn grasp synthesis
directly from image-text pairs. Our results show that vanilla integration of
CLIP with pretrained models transfers poorly in our challenging benchmark,
while CROG achieves significant improvements both in terms of grounding and
grasping. Extensive robot experiments in both simulation and hardware
demonstrate the effectiveness of our approach in challenging interactive object
grasping scenarios that include clutter.Comment: Poster CoRL 2023. Dataset and code available here:
https://github.com/gtziafas/OCID-VL
Language-guided Robot Grasping: CLIP-based Referring Grasp Synthesis in Clutter
Robots operating in human-centric environments require the integration of visual grounding and grasping capabilities to effectively manipulate objects based on user instructions. This work focuses on the task of referring grasp synthesis, which predicts a grasp pose for an object referred through natural language in cluttered scenes. Existing approaches often employ multi-stage pipelines that first segment the referred object and then propose a suitable grasp, and are evaluated in private datasets or simulators that do not capture the complexity of natural indoor scenes. To address these limitations, we develop a challenging benchmark based on cluttered indoor scenes from OCID dataset, for which we generate referring expressions and connect them with 4-DoF grasp poses. Further, we propose a novel end-to-end model (CROG) that leverages the visual grounding capabilities of CLIP to learn grasp synthesis directly from image-text pairs. Our results show that vanilla integration of CLIP with pretrained models transfers poorly in our challenging benchmark, while CROG achieves significant improvements both in terms of grounding and grasping. Extensive robot experiments in both simulation and hardware demonstrate the effectiveness of our approach in challenging interactive object grasping scenarios that include clutter
Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories
We propose Encyclopedic-VQA, a large scale visual question answering (VQA)
dataset featuring visual questions about detailed properties of fine-grained
categories and instances. It contains 221k unique question+answer pairs each
matched with (up to) 5 images, resulting in a total of 1M VQA samples.
Moreover, our dataset comes with a controlled knowledge base derived from
Wikipedia, marking the evidence to support each answer. Empirically, we show
that our dataset poses a hard challenge for large vision+language models as
they perform poorly on our dataset: PaLI [14] is state-of-the-art on OK-VQA
[37], yet it only achieves 13.0% accuracy on our dataset. Moreover, we
experimentally show that progress on answering our encyclopedic questions can
be achieved by augmenting large models with a mechanism that retrieves relevant
information from the knowledge base. An oracle experiment with perfect
retrieval achieves 87.0% accuracy on the single-hop portion of our dataset, and
an automatic retrieval-augmented prototype yields 48.8%. We believe that our
dataset enables future research on retrieval-augmented vision+language models.
It is available at
https://github.com/google-research/google-research/tree/master/encyclopedic_vqa .Comment: ICCV'2