42 research outputs found
Image Representations and New Domains in Neural Image Captioning
We examine the possibility that recent promising results in automatic caption
generation are due primarily to language models. By varying image
representation quality produced by a convolutional neural network, we find that
a state-of-the-art neural captioning algorithm is able to produce quality
captions even when provided with surprisingly poor image representations. We
replicate this result in a new, fine-grained, transfer learned captioning
domain, consisting of 66K recipe image/title pairs. We also provide some
experiments regarding the appropriateness of datasets for automatic captioning,
and find that having multiple captions per image is beneficial, but not an
absolute requirement.Comment: 11 Pages, 5 Images, To appear at EMNLP 2015's Vision + Learning
worksho
Text encoders are performance bottlenecks in contrastive vision-language models
Performant vision-language (VL) models like CLIP represent captions using a
single vector. How much information about language is lost in this bottleneck?
We first curate CompPrompts, a set of increasingly compositional image captions
that VL models should be able to capture (e.g., single object, to
object+property, to multiple interacting objects). Then, we train text-only
recovery probes that aim to reconstruct captions from single-vector text
representations produced by several VL models. This approach doesn't require
images, allowing us to test on a broader range of scenes compared to prior
work. We find that: 1) CLIP's text encoder falls short on object relationships,
attribute-object association, counting, and negations; 2) some text encoders
work significantly better than others; and 3) text-only recovery performance
predicts multi-modal matching performance on ControlledImCaps: a new evaluation
benchmark we collect+release consisting of fine-grained compositional
images+captions. Specifically -- our results suggest text-only recoverability
is a necessary (but not sufficient) condition for modeling compositional
factors in contrastive vision+language models. We release data+code
What's "up" with vision-language models? Investigating their struggle with spatial reasoning
Recent vision-language (VL) models are powerful, but can they reliably
distinguish "right" from "left"? We curate three new corpora to quantify model
comprehension of such basic spatial relations. These tests isolate spatial
reasoning more precisely than existing datasets like VQAv2, e.g., our What'sUp
benchmark contains sets of photographs varying only the spatial relations of
objects, keeping their identity fixed (see Figure 1: models must comprehend not
only the usual case of a dog under a table, but also, the same dog on top of
the same table). We evaluate 18 VL models, finding that all perform poorly,
e.g., BLIP finetuned on VQAv2, which nears human parity on VQAv2, achieves 56%
accuracy on our benchmarks vs. humans at 99%. We conclude by studying causes of
this surprising behavior, finding: 1) that popular vision-language pretraining
corpora like LAION-2B contain little reliable data for learning spatial
relationships; and 2) that basic modeling interventions like up-weighting
preposition-containing instances or fine-tuning on our corpora are not
sufficient to address the challenges our benchmarks pose. We are hopeful that
these corpora will facilitate further research, and we release our data and
code at https://github.com/amitakamath/whatsup_vlms.Comment: EMNLP 202
CHAMPAGNE: Learning Real-world Conversation from Large-Scale Web Videos
Visual information is central to conversation: body gestures and physical
behaviour, for example, contribute to meaning that transcends words alone. To
date, however, most neural conversational models are limited to just text. We
introduce CHAMPAGNE, a generative model of conversations that can account for
visual contexts. To train CHAMPAGNE, we collect and release YTD-18M, a
large-scale corpus of 18M video-based dialogues. YTD-18M is constructed from
web videos: crucial to our data collection pipeline is a pretrained language
model that converts error-prone automatic transcripts to a cleaner dialogue
format while maintaining meaning. Human evaluation reveals that YTD-18M is more
sensible and specific than prior resources (MMDialog, 1M dialogues), while
maintaining visual-groundedness. Experiments demonstrate that 1) CHAMPAGNE
learns to conduct conversation from YTD-18M; and 2) when fine-tuned, it
achieves state-of-the-art results on four vision-language tasks focused on
real-world conversations. We release data, models, and code.Comment: ICCV 2023, Project page: https://seungjuhan.me/champagn
Reframing Human-AI Collaboration for Generating Free-Text Explanations
Large language models are increasingly capable of generating fluent-appearing
text with relatively little task-specific supervision. But can these models
accurately explain classification decisions? We consider the task of generating
free-text explanations using human-written examples in a few-shot manner. We
find that (1) authoring higher quality prompts results in higher quality
generations; and (2) surprisingly, in a head-to-head comparison, crowdworkers
often prefer explanations generated by GPT-3 to crowdsourced explanations in
existing datasets. Our human studies also show, however, that while models
often produce factual, grammatical, and sufficient explanations, they have room
to improve along axes such as providing novel information and supporting the
label. We create a pipeline that combines GPT-3 with a supervised filter that
incorporates binary acceptability judgments from humans in the loop. Despite
the intrinsic subjectivity of acceptability judgments, we demonstrate that
acceptability is partially correlated with various fine-grained attributes of
explanations. Our approach is able to consistently filter GPT-3-generated
explanations deemed acceptable by humans.Comment: NAACL 2022 Camera-ready. 13 pages main + references, 14 pages
appendi
Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images
Weird, unusual, and uncanny images pique the curiosity of observers because
they challenge commonsense. For example, an image released during the 2022
world cup depicts the famous soccer stars Lionel Messi and Cristiano Ronaldo
playing chess, which playfully violates our expectation that their competition
should occur on the football field. Humans can easily recognize and interpret
these unconventional images, but can AI models do the same? We introduce
WHOOPS!, a new dataset and benchmark for visual commonsense. The dataset is
comprised of purposefully commonsense-defying images created by designers using
publicly-available image generation tools like Midjourney. We consider several
tasks posed over the dataset. In addition to image captioning, cross-modal
matching, and visual question answering, we introduce a difficult explanation
generation task, where models must identify and explain why a given image is
unusual. Our results show that state-of-the-art models such as GPT3 and BLIP2
still lag behind human performance on WHOOPS!. We hope our dataset will inspire
the development of AI models with stronger visual commonsense reasoning
abilities. Data, models and code are available at the project website:
whoops-benchmark.github.i