22 research outputs found
IRFL: Image Recognition of Figurative Language
Figures of speech such as metaphors, similes, and idioms allow language to be
expressive, invoke emotion, and communicate abstract ideas that might otherwise
be difficult to visualize. These figurative forms are often conveyed through
multiple modes, such as text and images, and frequently appear in advertising,
news, social media, etc. Understanding multimodal figurative language is an
essential component of human communication, and it plays a significant role in
our daily interactions. While humans can intuitively understand multimodal
figurative language, this poses a challenging task for machines that requires
the cognitive ability to map between domains, abstraction, commonsense, and
profound language and cultural knowledge. In this work, we propose the Image
Recognition of Figurative Language dataset to examine vision and language
models' understanding of figurative language. We leverage human annotation and
an automatic pipeline we created to generate a multimodal dataset and introduce
two novel tasks as a benchmark for multimodal figurative understanding. We
experiment with several baseline models and find that all perform substantially
worse than humans. We hope our dataset and benchmark will drive the development
of models that will better understand figurative language
VASR: Visual Analogies of Situation Recognition
A core process in human cognition is analogical mapping: the ability to
identify a similar relational structure between different situations. We
introduce a novel task, Visual Analogies of Situation Recognition, adapting the
classical word-analogy task into the visual domain. Given a triplet of images,
the task is to select an image candidate B' that completes the analogy (A to A'
is like B to what?). Unlike previous work on visual analogy that focused on
simple image transformations, we tackle complex analogies requiring
understanding of scenes.
We leverage situation recognition annotations and the CLIP model to generate
a large set of 500k candidate analogies. Crowdsourced annotations for a sample
of the data indicate that humans agree with the dataset label ~80% of the time
(chance level 25%). Furthermore, we use human annotations to create a
gold-standard dataset of 3,820 validated analogies. Our experiments demonstrate
that state-of-the-art models do well when distractors are chosen randomly
(~86%), but struggle with carefully chosen distractors (~53%, compared to 90%
human accuracy). We hope our dataset will encourage the development of new
analogy-making models. Website: https://vasr-dataset.github.io/Comment: Accepted to AAAI 2023. Website: https://vasr-dataset.github.io