We propose Encyclopedic-VQA, a large scale visual question answering (VQA)
dataset featuring visual questions about detailed properties of fine-grained
categories and instances. It contains 221k unique question+answer pairs each
matched with (up to) 5 images, resulting in a total of 1M VQA samples.
Moreover, our dataset comes with a controlled knowledge base derived from
Wikipedia, marking the evidence to support each answer. Empirically, we show
that our dataset poses a hard challenge for large vision+language models as
they perform poorly on our dataset: PaLI [14] is state-of-the-art on OK-VQA
[37], yet it only achieves 13.0% accuracy on our dataset. Moreover, we
experimentally show that progress on answering our encyclopedic questions can
be achieved by augmenting large models with a mechanism that retrieves relevant
information from the knowledge base. An oracle experiment with perfect
retrieval achieves 87.0% accuracy on the single-hop portion of our dataset, and
an automatic retrieval-augmented prototype yields 48.8%. We believe that our
dataset enables future research on retrieval-augmented vision+language models.
It is available at
https://github.com/google-research/google-research/tree/master/encyclopedic_vqa .Comment: ICCV'2