5 research outputs found
Follow-Up Differential Descriptions: Language Models Resolve Ambiguities for Image Classification
A promising approach for improving the performance of vision-language models
like CLIP for image classification is to extend the class descriptions (i.e.,
prompts) with related attributes, e.g., using brown sparrow instead of sparrow.
However, current zero-shot methods select a subset of attributes regardless of
commonalities between the target classes, potentially providing no useful
information that would have helped to distinguish between them. For instance,
they may use color instead of bill shape to distinguish between sparrows and
wrens, which are both brown. We propose Follow-up Differential Descriptions
(FuDD), a zero-shot approach that tailors the class descriptions to each
dataset and leads to additional attributes that better differentiate the target
classes. FuDD first identifies the ambiguous classes for each image, and then
uses a Large Language Model (LLM) to generate new class descriptions that
differentiate between them. The new class descriptions resolve the initial
ambiguity and help predict the correct label. In our experiments, FuDD
consistently outperforms generic description ensembles and naive LLM-generated
descriptions on 12 datasets. We show that differential descriptions are an
effective tool to resolve class ambiguities, which otherwise significantly
degrade the performance. We also show that high quality natural language class
descriptions produced by FuDD result in comparable performance to few-shot
adaptation methods.Comment: Code: https://github.com/BatsResearch/fud