We investigate compositional structures in data embeddings from pre-trained
vision-language models (VLMs). Traditionally, compositionality has been
associated with algebraic operations on embeddings of words from a pre-existing
vocabulary. In contrast, we seek to approximate representations from an encoder
as combinations of a smaller set of vectors in the embedding space. These
vectors can be seen as "ideal words" for generating concepts directly within
the embedding space of the model. We first present a framework for
understanding compositional structures from a geometric perspective. We then
explain what these compositional structures entail probabilistically in the
case of VLM embeddings, providing intuitions for why they arise in practice.
Finally, we empirically explore these structures in CLIP's embeddings and we
evaluate their usefulness for solving different vision-language tasks such as
classification, debiasing, and retrieval. Our results show that simple linear
algebraic operations on embedding vectors can be used as compositional and
interpretable methods for regulating the behavior of VLMs.Comment: 18 pages, 9 figures, 7 table