Building artificial intelligence (AI) systems on top of a set of foundation
models (FMs) is becoming a new paradigm in AI research. Their representative
and generative abilities learnt from vast amounts of data can be easily adapted
and transferred to a wide range of downstream tasks without extra training from
scratch. However, leveraging FMs in cross-modal generation remains
under-researched when audio modality is involved. On the other hand,
automatically generating semantically-relevant sound from visual input is an
important problem in cross-modal generation studies. To solve this
vision-to-audio (V2A) generation problem, existing methods tend to design and
build complex systems from scratch using modestly sized datasets. In this
paper, we propose a lightweight solution to this problem by leveraging
foundation models, specifically CLIP, CLAP, and AudioLDM. We first investigate
the domain gap between the latent space of the visual CLIP and the auditory
CLAP models. Then we propose a simple yet effective mapper mechanism
(V2A-Mapper) to bridge the domain gap by translating the visual input between
CLIP and CLAP spaces. Conditioned on the translated CLAP embedding, pretrained
audio generative FM AudioLDM is adopted to produce high-fidelity and
visually-aligned sound. Compared to previous approaches, our method only
requires a quick training of the V2A-Mapper. We further analyze and conduct
extensive experiments on the choice of the V2A-Mapper and show that a
generative mapper is better at fidelity and variability (FD) while a regression
mapper is slightly better at relevance (CS). Both objective and subjective
evaluation on two V2A datasets demonstrate the superiority of our proposed
method compared to current state-of-the-art approaches - trained with 86% fewer
parameters but achieving 53% and 19% improvement in FD and CS, respectively.Comment: 13 pages, 10 figures. Demo page: https://v2a-mapper.github.io