End-to-end Speech Translation (E2E ST) aims to directly translate source
speech into target text. Existing ST methods perform poorly when only extremely
small speech-text data are available for training. We observe that an ST
model's performance closely correlates with its embedding similarity between
speech and source transcript. In this paper, we propose Word-Aligned
COntrastive learning (WACO), a simple and effective method for extremely
low-resource speech-to-text translation. Our key idea is bridging word-level
representations for both speech and text modalities via contrastive learning.
We evaluate WACO and other methods on the MuST-C dataset, a widely used ST
benchmark, and on a low-resource direction Maltese-English from IWSLT 2023. Our
experiments demonstrate that WACO outperforms the best baseline by 9+ BLEU
points with only 1-hour parallel ST data. Code is available at
https://github.com/owaski/WACO.Comment: ACL 2023 Poste