127 research outputs found
M-SpeechCLIP: Leveraging Large-Scale, Pre-Trained Models for Multilingual Speech to Image Retrieval
This work investigates the use of large-scale, pre-trained models (CLIP and
HuBERT) for multilingual speech-image retrieval. For non-English speech-image
retrieval, we outperform the current state-of-the-art performance by a wide
margin when training separate models for each language, and show that a single
model which processes speech in all three languages still achieves retrieval
scores comparable with the prior state-of-the-art. We identify key differences
in model behavior and performance between English and non-English settings,
presumably attributable to the English-only pre-training of CLIP and HuBERT.
Finally, we show that our models can be used for mono- and cross-lingual
speech-text retrieval and cross-lingual speech-speech retrieval, despite never
having seen any parallel speech-text or speech-speech data during training.Comment: Submitted to ICASSP 202
Keyword localisation in untranscribed speech using visually grounded speech models
Keyword localisation is the task of finding where in a speech utterance a
given query keyword occurs. We investigate to what extent keyword localisation
is possible using a visually grounded speech (VGS) model. VGS models are
trained on unlabelled images paired with spoken captions. These models are
therefore self-supervised -- trained without any explicit textual label or
location information. To obtain training targets, we first tag training images
with soft text labels using a pretrained visual classifier with a fixed
vocabulary. This enables a VGS model to predict the presence of a written
keyword in an utterance, but not its location. We consider four ways to equip
VGS models with localisations capabilities. Two of these -- a saliency approach
and input masking -- can be applied to an arbitrary prediction model after
training, while the other two -- attention and a score aggregation approach --
are incorporated directly into the structure of the model. Masked-based
localisation gives some of the best reported localisation scores from a VGS
model, with an accuracy of 57% when the system knows that a keyword occurs in
an utterance and need to predict its location. In a setting where localisation
is performed after detection, an of 25% is achieved, and in a setting
where a keyword spotting ranking pass is first performed, we get a localisation
P@10 of 32%. While these scores are modest compared to the idealised setting
with unordered bag-of-word-supervision (from transcriptions), these models do
not receive any textual or location supervision. Further analyses show that
these models are limited by the first detection or ranking pass. Moreover,
individual keyword localisation performance is correlated with the tagging
performance from the visual classifier. We also show qualitatively how and
where semantic mistakes occur, e.g. that the model locates surfer when queried
with ocean.Comment: 10 figures, 5 table
Cross-Lingual Topic Prediction for Speech Using Translations
Given a large amount of unannotated speech in a low-resource language, can we
classify the speech utterances by topic? We consider this question in the
setting where a small amount of speech in the low-resource language is paired
with text translations in a high-resource language. We develop an effective
cross-lingual topic classifier by training on just 20 hours of translated
speech, using a recent model for direct speech-to-text translation. While the
translations are poor, they are still good enough to correctly classify the
topic of 1-minute speech segments over 70% of the time - a 20% improvement over
a majority-class baseline. Such a system could be useful for humanitarian
applications like crisis response, where incoming speech in a foreign
low-resource language must be quickly assessed for further action.Comment: Accepted to ICASSP 202
Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Mode
In this paper, we show that representations capturing syllabic units emerge
when training a self-supervised speech model with a visually-grounded training
objective. We demonstrate that a nearly identical model architecture (HuBERT)
trained with a masked language modeling loss does not exhibit this same
ability, suggesting that the visual grounding objective is responsible for the
emergence of this phenomenon. We propose the use of a minimum cut algorithm to
automatically predict syllable boundaries in speech, followed by a 2-stage
clustering method to group identical syllables together. We show that our model
not only outperforms a state-of-the-art syllabic segmentation method on the
language it was trained on (English), but also generalizes in a zero-shot
fashion to Estonian. Finally, we show that the same model is capable of
zero-shot generalization for a word segmentation task on 4 other languages from
the Zerospeech Challenge, in some cases beating the previous state-of-the-art.Comment: Interspeech 2023. Code & Model:
https://github.com/jasonppy/syllable-discover
- …