427 research outputs found
Improved acoustic word embeddings for zero-resource languages using multilingual transfer
Acoustic word embeddings are fixed-dimensional representations of
variable-length speech segments. Such embeddings can form the basis for speech
search, indexing and discovery systems when conventional speech recognition is
not possible. In zero-resource settings where unlabelled speech is the only
available resource, we need a method that gives robust embeddings on an
arbitrary language. Here we explore multilingual transfer: we train a single
supervised embedding model on labelled data from multiple well-resourced
languages and then apply it to unseen zero-resource languages. We consider
three multilingual recurrent neural network (RNN) models: a classifier trained
on the joint vocabularies of all training languages; a Siamese RNN trained to
discriminate between same and different words from multiple languages; and a
correspondence autoencoder (CAE) RNN trained to reconstruct word pairs. In a
word discrimination task on six target languages, all of these models
outperform state-of-the-art unsupervised models trained on the zero-resource
languages themselves, giving relative improvements of more than 30% in average
precision. When using only a few training languages, the multilingual CAE
performs better, but with more training languages the other multilingual models
perform similarly. Using more training languages is generally beneficial, but
improvements are marginal on some languages. We present probing experiments
which show that the CAE encodes more phonetic, word duration, language identity
and speaker information than the other multilingual models.Comment: 11 pages, 7 figures, 8 tables. arXiv admin note: text overlap with
arXiv:2002.02109. Submitted to the IEEE Transactions on Audio, Speech and
Language Processin
Acoustic Word Embeddings for Zero-Resource Languages Using Self-Supervised Contrastive Learning and Multilingual Adaptation
Acoustic word embeddings (AWEs) are fixed-dimensional representations of
variable-length speech segments. For zero-resource languages where labelled
data is not available, one AWE approach is to use unsupervised
autoencoder-based recurrent models. Another recent approach is to use
multilingual transfer: a supervised AWE model is trained on several
well-resourced languages and then applied to an unseen zero-resource language.
We consider how a recent contrastive learning loss can be used in both the
purely unsupervised and multilingual transfer settings. Firstly, we show that
terms from an unsupervised term discovery system can be used for contrastive
self-supervision, resulting in improvements over previous unsupervised
monolingual AWE models. Secondly, we consider how multilingual AWE models can
be adapted to a specific zero-resource language using discovered terms. We find
that self-supervised contrastive adaptation outperforms adapted multilingual
correspondence autoencoder and Siamese AWE models, giving the best overall
results in a word discrimination task on six zero-resource languages.Comment: Accepted to SLT 202
Neural approaches to spoken content embedding
Comparing spoken segments is a central operation to speech processing.
Traditional approaches in this area have favored frame-level dynamic
programming algorithms, such as dynamic time warping, because they require no
supervision, but they are limited in performance and efficiency. As an
alternative, acoustic word embeddings -- fixed-dimensional vector
representations of variable-length spoken word segments -- have begun to be
considered for such tasks as well. However, the current space of such
discriminative embedding models, training approaches, and their application to
real-world downstream tasks is limited. We start by considering ``single-view"
training losses where the goal is to learn an acoustic word embedding model
that separates same-word and different-word spoken segment pairs. Then, we
consider ``multi-view" contrastive losses. In this setting, acoustic word
embeddings are learned jointly with embeddings of character sequences to
generate acoustically grounded embeddings of written words, or acoustically
grounded word embeddings.
In this thesis, we contribute new discriminative acoustic word embedding
(AWE) and acoustically grounded word embedding (AGWE) approaches based on
recurrent neural networks (RNNs). We improve model training in terms of both
efficiency and performance. We take these developments beyond English to
several low-resource languages and show that multilingual training improves
performance when labeled data is limited. We apply our embedding models, both
monolingual and multilingual, to the downstream tasks of query-by-example
speech search and automatic speech recognition. Finally, we show how our
embedding approaches compare with and complement more recent self-supervised
speech models.Comment: PhD thesi
Allophant: Cross-lingual Phoneme Recognition with Articulatory Attributes
This paper proposes Allophant, a multilingual phoneme recognizer. It requires
only a phoneme inventory for cross-lingual transfer to a target language,
allowing for low-resource recognition. The architecture combines a
compositional phone embedding approach with individually supervised phonetic
attribute classifiers in a multi-task architecture. We also introduce
Allophoible, an extension of the PHOIBLE database. When combined with a
distance based mapping approach for grapheme-to-phoneme outputs, it allows us
to train on PHOIBLE inventories directly. By training and evaluating on 34
languages, we found that the addition of multi-task learning improves the
model's capability of being applied to unseen phonemes and phoneme inventories.
On supervised languages we achieve phoneme error rate improvements of 11
percentage points (pp.) compared to a baseline without multi-task learning.
Evaluation of zero-shot transfer on 84 languages yielded a decrease in PER of
2.63 pp. over the baseline.Comment: 5 pages, 2 figures, 2 tables, accepted to INTERSPEECH 2023; published
versio
Towards hate speech detection in low-resource languages: Comparing ASR to acoustic word embeddings on Wolof and Swahili
We consider hate speech detection through keyword spotting on radio
broadcasts. One approach is to build an automatic speech recognition (ASR)
system for the target low-resource language. We compare this to using acoustic
word embedding (AWE) models that map speech segments to a space where matching
words have similar vectors. We specifically use a multilingual AWE model
trained on labelled data from well-resourced languages to spot keywords in data
in the unseen target language. In contrast to ASR, the AWE approach only
requires a few keyword exemplars. In controlled experiments on Wolof and
Swahili where training and test data are from the same domain, an ASR model
trained on just five minutes of data outperforms the AWE approach. But in an
in-the-wild test on Swahili radio broadcasts with actual hate speech keywords,
the AWE model (using one minute of template data) is more robust, giving
similar performance to an ASR system trained on 30 hours of labelled data.Comment: Accepted to Interspeech 202
- …