2 research outputs found
Worse WER, but Better BLEU? Leveraging Word Embedding as Intermediate in Multitask End-to-End Speech Translation
Speech translation (ST) aims to learn transformations from speech in the
source language to the text in the target language. Previous works show that
multitask learning improves the ST performance, in which the recognition
decoder generates the text of the source language, and the translation decoder
obtains the final translations based on the output of the recognition decoder.
Because whether the output of the recognition decoder has the correct semantics
is more critical than its accuracy, we propose to improve the multitask ST
model by utilizing word embedding as the intermediate.Comment: Accepted by ACL 202
AlloST: Low-resource Speech Translation without Source Transcription
The end-to-end architecture has made promising progress in speech translation
(ST). However, the ST task is still challenging under low-resource conditions.
Most ST models have shown unsatisfactory results, especially in the absence of
word information from the source speech utterance. In this study, we survey
methods to improve ST performance without using source transcription, and
propose a learning framework that utilizes a language-independent universal
phone recognizer. The framework is based on an attention-based
sequence-to-sequence model, where the encoder generates the phonetic embeddings
and phone-aware acoustic representations, and the decoder controls the fusion
of the two embedding streams to produce the target token sequence. In addition
to investigating different fusion strategies, we explore the specific usage of
byte pair encoding (BPE), which compresses a phone sequence into a
syllable-like segmented sequence. Due to the conversion of symbols, a segmented
sequence represents not only pronunciation but also language-dependent
information lacking in phones. Experiments conducted on the Fisher
Spanish-English and Taigi-Mandarin drama corpora show that our method
outperforms the conformer-based baseline, and the performance is close to that
of the existing best method using source transcription.Comment: Accepted by Interspeech202