23 research outputs found
An Investigation of Monotonic Transducers for Large-Scale Automatic Speech Recognition
The two most popular loss functions for streaming end-to-end automatic speech
recognition (ASR) are the RNN-Transducer (RNN-T) and the connectionist temporal
classification (CTC) objectives. Both perform an alignment-free training by
marginalizing over all possible alignments, but use different transition rules.
Between these two loss types we can classify the monotonic RNN-T (MonoRNN-T)
and the recently proposed CTC-like Transducer (CTC-T), which both can be
realized using the graph temporal classification-transducer (GTC-T) loss
function. Monotonic transducers have a few advantages. First, RNN-T can suffer
from runaway hallucination, where a model keeps emitting non-blank symbols
without advancing in time, often in an infinite loop. Secondly, monotonic
transducers consume exactly one model score per time step and are therefore
more compatible and unifiable with traditional FST-based hybrid ASR decoders.
However, the MonoRNN-T so far has been found to have worse accuracy than RNN-T.
It does not have to be that way, though: By regularizing the training - via
joint LAS training or parameter initialization from RNN-T - both MonoRNN-T and
CTC-T perform as well - or better - than RNN-T. This is demonstrated for
LibriSpeech and for a large-scale in-house data set.Comment: Submitted to Interspeech 202
Streaming Audio-Visual Speech Recognition with Alignment Regularization
Recognizing a word shortly after it is spoken is an important requirement for
automatic speech recognition (ASR) systems in real-world scenarios. As a
result, a large body of work on streaming audio-only ASR models has been
presented in the literature. However, streaming audio-visual automatic speech
recognition (AV-ASR) has received little attention in earlier works. In this
work, we propose a streaming AV-ASR system based on a hybrid connectionist
temporal classification (CTC)/attention neural network architecture. The audio
and the visual encoder neural networks are both based on the conformer
architecture, which is made streamable using chunk-wise self-attention (CSA)
and causal convolution. Streaming recognition with a decoder neural network is
realized by using the triggered attention technique, which performs
time-synchronous decoding with joint CTC/attention scoring. For frame-level ASR
criteria, such as CTC, a synchronized response from the audio and visual
encoders is critical for a joint AV decision making process. In this work, we
propose a novel alignment regularization technique that promotes
synchronization of the audio and visual encoder, which in turn results in
better word error rates (WERs) at all SNR levels for streaming and offline
AV-ASR models. The proposed AV-ASR model achieves WERs of 2.0% and 2.6% on the
Lip Reading Sentences 3 (LRS3) dataset in an offline and online setup,
respectively, which both present state-of-the-art results when no external
training data are used.Comment: Submitted to ICASSP202