1 research outputs found
End-to-End Visual Speech Recognition for Small-Scale Datasets
Visual speech recognition models traditionally consist of two stages, feature
extraction and classification. Several deep learning approaches have been
recently presented aiming to replace the feature extraction stage by
automatically extracting features from mouth images. However, research on joint
learning of features and classification remains limited. In addition, most of
the existing methods require large amounts of data in order to achieve
state-of-the-art performance, otherwise they under-perform. In this work, we
present an end-to-end visual speech recognition system based on fully-connected
layers and Long-Short Memory (LSTM) networks which is suitable for small-scale
datasets. The model consists of two streams which extract features directly
from the mouth and difference images, respectively. The temporal dynamics in
each stream are modelled by a Bidirectional LSTM (BLSTM) and the fusion of the
two streams takes place via another BLSTM. An absolute improvement of 0.6%,
3.4%, 3.9%, 11.4% over the state-of-the-art is reported on the OuluVS2, CUAVE,
AVLetters and AVLetters2 databases, respectively.Comment: Submitted to Pattern Recognition Letter