2 research outputs found
Deformation Flow Based Two-Stream Network for Lip Reading
Lip reading is the task of recognizing the speech content by analyzing
movements in the lip region when people are speaking. Observing on the
continuity in adjacent frames in the speaking process, and the consistency of
the motion patterns among different speakers when they pronounce the same
phoneme, we model the lip movements in the speaking process as a sequence of
apparent deformations in the lip region. Specifically, we introduce a
Deformation Flow Network (DFN) to learn the deformation flow between adjacent
frames, which directly captures the motion information within the lip region.
The learned deformation flow is then combined with the original grayscale
frames with a two-stream network to perform lip reading. Different from
previous two-stream networks, we make the two streams learn from each other in
the learning process by introducing a bidirectional knowledge distillation loss
to train the two branches jointly. Owing to the complementary cues provided by
different branches, the two-stream network shows a substantial improvement over
using either single branch. A thorough experimental evaluation on two
large-scale lip reading benchmarks is presented with detailed analysis. The
results accord with our motivation, and show that our method achieves
state-of-the-art or comparable performance on these two challenging datasets.Comment: 7 pages, FG 202
Learn an Effective Lip Reading Model without Pains
Lip reading, also known as visual speech recognition, aims to recognize the
speech content from videos by analyzing the lip dynamics. There have been
several appealing progress in recent years, benefiting much from the rapidly
developed deep learning techniques and the recent large-scale lip-reading
datasets. Most existing methods obtained high performance by constructing a
complex neural network, together with several customized training strategies
which were always given in a very brief description or even shown only in the
source code. We find that making proper use of these strategies could always
bring exciting improvements without changing much of the model. Considering the
non-negligible effects of these strategies and the existing tough status to
train an effective lip reading model, we perform a comprehensive quantitative
study and comparative analysis, for the first time, to show the effects of
several different choices for lip reading. By only introducing some easy-to-get
refinements to the baseline pipeline, we obtain an obvious improvement of the
performance from 83.7% to 88.4% and from 38.2% to 55.7% on two largest public
available lip reading datasets, LRW and LRW-1000, respectively. They are
comparable and even surpass the existing state-of-the-art results