Bregler, C.

Hild, Hermann

Manke, Stefan

Waibel, Alex

KITopen

the task to recognize 10 isolated letters and used artificialmarkers on the lips. No visual feature extraction was inte-grated into their model.Also of interest are some psychological studies abouthuman speechreading and their approach to describe thehuman performance. This measurements could also beapplied to the performance analysis of automatedspeechreading systems. Dodd and Campbell [3], andDemorest and Bernstein [2] did some valuable work in thisarea.7. CONCLUSION AND FUTURE WORKWe have shown how a state-of-the-art speech recognitionsystem can be improved by considering additional visualinformation for the recognition process. This is true foroptimal recording conditions but even more for non-opti-mal recording conditions as they usually exist in realworld applications. Experiments were performed on theconnected letter recognition task, but similar results can beexpected for continuous speech recognition as well.Work is in progress to integrate not only the time inde-pendent weight sharing but also position independentweight sharing for the visual TDNN, in order to locate andtrack the lips. We are also on the way to largely increaseour database in order to achieve better recognition ratesand to train speaker independently. Investigations of dif-ferent approaches are still in progress in order to combinevisual and acoustic features and to apply different prepro-cessing to the visual data.ACKNOWLEDGEMENTSWe appreciate the help from the DEC on campusresearch center (CEC) for the initial data acquisition. Thisresearch is sponsored in part by the Land Baden Württem-berg (Landesschwerpunktprogramm Neuroinformatik),and the National Science Foundation.REFERENCES[1] Christian Benoit, Tahar Lallouache, Tayeb Moham-adi, and Christian Abry. A Set of French Visemes forVisual Speech Synthesis.Talking Machines: Theo-ries, Models, and Designs, 1992.[2] M.E. Demorest and L.E. Bernstein. ComputationalExplorations of Speechreading.In Submission.[3] B. Dodd and R. Campbell. Hearing by Eye: The Psy-chology fo Lipreading.Lawrence Erlbaum Press,1987.[4] C.G. Fischer. Confusion among visually perceivedconsonants.J. Speech Hearing Res., 11, 1968.[5] P. Haffner and A. Waibel. Multi-State Time DelayNeural Networks for Continuous Speech Recogni-tion. In Neural Information Processing Systems(NIPS 4). Morgan Kaufmann, April 1992.[6] H. Hild and A. Waibel. Connected Letter Recogni-tion with a Multi-State Time Delay Neural Network.To appear inNeural Information Processing Systems(NIPS 5).[7] K. Mase and A. Pentland. LIP READING: Auto-matic Visual Recognition of Spoken Words.Proc.Image Understanding and Machine Vision, OpticalSociety of America, June 1989.[8] H. Ney. The Use of a One-Stage Dynamic Program-ming Algorithm for Connected Word Recognition.IEEE International Conference on Acoustics,Speech, and Signal Processing, April 1984.[9] E. Petajan, B. Bischoff, D. Bodoff, and N.M. Brooke.An Improved Automatic Lipreading System toenhance Speech Recognition. InACM SIGCHI,1988.[10] D.A. Pomerleau. Neural Network Perception forMobile Robot Guidance. PhD Thesis, CMU.CMU-CS-92-115, February 1992.[11] P.W. Rander. Facetracking Using a Template BasedApproach.Personal Communication.[12] D.E. Rumelhart, G.E. Hinton, and R.J. Williams.Learning Internal Representations by Error Propaga-tion. Parallel Distributed Processing Vol. 1. MITPress, 1986.[13] David G. Stork, Greg Wolff, and Earl Levine. NeuralNetwork Lipreading System for Improved SpeechRecognition. InIJCNN, June 1992.[14] A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, andK. Lang. Phoneme Recognition Using Time-DelayNeural Networks.IEEE Transactions on Acoustics,Speech, and Signal Processing, 37(3):328-339,March 1989.[15] B.P. Yuhas, M.H. Goldstein, and T.J. Sejnowski.Integration of Acoustic and Visual Speech Signalsusing Neural Networks.IEEE CommunicationsMagazine,[16] John B. Hampshire II and Alexander H. Waibel. ANovel Objective Function for Improved PhonemeRecognition Using Time-Delay Neural Networks.IEEE Transactions on Neural Networks, 1(2), June1990.classify /b/ and /p/ based only on visual information wouldlead to recognition rates not better than guessing, or thenet perhaps would get sensitive for features which areuncorelated to the produced speech. This leads to thedesign of a smaller set of visual distinguishable units inspeech, so called “visemes”. We investigate a new set of42 visemes and a 1-to-n mapping from the viseme set tothe phoneme set. The mapping is necessary for the com-bined layer, in order to calculate the combined acousticand visual hyphotheses for the DTW layer. For examplethe hypotheses for /b/ and /p/ are built out of the sameviseme /b_or_p/ but the different phonemes /b/ and /p/respectly.5. SIMULATION RESULTSOur database consists of 114 and 350 letter sequencesspelled by two male speakers. They consist of names andrandom sequences. The first data set was split into 75training and 39 test sequences (speaker msm). The seconddata set was split into 200 training and 150 test sequences(speaker mcb).Best results were achieved with 15 hidden units in theacoustic subnet and 7 hidden units in the visual subnet.Obviously visual speech data contains less informationthan acoustic data. Therefore better generalization wasachieved with as little as 7 hidden units.Backpropagation was applied with a learning rate of0.05 and momentum of 0.5. We applied different errorfunctions to compute the error derivatives. For bootstrap-ping the McClelland error measure was applied, and forthe global training on letter targets the Classification Fig-ure of Merit [16] was applied.Table 1 summarizes the recognition performanceresults on the sentence level. Errors are misclassifiedwords, insertion, and deletion errors. For speaker “msm”,we get an error reduction on clean data from 11.2%(acoustic only) down to 6.8% with additional visual data.With noise added to the acoustic data, the error rate was52.8%, and could be reduced down to 24.4% with lipread-Acoustic Visual Combinedmsm/clean 88.8% 31.6% 93.2%msm/noisy 47.2% 31.6% 75.6%mcb/clean 97.0% 46.9% 97.2%mcb/noisy 59.0% 46.9% 69.6%Table 1: Results in word accuracy (words correctminus insertion and deletion errors)ing, which means an error reduction to less than half of thepure acoustic recognition. For speaker “mcb”, we couldnot get the same error reduction. Obviously the pronuncia-tion of speaker “mcb” was better, but doing that, he wasnot moving his lips so much.It also should be noted that in the pure visual recogni-tion a lot of the errors are caused by insertion and deletionerrors. When we presented the letters with known bound-aries, we came to visual recognition rates of up to 50.2%.The results of table 1 were achieved with histogram-nor-malized grey-value images. Experiments with 2D-FFTimages are still in progress. In our initial 2D-FFT simula-tions we come to visual recognition errors, which are onaverage about 8% higher than the grey-level coding recog-nition errors.We also took a closer look to the dynamic behavior ofthe entropy-weights. Figure 3 shows the weights from theacoustic and visual TDNN to the combined layer over timeduring the letter sequence M-I-E was spoken. The upperdots represent the acoustic weight A and the lower dots thevisual weight V, whereA=0.5+ (entropy(Visual-TDNN)-entropy(Acoustic-TDNN))/2KandV=1.0-A.Big white dots represent weights close to 1.0 and bigblack dots weights close to 0.0. K is the maximum entropydifference in the training set. At the end of the /m/-pho-neme when the lips are closed, V is higher than A. Obvi-ously there the visual hypotheses are more certain than theacoustic ones. During the /ie/-phoneme the acoustichypotheses are more certain than the visual ones, whichalso makes sense.6. RELATED WORKThe interest in automated speechreading (or lipreading) isgrowing recently. As a non-connectionistic approach thework of Petajan et al. [9] should be mentioned. Yuhas et al.[15] did use a neural network for vowel recognition, work-ing on static images. Stork et al. [13] used a conventionalTDNN (without DTW) for speechreading. They limited/ae/ /m//#//ie//#//eh/Figure 3: Entropy-WeightsAVAVdetection, some “hard decisions” are made, which mayhide useful information for the later learning scheme. Infact it has been reported [10] that such edge detectors arelearned automatically in cases were it is necessary.We apply two alternative preprocessing techniques:Histogram normalized grey-value coding, or 2 dimen-sional Fourier transformation. In both cases we just con-sider an area of interest (AOI) centered around the lips,and low pass filter these AOIs. The AOIs were initiallysegmented by hand, but an automatic procedure is nowalso available [11].Grey-Value coding:We found that a 24x16 pixel reso-lution is enough to recognize lip shapes and movements(Figure 1). Each of these AOI pixels is the average grey-value of a small square in the original image (low pass fil-ter). The grey-levels are rescaled in such a way that thedarkest/brightest 5% in the histogram are coded with -1.0/1.0. The remaining 90% are scaled linear between -1.0 and1.0.2D-FFT: The AOI is rescaled to a 64x64 pixel imageso that the 2 dimensional FFT results also with 64x64coefficients. We just consider the log magnitudes of thefirst 13x13 FFT coefficients and rescale them to [-1.0, 1.0].(After multiplying the complex FFT space with a 13x13window and applying the inverse FFT, we could still rec-ognize in the resulting low passed original image the dis-tinct lip shapes and movements.) The motivation forconsidering the FFT is, that this coding is spatial shiftinvariant. It makes the recognition more stable againstinaccurate AOI positioning.3. SYSTEM ARCHITECTUREAs recognition system we use a modular MS-TDNN [6].Figure 2 shows the architecture. The preprocessed acous-tic and visual data are fed into two front-end TDNNs [14],respectively. Each TDNN consists of an input layer, onehidden layer and the phone-state layer. Backpropagationwas applied to train the networks in a bootstrapping phase,to fit phoneme targets.Above the two phone-state layers, the Dynamic TimeWarping algorithm [8] is applied (in the DTW layer) tofind the optimal path of phone-hypotheses for the wordmodels (German alphabet). In the letter layer the activa-Figure 1: Typical AOIstions of the phone-state units along the optimal paths areaccumulated. The highest score of the letter units repre-sents the recognized letter. In a second phase the networksare trained to fit letter targets. The error derivatives arebackpropagated from the letter units through the best pathin the DTW layer down to the front-end TDNNs, ensuringthat the network is optimized for the actual evaluationtask, which is letter and not phoneme recognition. Asbefore, the acoustic and visual subnets are trained individ-ually.In the final “combined mode” of the recognizer, a com-bined phone-state layer is included between the front-endTDNNs and the DTW layer. The activation of each com-bined phone-state unit is the weighted sum of the regard-ing acoustic phone-state unit and visual phone-state unit.We call these weights “entropy-weights”, because theirvalues are proportional to the relative entropy between allacoustic phone-state activations and all visual phone-stateactivations. Hypotheses with higher uncertainty (higherentropy) are weighted lower than hypotheses with loweruncertainty.4. PHONEMES & VISEMESFor the acoustic classification we use a set of 65 phoneme-states (phoneme-to-phoneme transition states included).They represent a reasonable choice of smallest acousticdistinguishable units in German speech, and the TDNNarchitecture is very well suited to be trained as a classifierfor them.For visual features this will be different. Distinctsounds are generated by distinct vocal tract positions, andvoiced/unvoiced excitations. External features of the vocaltract like the lips, part of the tongue and teeth, contributeonly in part to the sound generation. I.e. /b/ and /p/ aregenerated by similar lip-movements, and cannot be distin-guished with pure visual information. Training a TDNN toFigure 2: Neural Network ArchitectureAcoustic TDNN Visual TDNNCombined LayerDTW LayerLetter Hypotheses (26 German Letters)Phoneme/Viseme-State LayerHidden LayerInput Layer(FFT or AOIs)ttIMPROVING CONNECTED LETTER RECOGNITION BY LIPREADINGChristoph Bregler*, Hermann Hild, Stefan Manke, and Alex WaibelABSTRACTIn this paper we show how recognition performance inautomated speech perception can be significantlyimproved by additional Lipreading, so called “Speech-reading”. We show this on an extension of an existingstate-of-the-art speech recognition system, a modular MS-TDNN. The acoustic and visual speech data is preclassi-fied in two separate front-end phoneme TDNNs and com-bined to acoustic-visual hypotheses for the Dynamic TimeWarping algorithm. This is shown on a connected wordrecognition problem, the notoriously difficult letter spell-ing task. With speechreading we could reduce the errorrate up to half of the error rate of the pure acoustic recog-nition.1. INTRODUCTIONAutomated speech perception systems still performpoorly, when it comes to real world applications. Mostapproaches are very sensitive to background noise or failtotally when more than one speaker talks simultaneously(cocktail party effect), as it often happens in offices, cock-pits, outdoors and other real world environments.Humans deal with this distortions in considering addi-tional sources. Very often misclassified acoustic signalscan be corrected with the use of higher level context infor-mation. In recognition systems this is partly covered bylanguage models or grammars. Psychological studies haveshown [3], that on the lower level additional informationcontributes to human hearing as well. Besides the acousticsignal from both ears, visual information, mostly lipmove-ments, are subconsciously involved in the recognition pro-cess. This source is even more important for hearingimpaired people, but also contributes significantly for nor-mal hearing recognition.We investigate this phenomena on the letter spellingtask. No grammars or other higher level information areemployed. If visual information is missing as well, evenhumans perform poorly. Just remember how hard it is torecognize spelled names at the telephone.The spelling task is seen as a connected word recogni-tion problem. As words we take the highly ambiguous 26German letters. A test person in front of a microphone andvideo camera is spelling names and random lettersequences in German. We did not care about high qualityrecordings, we even degraded the acoustic signal with arti-ficial noise to simulate some real world conditions.As speech recognition system we present an extensionof an existing Multi-State Time Delay Neural Networkarchitecture (MS-TDNN) [6] for handling both modalities,acoustic and visual sensor input. It is shown how recogni-tion performance with integrated acoustic and visual infor-mation achieves significant improvements over acousticinput only.2. BIMODAL ACQUISITION AND PRE-PROCESSINGOur recording setup consists of a conventional NTSCcamera and microphone. The video images are grabbed inreal-time (30 fullframes/sec) into our workstation and aresaved as 256x256 pixel images with 8bit grey-level infor-mation per pixel. This squared region covers the full faceof the speaker. In parallel the acoustic data is sampled at a16KHz rate and 12bit resolution. Also timestamps weresaved, because the correct synchronization between audioand video signals is critical for later processing.For acoustic preprocessing we follow the establishedapproach to apply FFT on the Hamming windowed speechdata in order to get 16 Melscale Fourier coefficients at a 10msec frame rate. For visual preprocessing there is still theactive discussion, how much preprocessing heuristics isappropriate before some connectionist classificationschemes are applied to the data. We follow the idea toallow only transformations with fairly low informationreduction. In other preprocessing algorithms like edgeUniversity of KarlsruheDepartment of Computer ScienceAm Fasanengarten 57500 Karlsruhe 1Germanybregler@ira.uka.de, manke@ira.uka.deCarnegie Mellon UniversitySchool of Computer SciencePittsburghPennsylvania 15213U.S.A.hhild@cs.cmu.edu, ahw@cs.cmu.edu* The author is now with International Computer ScienceInstitute, 1947 Center Street, Berkeley, CA 94704

Improving connected letter recognition by lipreading

https://publikationen.bibliothek.kit.edu/366396/1145

Improving connected letter recognition by lipreading

Abstract

Similar works

Full text

Available Versions

KITopen