Bregler, C.

Hild, Hermann

Manke, Stefan

Waibel, Alex

English

KITopen

Image Understanding and Machine Vision, OpticalSociety of America, June 1989.[8] H. Ney. The Use of a One-Stage Dynamic Program-ming Algorithm for Connected Word Recognition.IEEE International Conference on Acoustics,Speech, and Signal Processing, April 1984.[9] E. Petajan, B. Bischoff, D. Bodoff, and N.M. Brooke.An Improved Automatic Lipreading System toenhance Speech Recognition. InACM SIGCHI,1988.[10] D.A. Pomerleau. Neural Network Perception forMobile Robot Guidance. PhD Thesis, CMU.CMU-CS-92-115, February 1992.[11] P.W. Rander. Facetracking Using a Template BasedApproach.Personal Communication.[12] D.E. Rumelhart, G.E. Hinton, and R.J. Williams.Learning Internal Representations by Error Propaga-tion. Parallel Distributed Processing Vol. 1. MITPress, 1986.[13] David G. Stork, Greg Wolff, and Earl Levine. NeuralNetwork Lipreading System for Improved SpeechRecognition. InIJCNN, June 1992.[14] A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, andK. Lang. Phoneme Recognition Using Time-DelayNeural Networks.IEEE Transactions on Acoustics,Speech, and Signal Processing, 37(3):328-339,March 1989.[15] B.P. Yuhas, M.H. Goldstein, and T.J. Sejnowski.Integration of Acoustic and Visual Speech Signalsusing Neural Networks.IEEE CommunicationsMagazine,[16] John B. Hampshire II and Alexander H. Waibel. ANovel Objective Function for Improved PhonemeRecognition Using Time-Delay Neural Networks.IEEE Transactions on Neural Networks, 1(2), June1990.[17] P. Haffner, M. Frannzini, and A. Waibel. IntegratingTime Alignment and Connectionist Networks forHigh Performance Continuous Speech Recognition.Proc. International Conference on Acoustics,Speech, and Signal Processing, 1991.malized grey-value images. Experiments with 2D-FFTimages are still in progress. In our initial 2D-FFT simula-tions we come to visual recognition errors, which are onaverage about 8% higher than the grey-level coding recog-nition errors.We also took a closer look to the dynamic behavior ofthe entropy-weights. Figure 3 shows the weights from theacoustic and visual TDNN to the combined layer over timeduring the letter sequence M-I-E was spoken. The upperdots represent the acoustic weight A and the lower dots thevisual weight V, whereA=0.5+ (entropy(Visual-TDNN)-entropy(Acoustic-TDNN))/2KandV=1.0-A.Big white dots represent weights close to 1.0 and bigblack dots weights close to 0.0. K is the maximum entropydifference in the training set. At the end of the /m/-pho-neme when the lips are closed, V is higher than A. Obvi-ously there the visual hypotheses are more certain than theacoustic ones. During the /ie/-phoneme the acoustichypotheses are more certain than the visual ones, whichalso makes sense.VI. OTHER APPROACHESThe interest in automated speechreading (or lipreading)is growing recently. As a non-connectionistic approach thework of Petajan et al. [9] should be mentioned. Yuhas et al.[15] did use a neural network for vowel recognition, work-ing on static images. Stork et al. [13] used a conventionalTDNN (without DTW) for speechreading. They limitedthe task to recognize 10 isolated letters and used artificialmarkers on the lips. No visual feature extraction was inte-grated into their model.Also of interest are some psychological studies abouthuman speechreading and their approach to describe thehuman performance. This measurements could also beapplied to the performance analysis of automatedspeechreading systems. Dodd and Campbell [3], andDemorest and Bernstein [2] did some valuable work in thisarea./ae/ /m//#//ie//#//eh/Figure 3: Entropy-WeightsAVAVVII. C ONCLUSION AND FUTURE WORKWe have shown how a state-of-the-art speech recogni-tion system can be improved by considering additionalvisual information for the recognition process. This is truefor optimal recording conditions but even more for non-optimal recording conditions as they usually exist in realworld applications. Experiments were performed on theconnected letter recognition task, but similar results can beexpected for continuous speech recognition as well.Work is in progress to integrate not only the time inde-pendent weight sharing but also position independentweight sharing for the visual TDNN, in order to locate andtrack the lips. We are also on the way to largely increaseour database in order to achieve better recognition ratesand to train speaker independently. Investigations of dif-ferent approaches are still in progress in order to combinevisual and acoustic features and to apply different prepro-cessing to the visual data.ACKNOWLEDGEMENTSWe appreciate the help from the DEC on campusresearch center (CEC) for the initial data acquisition. Thisresearch is sponsored in part by the Land Baden Württem-berg (Landesschwerpunktprogramm Neuroinformatik),and the National Science Foundation.REFERENCES[1] Christian Benoit, Tahar Lallouache, Tayeb Moham-adi, and Christian Abry. A Set of French Visemes forVisual Speech Synthesis.Talking Machines: Theo-ries, Models, and Designs, 1992.[2] M.E. Demorest and L.E. Bernstein. ComputationalExplorations of Speechreading.In Submission.[3] B. Dodd and R. Campbell. Hearing by Eye: The Psy-chology fo Lipreading.Lawrence Erlbaum Press,1987.[4] C.G. Fischer. Confusion among visually perceivedconsonants.J. Speech Hearing Res., 11, 1968.[5] P. Haffner and A. Waibel. Multi-State Time DelayNeural Networks for Continuous Speech Recogni-tion. In Neural Information Processing Systems(NIPS 4). Morgan Kaufmann, April 1992.[6] H. Hild and A. Waibel. Connected Letter Recogni-tion with a Multi-State Time Delay Neural Network.To appear inNeural Information Processing Systems(NIPS 5).[7] K. Mase and A. Pentland. LIP READING: Auto-matic Visual Recognition of Spoken Words.Proc.In a MS-TDNN the hierarchy continues above thephone-state layer with the Multi-State (MS) units [17] orbetter the DTW layer and word layer. In the forward passof the network the DTW layer performs the DynamicTime Warping algorithm [8] with the phoneme hypthesesas input to find the optimal path for the word models (Ger-man alphabet). The activations of the phone-state unitsalong the optimal paths are accumulated in the word layer.The word unit with the highest score represents the recog-nized letter. In a second learning phase (after the TDNNbootstrapping) the networks are trained to fit word targets.The error derivatives are backpropagated from the wordunits through the best path in the DTW layer down to thefront-end TDNNs, ensuring that the network is optimizedfor the actual evaluation task, which is letter and not pho-neme recognition. As before, the acoustic and visual sub-nets are trained individually.In the final “combined mode” of the recognizer, a com-bined phone-state layer is included between the front-endTDNNs and the DTW layer. The activation of each com-bined phone-state unit is the weighted sum of the regard-ing acoustic phone-state unit and visual phone-state unit.We call these weights “entropy-weights”, because theirvalues are proportional to the relative entropy between allacoustic phone-state activations and all visual phone-stateactivations. Hypotheses with higher uncertainty (higherentropy) are weighted lower than hypotheses with loweruncertainty.IV. PHONEME AND VISEME CODINGFor the acoustic classification we use a set of 65 pho-neme-states (phoneme-to-phoneme transition statesincluded). They represent a reasonable choice of smallestacoustic distinguishable units in German speech, and theTDNN architecture is very well suited to be trained as aclassifier for them.For visual features this will be different. Distinct soundsare generated by distinct vocal tract positions, and voiced/unvoiced excitations. External features of the vocal tractlike the lips, part of the tongue and teeth, contribute onlyin part to the sound generation. I.e. /b/ and /p/ are gener-ated by similar lip-movements, and cannot be distin-guished with pure visual information. Training a TDNN toclassify /b/ and /p/ based only on visual information wouldlead to recognition rates not better than guessing, or thenet perhaps would get sensitive for features which areuncorelated to the produced speech. This leads to thedesign of a smaller set of visual distinguishable units inspeech, so called “visemes”. We investigate a new set of42 visemes and a 1-to-n mapping from the viseme set tothe phoneme set. The mapping is necessary for the com-bined layer, in order to calculate the combined acousticand visual hyphotheses for the DTW layer. For examplethe hypotheses for /b/ and /p/ are built out of the sameviseme /b_or_p/ but the different phonemes /b/ and /p/respectly.V. EXPERIMENTSOur database consists of 114 and 350 letter sequencesspelled by two male speakers. They consist of names andrandom sequences. The first data set was split into 75training and 39 test sequences (speaker msm). The seconddata set was split into 200 training and 150 test sequences(speaker mcb).Best results were achieved with 15 hidden units in theacoustic subnet and 7 hidden units in the visual subnet.Obviously visual speech data contains less informationthan acoustic data. Therefore better generalization wasachieved with as little as 7 hidden units.Backpropagation was applied with a learning rate of0.05 and momentum of 0.5. We applied different errorfunctions to compute the error derivatives. For bootstrap-ping the McClelland error measure was applied, and forthe global training on letter targets the Classification Fig-ure of Merit [16] was applied.Table 1 summarizes the recognition performance resultson the sentence level. Errors are misclassified words,insertion, and deletion errors. For speaker “msm”, we getan error reduction on clean data from 11.2% (acousticonly) down to 6.8% with additional visual data. With noiseadded to the acoustic data, the error rate was 52.8%, andcould be reduced down to 24.4% with lipreading, whichmeans an error reduction to less than half of the pureacoustic recognition. For speaker “mcb”, we could not getthe same error reduction. Obviously the pronunciation ofspeaker “mcb” was better, but doing that, he was not mov-ing his lips so much.It also should be noted that in the pure visual recogni-tion a lot of the errors are caused by insertion and deletionerrors. When we presented the letters with known bound-aries, we came to visual recognition rates of up to 50.2%.The results of table 1 were achieved with histogram-nor-Acoustic Visual Combinedmsm/clean 88.8% 31.6% 93.2%msm/noisy 47.2% 31.6% 75.6%mcb/clean 97.0% 46.9% 97.2%mcb/noisy 59.0% 46.9% 69.6%Table 1: Results in word accuracy (words correctminus insertion and deletion errors)reported that necessary techniques like edge detection areautomatically learned by multi-layer perceptrons [10].We apply two alternative preprocessing techniques:Histogram normalized grey-value coding, or 2 dimen-sional Fourier transformation. In both cases we just con-sider an area of interest (AOI) centered around the lips,and low pass filter these AOIs. The AOIs were initiallysegmented by hand, but an automatic procedure is nowalso available1 [11].Grey-Value coding:We found that a 24x16 pixel reso-lution is enough to recognize lip shapes and movements(Figure 1). Each of these AOI pixels is the average grey-value of a small square in the original image (low pass fil-ter). The grey-levels are rescaled in such a way that thedarkest/brightest 5% in the histogram are coded with -1.0/1.0. The remaining 90% are scaled linear between -1.0 and1.0.2D-FFT: The AOI is rescaled to a 64x64 pixel image sothat the 2 dimensional FFT results also with 64x64 coeffi-cients. We just consider the log magnitudes of the first13x13 FFT coefficients and rescale them to [-1.0, 1.0].(After multiplying the complex FFT space with a 13x13window and applying the inverse FFT, we could still rec-ognize in the resulting low passed original image the dis-tinct lip shapes and movements.) The motivation forconsidering the FFT is, that this coding is spatial shiftinvariant. It makes the recognition more stable againstinaccurate AOI positioning.III. T HE MODULAR MS-TDNN ARCHITECTURE Time Delay Neural Networks are very well suited forlow-level phoneme classification [14]. Because they act asgeneral time-shift invariant feature extractors they are notlimited to acoustic speech input only. In our case we wantto process image sequences as well. The extracted visual1.  We don’t want to hide the fact, that the complex task of find-ing and tracking the lips is very critical to the success of our rec-ognizer. The Real Time Face Tracking System we are using rightnow is a template matching based approach and also used in awider context of multi-modal human computer interfaceresearch, were tracking other parts of the face like for examplethe eyes is involved as well. Work is in progess to solve thisproblem with the connectionist architecture as well.Figure 1: Typical AOIsfeatures of the lips change over time in a similar mannerlike the acoustic speech signal, so we use for the visualclassification a similar TDNN architecture. Figure 2 showsthe architecture. The preprocessed acoustic and visual dataare fed into two front-end TDNNs [14], respectively. EachTDNN consists of an input layer, one hidden layer and thephone-state layer. Backpropagation was applied to trainthe networks in a bootstrapping phase, to fit phoneme tar-gets.AcousticPreproc. FindLip-AOIHistogr.Normaliz.2D-FFTORFFTAcoustic Input Visual InputHidden Units Hidden UnitsPhonemeHypothesesVisemeHypothesesCombinedHypthesesDTWLayerEn-tropyWord HypothesesPreprocessingTDNNFigure 2: Neural Network ArchitectureBIMODAL SENSOR INTEGRATION ON THE EXAMPLE OF“SPEECH-READING”Abstract -- In this paper we show how recognition perfor-mance in automated speech perception can be significantlyimproved by additional Lipreading, so called “speech-read-ing”. We show this on an extension of an existing state-of-the-art speech recognition system, a modular MS-TDNN. Theacoustic and visual speech data is preclassified in two sepa-rate front-end phoneme TDNNs and combined to acoustic-visual hypotheses for the Dynamic Time Warping algorithm.This is shown on a connected word recognition problem, thenotoriously difficult letter spelling task. With speech-readingwe could reduce the error rate up to half of the error rate ofthe pure acoustic recognition.I. INTRODUCTIONRecent development in the design of human computerinterfaces (HCI) requests a new field of research: Multi-Modal Recognition. There are strong research efforts in allkind of alternative computer input methods on the way,like speech input, or visual input, but usually all differentrecognition methods are treated separately. When it comesto real-world-applications, single recognition methodshave poor recognition performance or are limited to verysmall domains. Besides considering a lot of backgroundknowledge, humans come to much better recognition per-formance in combining or substituting the different inputsources itself in a flexible way. In speech recognition themain source is the acoustic signal, but if the signal is oflow quality or ambiguous, visual information, i.e. lip-movements can contribute to the recognition process aswell. In fact this is a well studied phenomena in humanpsychology often called “speech-reading”. Research withhearing impaired people and also normal hearing peoplehas shown, that lipreading is subconsciously involved intohuman perception in a significant manner [3].We investigate this phenomena on a state-of-the-artspeech recognition architecture, a so called Multi StateTime-Delay Neural Network (MS-TDNN [6]). Weextended the architecture in a way, that besides the acous-tic speech signals also visual signals (i.e. lipmovements)are integrated in the learning and recognition process. Aspeaker is set in front of a microphone and a camera, spell-ing German names and random letter sequences. Thespelling task is seen as a connected word recognition prob-lem. As words we take the highly ambiguous 26 Germanletters. We do not care about high quality recordings, weeven degrade the acoustic signal with artificial noise tosimulate real-world conditions. The camera covers the fullface, but as network input we just consider an area ofinterest (AOI) centered around the lips. We show how rec-ognition performance with additional visual input furtherimproves over acoustic recognition only.II. ACOUSTICAND VISUAL INPUTREPRESENTATIONOur bimodal input devices are a microphone and a con-ventional NTSC camera. The acoustic signal is sampledwith a 16KHz rate and 12 bit accuracy. Every 10ms wecompute 16 Melscale Fourier coefficients on the HammingWindowed speech samples as input for the later classifier.Due to the NTSC standard, the video images are sam-pled with a much lower frequency. We digitize roughlyevery 33 ms a 256x256 pixel area covering the face of thespeaker. We use 8bit gray-value coding for each pixel.Also timestamps were saved, because the correct synchro-nization between audio and video signals is critical for thelater classification.In vision there does not exist such a well establishedbody of preprocessing techniques like in speech recogni-tion. Basically preprocessing for connectionist vision issplit in two different “schools of believes”: 1. Apply asmuch as possible preprocessing in a manner, that just somefew characteristic features are fed into the classifier. 2.Apply only transformations with very little informationreduction and let the learning scheme find out the charac-teristic features. Edge detection would be an example forthe first case and 2D-FFT for the second case.We follow the second approach, because we believe thatconventional image processing techniques may cut out inan early stage important features, which could be usefulfor the later global learning scheme. In fact it has beenChristoph Bregler*, Stefan MankeUniversity of KarlsruheDepartment of Computer ScienceAm Fasanengarten 57500 Karlsruhe 1Germanybregler@icsi.berkeley.edu, manke@ira.uka.deHermann Hild, Alex WaibelCarnegie Mellon UniversitySchool of Computer SciencePittsburghPennsylvania 15213U.S.A.hhild@cs.cmu.edu, ahw@cs.cmu.edu* The author is now with International Computer ScienceInstitute, 1947 Center Street, Berkeley, CA 94704

Bimodal sensor integration on the example of speech-reading

https://publikationen.bibliothek.kit.edu/366496/1146

Bimodal sensor integration on the example of speech-reading

Abstract

Similar works

Full text

Available Versions

KITopen