Effects of phonetic context on audio-visual intelligibility of French.

International audienceBimodal perception leads to better speech understanding than auditory perception alone. We evaluated the overall benefit of lip-reading on natural utterances of French produced by a single speaker. Eighteen French subjects with good audition and vision were administered a closed set identification test of VCVCV nonsense words consisting of three vowels [i, a, y] and six consonants [b, v, z, 3, R, l]. Stimuli were presented under both auditory and audio-visual conditions with white noise added at various signal-to-noise ratios. Identification scores were higher in the bimodal condition than in the auditory-alone condition, especially in situations where acoustic information was reduced. The auditory and audio-visual intelligibility of the three vowels [i, a, y] averaged over the six consonantal contexts was evaluated as well. Two different hierarchies of intelligibility were found. Auditorily, [a] was most intelligible, followed by [i] and then by [y]; whereas visually [y] was most intelligible, followed by [a] and [i]. We also quantified the contextual effects of the three vowels on the auditory and audio-visual intelligibility of the consonants. Both the auditory and the audio-visual intelligibility of surrounding consonants was highest in the [a] context, followed by the [i] context and lastly the [y] context

Selective Visual Masking in Speechreading

Using digital video technology, selective aspects of a face can be masked by identifying the pixels that represent it and then, by adjusting the gray levels, effectively eliminate that facial aspect. In groups of young adults with normal vision and hearing, consonant-viseme recognition was measured far closed sets of vowels-consonant-vowel disyllables. In the first experiment viseme recognition was measured while the tongue and teeth were masked and while the entire mouth was masked. The results showed that masking of the tongue and teeth had little effect on viseme recognition, and when the entire mouth was masked, participants continued to identify consonant visemes with 70% or greater accuracy in the/a/and/o/vowel contexts. In the second experiment, viseme recognition was measured when the upper part of the face and the mouth were masked and when the lower part of the face and the mouth were masked. The results showed that when the mouth and the upper part of the face were masked, performance was poor, but information was available to identify the consonant-viseme/f/. When the mouth and the lower part of the face were masked, viseme recognition was quite poor, but information was available to discriminate the consonant-viseme /p/ from other consonant visemes