The role of visual cues in speech comprehension: evidence from brain and behaviour

Face-to-face communication is multimodal. It comprises a plethora of linguistic and non-linguistic cues, such as gestures, face and body movements, eye gaze, and prosody, that modulate language processing. However, these cues have been primarily studied in isolation and it remains unclear whether they interact and, if so, how they affect speech comprehension. This thesis aims to assess the role and the relationship of two multimodal cues: co-speech gestures and mouth movements. The claims presented in this thesis are supported by findings from studies on neurotypical adults and people with post-stroke aphasia. First, I developed a novel quantification of how informative mouth and facial movements are and provided informativeness norms for more than 2,000 English words. Such quantification allows studying the effects of speech and gestures while controlling for the invariably present mouth movements. Second, I investigated the impact of iconic gestures (i.e., gestures referring to features and properties of objects or actions in an imagistic way) and mouth informativeness on word comprehension under clear and challenging listening conditions. I found that both cues contribute to speech comprehension and that listeners dynamically process visual cues depending on their informativeness and listening conditions. Next, I showed that gestures and mouth movements also benefit discourse comprehension, but they do so differently: Listeners always take gestures into account, whereas mouth movements are only beneficial in challenging listening conditions. Finally, I investigated the impact of mouth movements on speech comprehension in aphasia. I showed that people with aphasia benefit less from mouth movements than neurotypical adults and that this effect is contingent upon lesion locations. This thesis brings together research on gestures and mouth movements and demonstrates that visual cues benefit speech comprehension interactively and dynamically depending on cue informativeness, listening conditions, and neuroanatomical profiles of people with aphasia