The Neurobiology of Audiovisual Integration: A Voxel-Based Lesion Symptom Mapping Study

abstract: Audiovisual (AV) integration is a fundamental component of face-to-face communication. Visual cues generally aid auditory comprehension of communicative intent through our innate ability to “fuse” auditory and visual information. However, our ability for multisensory integration can be affected by damage to the brain. Previous neuroimaging studies have indicated the superior temporal sulcus (STS) as the center for AV integration, while others suggest inferior frontal and motor regions. However, few studies have analyzed the effect of stroke or other brain damage on multisensory integration in humans. The present study examines the effect of lesion location on auditory and AV speech perception through behavioral and structural imaging methodologies in 41 left-hemisphere participants with chronic focal cerebral damage. Participants completed two behavioral tasks of speech perception: an auditory speech perception task and a classic McGurk paradigm measuring congruent (auditory and visual stimuli match) and incongruent (auditory and visual stimuli do not match, creating a “fused” percept of a novel stimulus) AV speech perception. Overall, participants performed well above chance on both tasks. Voxel-based lesion symptom mapping (VLSM) across all 41 participants identified several regions as critical for speech perception depending on trial type. Heschl’s gyrus and the supramarginal gyrus were identified as critical for auditory speech perception, the basal ganglia was critical for speech perception in AV congruent trials, and the middle temporal gyrus/STS were critical in AV incongruent trials. VLSM analyses of the AV incongruent trials were used to further clarify the origin of “errors”, i.e. lack of fusion. Auditory capture (auditory stimulus) responses were attributed to visual processing deficits caused by lesions in the posterior temporal lobe, whereas visual capture (visual stimulus) responses were attributed to lesions in the anterior temporal cortex, including the temporal pole, which is widely considered to be an amodal semantic hub. The implication of anterior temporal regions in AV integration is novel and warrants further study. The behavioral and VLSM results are discussed in relation to previous neuroimaging and case-study evidence; broadly, our findings coincide with previous work indicating that multisensory superior temporal cortex, not frontal motor circuits, are critical for AV integration.Dissertation/ThesisMasters Thesis Communication Disorders 201

Auditory-Visual Integration during the Perception of Spoken Arabic

This thesis aimed to investigate the effect of visual speech cues on auditory-visual integration during speech perception in Arabic. Four experiments were conducted two of which were cross linguistic studies using Arabic and English listeners. To compare the influence of visual speech in Arabic and English listeners chapter 3 investigated the use of visual components of auditory-visual stimuli in native versus non-native speech using the McGurk effect. The experiment suggested that Arabic listeners’ speech perception was influenced by visual components of speech to a lesser degree compared to English listeners. Furthermore, auditory and visual assimilation was observed for non-native speech cues. Additionally when the visual cue was an emphatic phoneme the Arabic listeners incorporated the emphatic visual cue in their McGurk response. Chapter 4, investigated whether the lower McGurk effect response in Arabic listeners found in chapter 3 was due to a bottom-up mechanism of visual processing speed. Chapter 4, using auditory-visual temporal asynchronous conditions, concluded that the differences in McGurk response percentage was not due to bottom-up mechanism of visual processing speed. This led to the question of whether the difference in auditory-visual integration of speech could be due to more ambiguous visual cues in Arabic compared to English. To explore this question it was first necessary to identify visemes in Arabic. Chapter 5 identified 13 viseme categories in Arabic, some emphatic visemes were visually distinct from their non-emphatic counterparts and a greater number of phonemes within the guttural viseme category were found compared to English. Chapter 6 evaluated the visual speech influence across the 13 viseme categories in Arabic measured by the McGurk effect. It was concluded that the predictive power of visual cues and the contrast between visual and auditory speech components will lead to an increase in the McGurk response percentage in Arabic

Contributions of visual speech, visual distractors, and cognition to speech perception in noise for younger and older adults

Older adults report that understanding speech in noisy situations (e.g., a restaurant) is difficult. Repeated experiences of frustration in noisy situations may cause older adults to withdraw socially, increasing their susceptibility to mental and physical illness. Understanding the factors that contribute to older adults’ difficulty in noise, and in turn, what might be able to alleviate this difficulty, is therefore an important area of research. The experiments in this thesis investigated how sensory and cognitive factors, in particular attention, affect older and younger adults’ ability to understand speech in noise. First, the performance of older as well as younger adults on a standardised speech perception in noise task and on a series of cognitive and hearing tasks was assessed. A correlational analysis indicated that there was no reliable association between pure-tone audiometry and speech perception in noise performance but that there was some evidence of an association between auditory attention and speech perception in noise performance for older adults. Next, a series of experiments were conducted that aimed to investigate the role of attention in gaining a visual speech benefit in noise. These auditory-visual experiments were largely motivated by the idea that as the visual speech benefit is the largest benefit available to listeners in noisy situations, any reduction in this benefit, particularly for older adults, could exacerbate difficulties understanding speech in noise. For the first auditory-visual experiments, whether increasing the number of visual distractors displayed affected the visual speech benefit in noise for younger and older adults when the SNR was -6dB (Experiment 1) and when the SNR was -1dB (Experiment 2) was tested. For both SNRs, the magnitude of older adults’ visual speech benefit reduced by approximately 50% each time an additional visual distractor was presented. Younger adults showed the same pattern when the SNR was - 6dB, but unlike older adults, were able to get a full visual speech benefit when one distractor was presented and the SNR was -1dB. As discussed in Chapter 3, a possible interpretation of these results is that combining auditory and visual speech requires attentional resources. To follow up the finding that visual distractors had a detrimental impact on the visual speech benefit, particularly for older adults, the experiment in Chapter 4 tested whether presenting a salient visual cue that indicated the location of the target talker would help older adults get a visual speech benefit. The results showed that older adults did not benefit from the cue, whereas younger adults did. As older adults should have had sufficient time to switch their gaze and/or attention to the location of the target talker, the failure to find a cueing effect suggests that age related declines in inhibition likely affected older adults’ ability to ignore the visual distractor. The final experiment tested whether the visual speech benefit and the visual distraction effect found for older adults in Chapter 4 transferred to a conversationcomprehension style task (i.e., The Question-and-Answer Task). The results showed that younger and older adults’ performance improved on an auditory-visual condition in comparison to an auditory-only condition and that this benefit did not reduce when a visual distractor was presented. To explain the absence of a distraction effect, several properties of the visual distractor presented were discussed. Together, the experiments in this thesis suggest that the roles of attention and visual distraction should be considered when trying to understand the communication difficulties that older adults experience in noisy situations

Early multisensory attention as a foundation for learning in multicultural Switzerland

Traditional laboratory research on visual attentional control has largely focused on adults, treated one sensory modality at a time, and neglected factors that are a constituent part of information processing in real-world contexts. Links between visual-only attentional control and children’s educational skills have emerged, but they still do not provide enough information about school learning. The present thesis addressed these gaps in knowledge through the following aims: 1) to shed light on the development of the neuro-cognitive mechanisms of attention engaged by multisensory objects in a bottom-up fashion, together with attentional control over visual objects in a top-down fashion, 2) to investigate the links between developing visual and multisensory attentional control and children’s basic literacy and numeracy attainment, and 3) to explore how contextual factors, such as the temporal predictability of a stimulus or the semantic relationships between stimulus features, further influence attentional control mechanisms. To investigate these aims, 115 primary school children and 39 adults from the French-speaking part of Switzerland were tested on their behavioural performance on a child-friendly, multisensory version of the Folk et al. (1992) spatial cueing paradigm, while 129-channel EEG was recorded. EEG data were analysed in a traditional framework (the N2pc ERP component) and a multivariate Electrical Neuroimaging (EN) framework. Taken together, our results demonstrated that children’s visual attentional control reaches adult-like levels at around 7 years of age, or 3rd grade, although children as young as 5 (at school entry) may already be sensitive to the goal- relevance of visual objects. Multisensory attentional control may develop only later. Namely, while 7-year-old children (3rd grade) can be sensitive to the multisensory nature of objects, such sensitivity may only reach an adult-like state at 9 years of age (5th grade). As revealed by EN, both bottom-up multisensory control of attention and top-down visual control of attention are supported by the recruitment of distinct networks of brain generators at each level of schooling experience. Further, at each level of schooling, the involvement of specific sets of brain generators was correlated with literacy and numeracy attainment. In adults, visual and multisensory attentional control were further jointly influenced by contextual factors. The semantic relationship between stimulus features directly influenced visual and multisensory attentional control. In the absence of such semantic links, however, it was the predictability of stimulus onset that influenced visual and multisensory attentional control. Throughout this work, the N2pc component was not sensitive to multisensory or contextual effects in adults, or even traditional visual attention effects in children, and it was owing to EN that the mechanisms of visual and multisensory attentional control were clarified. The present thesis demonstrates the strength of combining behavioural and EEG/ERP markers of attentional control with advanced EEG analytical techniques for investigating the development of attentional control in settings that closely approximate those that we encounter in everyday life

The Biolinguistics of Autism: Emergent Perspectives

This contribution attempts to import the study of autism into the biolinguistics program by reviewing the current state of knowledge on its neurobiology, physiology and verbal phenotypes from a comparative vantage point. A closer look at alternative approaches to the primacy of social cognition impairments in autism spectrum disorders suggests fundamental differences in every aspect of language comprehension and production, suggesting productive directions of research in auditory and visual speech processing as well as executive control. Strong emphasis is put on the great heterogeneity of autism phenotypes, raising important caveats towards an all-or-nothing classification of autism. The study of autism brings interesting clues about the nature and evolution of language, in particular its ontological connections with musical and visual perception as well as executive functions and generativity. Success in this endeavor hinges upon expanding beyond the received wisdom of autism as a purely social disorder and favoring a “cognitive style” approach increasingly called for both inside and outside the autistic community

Audiovisual Integration in Apraxia of Speech: EEG Evidence for Processing Differences

Speech perception is a unique audiovisual experience in part because timing of the speech signal is influenced by simultaneous overlapping gestures in coarticulation. Apraxia of speech (AOS) is a motor planning disorder that impairs coarticulation. Imaging studies show that brain regions damaged in AOS are critical to audiovisual speech perception. Although AOS is a motor planning disorder, individuals with AOS may have a disruption to the perceptual system for speech gestures. To evaluate this hypothesis we investigated audiovisual mismatch negativity (MMN) brain responses in adults with damage to Broca’s area (n =5) compared to a healthy age-matched comparison group (n = 5). We utilized the McGurk effect, in which incongruent auditory and visual information alters perception. Participants viewed videos of a speaker articulating the syllable /ba/ (standard) for 80% trials and /ga/ (deviant) for 20% of the trials while the auditory stimulus /ba/ remained consistent throughout. Responses to this McGurk audiovisual condition were compared to an inverse McGurk audiovisual condition in which the visual stimulus remained constant while the auditory stimulus changed, and a visual-only condition without sound to control for evoked activity from changes to the visual stimulus. Incongruent McGurk deviants elicited an MMN over left hemisphere electrodes in the comparison group, while the AOS group exhibited a later, attention-based response, a P300. The AOS group similarly responded to inverse McGurk deviants, which do not require fusion of the percept, with a P300 response, indicating that auditory and visual aspects of the incongruent McGurk deviants were not integrated. In the visual-only control condition, the AOS group showed a left-lateralized MMN, suggesting greater influence of visual processing when confronted with conflicting multisensory information compared to the comparison group. Overall, the comparison group’s responses were indicative of early and automatic audiovisual integration of incongruent McGurk percepts while the responses of the AOS group showed contributions of both attentional and visual processing. The timing of the response in the AOS group was correlated with speech production characteristics of apraxia, as well as performance on taxing motor speech tasks. Results of this study support the hypothesis that AOS is a disorder beyond motor planning, with implications for higher-level linguistic and cognitive systems

Audiovisual processing in aphasic and non-brain-damaged listeners:the whole is more than the sum of its parts

Spraakverwerking is een taak, die (meestal) zonder veel moeite gedaan wordt. Slechts als de verwerking verstoord is, bijvoorbeeld als gevolg van hersenletsel, merken we de complexiteit ervan op. Dörte Hessler deed onderzoek naar dit fenomeen. Niet alleen auditieve, maar ook audiovisuele verwerking van klanken komt aan bod. Uit het onderzoek kwam allereerst naar voren dat mensen met een afasie (een taalstoornis die optreedt als gevolg van hersenletsel) meer moeite hebben met het herkennen van kleine dan van grote klankverschillen. Klanken kunnen bijvoorbeeld verschillen in de manier waarop de klank wordt gemaakt, de plaats waar dat gebeurt en het feit of de stembanden gaan trillen bij een klank. Klanken die op al deze drie onderdelen van elkaar verschilden bleken eenvoudiger te herkennen dan klanken die maar op één onderdeel verschilden. Het lastigste onderscheid was te maken bij klanken die alleen verschilden in het al of niet laten trillen van de stembanden (bijvoorbeeld het verschil tussen p of b). Hersenreacties van luisteraars zonder taalproblemen lieten in het verlengde hiervan zien hersengolven een sterkere reactie vertonen wanneer de verschillen tussen klanken klein zijn. Dit hangt waarschijnlijk samen met de extra aandacht die nodig is om deze kleinere verschillen te verwerken. Het onderzoek wees verder uit dat visuele ondersteuning (liplezen), die een positieve invloed heeft op de spraakverwerking, zich niet beperkt tot hele duidelijke herkenbare klankkenmerken, zoals de plaats van uitspraak, maar ook op de manier van uitspreken en de stembandtrilling. Ook personen zonder hersenbeschadiging tonen een effect van liplezen: hun reactietijden dalen, als ze een doelklank moeten kiezen. Verder werden ook hun hersenreacties beïnvloed: auditieve en audiovisuele input leidden tot duidelijke verschillen in reactiepatronen. Verwerking was eenvoudiger bij een audiovisueel aan bod van een klank

Cross-modal Effects In Tactile And Visual Signaling

Using a wearable tactile display three experiments were conducted in which tactile messages were created emulating five standard US Army and Marine arm and hand signals for the military commands, namely: Attention , Halt , Rally , Move Out , and Nuclear Biological or Chemical event (NBC) . Response times and accuracy rates were collected for novices responding to visual and tactile representations of these messages, which were displayed either alone or together in congruent or incongruent combinations. Results indicated synergistic effects for concurrent, congruent message presentations showing superior response times when compared to individual presentations in either modality alone. This effect was mediated by participant strategy. Accuracy similarly improved when both the tactile and visual presentation were concurrently displayed as opposed to separately. In a low workload condition, participants could largely attend to a particular modality, with little interference from competing signals. If participants were not given instructions as to which modality to attend to, participants chose that modality which was received first. Lastly, initial learning and subsequent training of intuitive tactile signals occurred rapidly with large gains in performance in short training periods. These results confirm the promise for tactile messages to augment visual messaging in challenging and stressful environments particularly when visual messaging is maybe preferred but is not always feasible or possible