Investigating the Neural Basis of Audiovisual Speech Perception with Intracranial Recordings in Humans

Speech is inherently multisensory, containing auditory information from the voice and visual information from the mouth movements of the talker. Hearing the voice is usually sufficient to understand speech, however in noisy environments or when audition is impaired due to aging or disabilities, seeing mouth movements greatly improves speech perception. Although behavioral studies have well established this perceptual benefit, it is still not clear how the brain processes visual information from mouth movements to improve speech perception. To clarify this issue, I studied the neural activity recorded from the brain surfaces of human subjects using intracranial electrodes, a technique known as electrocorticography (ECoG). First, I studied responses to noisy speech in the auditory cortex, specifically in the superior temporal gyrus (STG). Previous studies identified the anterior parts of the STG as unisensory, responding only to auditory stimulus. On the other hand, posterior parts of the STG are known to be multisensory, responding to both auditory and visual stimuli, which makes it a key region for audiovisual speech perception. I examined how these different parts of the STG respond to clear versus noisy speech. I found that noisy speech decreased the amplitude and increased the across-trial variability of the response in the anterior STG. However, possibly due to its multisensory composition, posterior STG was not as sensitive to auditory noise as the anterior STG and responded similarly to clear and noisy speech. I also found that these two response patterns in the STG were separated by a sharp boundary demarcated by the posterior-most portion of the Heschl’s gyrus. Second, I studied responses to silent speech in the visual cortex. Previous studies demonstrated that visual cortex shows response enhancement when the auditory component of speech is noisy or absent, however it was not clear which regions of the visual cortex specifically show this response enhancement and whether this response enhancement is a result of top-down modulation from a higher region. To test this, I first mapped the receptive fields of different regions in the visual cortex and then measured their responses to visual (silent) and audiovisual speech stimuli. I found that visual regions that have central receptive fields show greater response enhancement to visual speech, possibly because these regions receive more visual information from mouth movements. I found similar response enhancement to visual speech in frontal cortex, specifically in the inferior frontal gyrus, premotor and dorsolateral prefrontal cortices, which have been implicated in speech reading in previous studies. I showed that these frontal regions display strong functional connectivity with visual regions that have central receptive fields during speech perception

The Neurobiology of Audiovisual Integration: A Voxel-Based Lesion Symptom Mapping Study

abstract: Audiovisual (AV) integration is a fundamental component of face-to-face communication. Visual cues generally aid auditory comprehension of communicative intent through our innate ability to “fuse” auditory and visual information. However, our ability for multisensory integration can be affected by damage to the brain. Previous neuroimaging studies have indicated the superior temporal sulcus (STS) as the center for AV integration, while others suggest inferior frontal and motor regions. However, few studies have analyzed the effect of stroke or other brain damage on multisensory integration in humans. The present study examines the effect of lesion location on auditory and AV speech perception through behavioral and structural imaging methodologies in 41 left-hemisphere participants with chronic focal cerebral damage. Participants completed two behavioral tasks of speech perception: an auditory speech perception task and a classic McGurk paradigm measuring congruent (auditory and visual stimuli match) and incongruent (auditory and visual stimuli do not match, creating a “fused” percept of a novel stimulus) AV speech perception. Overall, participants performed well above chance on both tasks. Voxel-based lesion symptom mapping (VLSM) across all 41 participants identified several regions as critical for speech perception depending on trial type. Heschl’s gyrus and the supramarginal gyrus were identified as critical for auditory speech perception, the basal ganglia was critical for speech perception in AV congruent trials, and the middle temporal gyrus/STS were critical in AV incongruent trials. VLSM analyses of the AV incongruent trials were used to further clarify the origin of “errors”, i.e. lack of fusion. Auditory capture (auditory stimulus) responses were attributed to visual processing deficits caused by lesions in the posterior temporal lobe, whereas visual capture (visual stimulus) responses were attributed to lesions in the anterior temporal cortex, including the temporal pole, which is widely considered to be an amodal semantic hub. The implication of anterior temporal regions in AV integration is novel and warrants further study. The behavioral and VLSM results are discussed in relation to previous neuroimaging and case-study evidence; broadly, our findings coincide with previous work indicating that multisensory superior temporal cortex, not frontal motor circuits, are critical for AV integration.Dissertation/ThesisMasters Thesis Communication Disorders 201

Brain activity during traditional textbook and audiovisual-3D learning

Introduction: Audiovisual educational tools have increasingly been used during the past years to complement and compete with traditional textbooks. However, little is known as to how the brain processes didactic information presented in different formats. We directly assessed brain activity during learning using both traditional textbook and audiovisual-3D material. Methods: A homogeneous sample of 30 young adults with active study habits was assessed. Educational material on the subject of Cardiology was adapted to be presented during the acquisition of functional MRI. Results: When tested after image acquisition, participants obtained similar examination scores for both formats. Evoked brain activity was robust during both traditional textbook and audiovisual-3D lessons, but a greater number of brain systems were implicated in the processing of audiovisual-3D information, consistent with its multisource sensory nature. However, learning was not associated with group mean brain activations, but was instead predicted by distinct functional MRI signal changes in the frontal lobes and showed distinct cognitive correlates. In the audiovisual-3D version, examination scores were positively correlated with late-evoked prefrontal cortex activity and working memory, and negatively correlated with language-related frontal areas and verbal memory. As for the traditional textbook version, the fewer results obtained suggested the opposite pattern, with examination scores negatively correlating with prefrontal cortex activity evoked during the lesson. Conclusions: Overall, the results indicate that a similar level of knowledge may be achieved via different cognitive strategies. In our experiment, audiovisual learning appeared to benefit from prefrontal executive resources (as opposed to memorizing verbal information) more than traditional textbook learning

Inferior frontal oscillations reveal visuo-motor matching for actions and speech: evidence from human intracranial recordings.

The neural correspondence between the systems responsible for the execution and recognition of actions has been suggested both in humans and non-human primates. Apart from being a key region of this visuo-motor observation-execution matching (OEM) system, the human inferior frontal gyrus (IFG) is also important for speech production. The functional overlap of visuo-motor OEM and speech, together with the phylogenetic history of the IFG as a motor area, has led to the idea that speech function has evolved from pre-existing motor systems and to the hypothesis that an OEM system may exist also for speech. However, visuo-motor OEM and speech OEM have never been compared directly. We used electrocorticography to analyze oscillations recorded from intracranial electrodes in human fronto-parieto-temporal cortex during visuo-motor (executing or visually observing an action) and speech OEM tasks (verbally describing an action using the first or third person pronoun). The results show that neural activity related to visuo-motor OEM is widespread in the frontal, parietal, and temporal regions. Speech OEM also elicited widespread responses partly overlapping with visuo-motor OEM sites (bilaterally), including frontal, parietal, and temporal regions. Interestingly a more focal region, the inferior frontal gyrus (bilaterally), showed both visuo-motor OEM and speech OEM properties independent of orolingual speech-unrelated movements. Building on the methodological advantages in human invasive electrocorticography, the present findings provide highly precise spatial and temporal information to support the existence of a modality-independent action representation system in the human brain that is shared between systems for performing, interpreting and describing actions

Left-handed musicians show a higher probability of atypical cerebral dominance for language

Music processing and right hemispheric language lateralization share a common network in the right auditory cortex and its frontal connections. Given that the development of hemispheric language dominance takes place over several years, this study tested whether musicianship could increase the probability of observing right language dominance in left-handers. Using a classic fMRI language paradigm, results showed that atypical lateralization was more predominant in musicians (40%) than in nonmusicians (5%). Comparison of left-handers with typical left and atypical right lateralization revealed that: (a) atypical cases presented a thicker right pars triangularis and more gyrified left Heschl's gyrus; and (b) the right pars triangularis of atypical cases showed a stronger intra-hemispheric functional connectivity with the right angular gyrus, but a weaker interhemispheric functional connectivity with part of the left Broca's area. Thus, musicianship is the first known factor related to a higher prevalence of atypical language dominance in healthy left-handed individuals. We suggest that differences in the frontal and temporal cortex might act as shared predisposing factors to both musicianship and atypical language lateralization

Neural correlates of the processing of co-speech gestures

In communicative situations, speech is often accompanied by gestures. For example, speakers tend to illustrate certain contents of speech by means of iconic gestures which are hand movements that bear a formal relationship to the contents of speech. The meaning of an iconic gesture is determined both by its form as well as the speech context in which it is performed. Thus, gesture and speech interact in comprehension. Using fMRI, the present study investigated what brain areas are involved in this interaction process. Participants watched videos in which sentences containing an ambiguous word (e.g. She touched the mouse) were accompanied by either a meaningless grooming movement, a gesture supporting the more frequent dominant meaning (e.g. animal) or a gesture supporting the less frequent subordinate meaning (e.g. computer device). We hypothesized that brain areas involved in the interaction of gesture and speech would show greater activation to gesture-supported sentences as compared to sentences accompanied by a meaningless grooming movement. The main results are that when contrasted with grooming, both types of gestures (dominant and subordinate) activated an array of brain regions consisting of the left posterior superior temporal sulcus (STS), the inferior parietal lobule bilaterally and the ventral precentral sulcus bilaterally. Given the crucial role of the STS in audiovisual integration processes, this activation might reflect the interaction between the meaning of gesture and the ambiguous sentence. The activations in inferior frontal and inferior parietal regions may reflect a mechanism of determining the goal of co-speech hand movements through an observation-execution matching process

On the context-dependent nature of the contribution of the ventral premotor cortex to speech perception

What is the nature of the interface between speech perception and production, where auditory and motor representations converge? One set of explanations suggests that during perception, the motor circuits involved in producing a perceived action are in some way enacting the action without actually causing movement (covert simulation) or sending along the motor information to be used to predict its sensory consequences (i.e., efference copy). Other accounts either reject entirely the involvement of motor representations in perception, or explain their role as being more supportive than integral, and not employing the identical circuits used in production. Using fMRI, we investigated whether there are brain regions that are conjointly active for both speech perception and production, and whether these regions are sensitive to articulatory (syllabic) complexity during both processes, which is predicted by a covert simulation account. A group of healthy young adults (1) observed a female speaker produce a set of familiar words (perception), and (2) observed and then repeated the words (production). There were two types of words, varying in articulatory complexity, as measured by the presence or absence of consonant clusters. The simple words contained no consonant cluster (e.g. “palace”), while the complex words contained one to three consonant clusters (e.g. “planet”). Results indicate that the left ventral premotor cortex (PMv) was significantly active during speech perception and speech production but that activation in this region was scaled to articulatory complexity only during speech production, revealing an incompletely specified efferent motor signal during speech perception. The right planum temporal (PT) was also active during speech perception and speech production, and activation in this region was scaled to articulatory complexity during both production and perception. These findings are discussed in the context of current theories of speech perception, with particular attention to accounts that include an explanatory role for mirror neurons

MRI neuroimaging: language recovery in adult aphasia due to stroke

Thesis (Ph.D.)--Boston UniversityThis research focuses on the contribution of magnetic resonance imaging (MRI) to understanding recovery and treatment of aphasia in adults who have suffered a stroke. There are three parts. Part 1 presents the feasibility of the application of an overt, picture-naming, functional MRI (fMRI) paradigm to examine neural activity in chronic, nonfluent aphasia (four mild-moderate and one severe nonfluent/global patient). The advantages and disadvantages of an overt, object picture-naming, fMRI block-design paradigm are discussed. An overt naming fMRI design has potential as a method to provide insight into recovery from adult aphasia including plasticity of the brain after left hemisphere stroke and response to treatment. Part 2 uses the overt naming fMRI paradigm to examine changes in neural activity (neural plasticity) after a two-week series of repetitive transcranial magnetic stimulation (rTMS) treatments to improve picture naming in chronic nonfluent aphasia. An overview of rTMS and rationale for use of rTMS as a clinical treatment for aphasia is provided. Patterns of fMRI activation are examined in two patients with chronic nonfluent aphasia following a two-week series of 1 Hz rTMS treatments to suppress the right pars triangularis portion of the right hemisphere, Broca's homologue. One patient responded well, and the other did not. Differences in fMRI activation in response to the rTMS treatment for the two patients may be due to differences in the patients' lesion sites and extent of damage within each lesion site. Part 3 examines the area of the corpus callosum (CC) in 21 chronic nonfluent aphasia patients and 13 ageequivalent controls using structural MRI. Understanding brain morphology and potential atrophy of the CC in chronic stroke patients may shed light on alterations in the interhemispheric dynamics after stroke, especially patterns of brain reorganization during post-stroke language recovery. A decrease in interhemispheric connections has implications for mechanisms of language recovery and potential success with specific treatment methods. Future directions of both structural and functional neuroimaging to study language recovery in adult aphasia are discussed

Who is that? Brain networks and mechanisms for identifying individuals

Social animals can identify conspecifics by many forms of sensory input. However, whether the neuronal computations that support this ability to identify individuals rely on modality-independent convergence or involve ongoing synergistic interactions along the multiple sensory streams remains controversial. Direct neuronal measurements at relevant brain sites could address such questions, but this requires better bridging the work in humans and animal models. Here, we overview recent studies in nonhuman primates on voice and face identity-sensitive pathways and evaluate the correspondences to relevant findings in humans. This synthesis provides insights into converging sensory streams in the primate anterior temporal lobe (ATL) for identity processing. Furthermore, we advance a model and suggest how alternative neuronal mechanisms could be tested

The Role of Speech Production System in Audiovisual Speech Perception

Seeing the articulatory gestures of the speaker significantly enhances speech perception. Findings from recent neuroimaging studies suggest that activation of the speech motor system during lipreading enhance speech perception by tuning, in a top-down fashion, speech-sound processing in the superior aspects of the posterior temporal lobe. Anatomically, the superior-posterior temporal lobe areas receive connections from the auditory, visual, and speech motor cortical areas. Thus, it is possible that neuronal receptive fields are shaped during development to respond to speech-sound features that coincide with visual and motor speech cues, in contrast with the anterior/lateral temporal lobe areas that might process speech sounds predominantly based on acoustic cues. The superior-posterior temporal lobe areas have also been consistently associated with auditory spatial processing. Thus, the involvement of these areas in audiovisual speech perception might partly be explained by the spatial processing requirements when associating sounds, seen articulations, and one’s own motor movements. Tentatively, it is possible that the anterior “what” and posterior “where / how” auditory cortical processing pathways are parts of an interacting network, the instantaneous state of which determines what one ultimately perceives, as potentially reflected in the dynamics of oscillatory activity