Multi-Level Audio-Visual Interactions in Speech and Language Perception

That we perceive our environment as a unified scene rather than individual streams of auditory, visual, and other sensory information has recently provided motivation to move past the long-held tradition of studying these systems separately. Although they are each unique in their transduction organs, neural pathways, and cortical primary areas, the senses are ultimately merged in a meaningful way which allows us to navigate the multisensory world. Investigating how the senses are merged has become an increasingly wide field of research in recent decades, with the introduction and increased availability of neuroimaging techniques. Areas of study range from multisensory object perception to cross-modal attention, multisensory interactions, and integration. This thesis focuses on audio-visual speech perception, with special focus on facilitatory effects of visual information on auditory processing. When visual information is concordant with auditory information, it provides an advantage that is measurable in behavioral response times and evoked auditory fields (Chapter 3) and in increased entrainment to multisensory periodic stimuli reflected by steady-state responses (Chapter 4). When the audio-visual information is incongruent, the combination can often, but not always, combine to form a third, non-physically present percept (known as the McGurk effect). This effect is investigated (Chapter 5) using real word stimuli. McGurk percepts were not robustly elicited for a majority of stimulus types, but patterns of responses suggest that the physical and lexical properties of the auditory and visual stimulus may affect the likelihood of obtaining the illusion. Together, these experiments add to the growing body of knowledge that suggests that audio-visual interactions occur at multiple stages of processing

Modified locus equations categorize stop place in a perceptually realistic time frame

This is the publisher's version, also available electronically from http://scitation.aip.org/content/asa/journal/jasa/131/6/10.1121/1.4722169.Slope and y-intercepts of locus equations have previously been shown to successfully classify place of articulation for English voiced stop consonants when derived from measurements at vowel onset and vowel midpoint. However, listeners are capable of identifying English voiced stops when less than 30 ms of vowel is presented. The present results show that modified locus equation measurements made within the first several pitch periods of a vowel following an English voiced stop were also successful at classifying place of articulation, consistent with the amount of vocalic information necessary for perceptual identification of English voiced stops /b d g/

Contingent categorization in speech perception

This is an Accepted Manuscript of an article published by Taylor & Francis in Language Cognition and Neuroscience in 2014, available online: http://www.tandfonline.com/10.1080/01690965.2013.824995.The speech signal is notoriously variable, with the same phoneme realized differently depending on factors like talker and phonetic context. Variance in the speech signal has led to a proliferation of theories of how listeners recognize speech. A promising approach, supported by computational modeling studies, is contingent categorization, wherein incoming acoustic cues are computed relative to expectations. We tested contingent encoding empirically. Listeners were asked to categorize fricatives in CV syllables constructed by splicing the fricative from one CV syllable with the vowel from another CV syllable. The two spliced syllables always contained the same fricative, providing consistent bottom-up cues; however on some trials, the vowel and/or talker mismatched between these syllables, giving conflicting contextual information. Listeners were less accurate and slower at identifying the fricatives in mismatching splices. This suggests that listeners rely on context information beyond bottom-up acoustic cues during speech perception, providing support for contingent categorization

A Sound-Sensitive Source of Alpha Oscillations in Human Non-Primary Auditory Cortex

Copyright © 2019 Billig, Herrmann et al. The functional organization of human auditory cortex can be probed by characterizing responses to various classes of sound at different anatomical locations. Along with histological studies this approach has revealed a primary field in posteromedial Heschl\u27s gyrus (HG) with pronounced induced high-frequency (70-150 Hz) activity and short-latency responses that phase-lock to rapid transient sounds. Low-frequency neural oscillations are also relevant to stimulus processing and information flow, however, their distribution within auditory cortex has not been established. Alpha activity (7-14 Hz) in particular has been associated with processes that may differentially engage earlier versus later levels of the cortical hierarchy, including functional inhibition and the communication of sensory predictions. These theories derive largely from the study of occipitoparietal sources readily detectable in scalp electroencephalography. To characterize the anatomical basis and functional significance of less accessible temporal-lobe alpha activity we analyzed responses to sentences in seven human adults (4 female) with epilepsy who had been implanted with electrodes in superior temporal cortex. In contrast to primary cortex in posteromedial HG, a non-primary field in anterolateral HG was characterized by high spontaneous alpha activity that was strongly suppressed during auditory stimulation. Alpha-power suppression decreased with distance from anterolateral HG throughout superior temporal cortex, and was more pronounced for clear compared to degraded speech. This suppression could not be accounted for solely by a change in the slope of the power spectrum. The differential manifestation and stimulus-sensitivity of alpha oscillations across auditory fields should be accounted for in theories of their generation and function.SIGNIFICANCE STATEMENT To understand how auditory cortex is organized in support of perception, we recorded from patients implanted with electrodes for clinical reasons. This allowed measurement of activity in brain regions at different levels of sensory processing. Oscillations in the alpha range (7-14 Hz) have been associated with functions including sensory prediction and inhibition of regions handling irrelevant information, but their distribution within auditory cortex is not known. A key finding was that these oscillations dominated in one particular non-primary field, anterolateral Heschl\u27s gyrus, and were suppressed when subjects listened to sentences. These results build on our knowledge of the functional organization of auditory cortex and provide anatomical constraints on theories of the generation and function of alpha oscillations

Common Fronto-temporal Effective Connectivity in Humans and Monkeys

Cognitive pathways supporting human language and declarative memory are thought to have uniquely evolutionarily differentiated in our species. However, cross-species comparisons are missing on site-specific effective connectivity between regions important for cognition. We harnessed a new approach using functional imaging to visualize the impact of direct electrical brain stimulation in human neurosurgery patients. Applying the same approach with macaque monkeys, we found remarkably comparable patterns of effective connectivity between auditory cortex and ventro-lateral prefrontal cortex (vlPFC) and parahippocampal cortex in both species. Moreover, in humans electrical tractography revealed rapid evoked potentials in vlPFC from stimulating auditory cortex and speech sounds drove vlPFC, consistent with prior evidence in monkeys of direct projections from auditory cortex to vocalization responsive regions in vlPFC. The results identify a common effective connectivity signature that from auditory cortex is equally direct to vlPFC and indirect to the hippocampus (via parahippocampal cortex) in human and nonhuman primates

Common Fronto-temporal Effective Connectivity in Humans and Monkeys

Human brain pathways supporting language and declarative memory are thought to have differentiated substantially during evolution. However, cross-species comparisons are missing on site-specific effective connectivity between regions important for cognition. We harnessed functional imaging to visualize the effects of direct electrical brain stimulation in macaque monkeys and human neurosurgery patients. We discovered comparable effective connectivity between caudal auditory cortex and both ventro-lateral prefrontal cortex (VLPFC, including area 44) and parahippocampal cortex in both species. Human-specific differences were clearest in the form of stronger hemispheric lateralization effects. In humans, electrical tractography revealed remarkably rapid evoked potentials in VLPFC following auditory cortex stimulation and speech sounds drove VLPFC, consistent with prior evidence in monkeys of direct auditory cortex projections to homologous vocalization-responsive regions. The results identify a common effective connectivity signature in human and nonhuman primates, which from auditory cortex appears equally direct to VLPFC and indirect to the hippocampus