Electrophysiological evidence for differences between fusion and combination illusions in audiovisual speech perception

Accepted manuscript online: 4 October 2017Incongruent audiovisual speech stimuli can lead to perceptual illusions such as fusions or combinations. Here, we investigated the underlying audiovisual integration process by measuring ERPs. We observed that visual speech-induced suppression of P2 amplitude (which is generally taken as a measure of audiovisual integration) for fusions was similar to suppression obtained with fully congruent stimuli, whereas P2 suppression for combinations was larger. We argue that these effects arise because the phonetic incongruency is solved differently for both types of stimuli.MB was supported by the Spanish Ministry of Economy and Competitiveness (MINECO grant FPDI-2013-15661) and the Netherlands Organization for Scientific Research (NWO VENI grant 275-89-027)

Suppression of the auditory N1 by visual anticipatory motion is modulated by temporal and identity predictability

The amplitude of the auditory N1 component of the event-related potential (ERP) is typically suppressed when a sound is accompanied by visual anticipatory information that reliably predicts the timing and identity of the sound. While this visually-induced suppression of the auditory N1 is considered an early electrophysiological marker of fulfilled prediction, it is not yet fully understood whether this internal predictive coding mechanism is primarily driven by the temporal characteristics, or by the identity features of the anticipated sound. The current study examined the impact of temporal and identity predictability on suppression of the auditory N1 by visual anticipatory motion with an ecologically valid audiovisual event (a video of a handclap). Predictability of auditory timing and identity was manipulated in three different conditions in which sounds were either played in isolation, or in conjunction with a video that either reliably predicted the timing of the sound, the identity of the sound, or both the timing and identity. The results showed that N1 suppression was largest when the video reliably predicted both the timing and identity of the sound, and reduced when either the timing or identity of the sound was unpredictable. The current results indicate that predictions of timing and identity are both essential elements for predictive coding in audition

The effects of stereo disparity on the behavioural and electrophysiological correlates of audio-visual motion in depth.

Motion is represented by low-level signals, such as size-expansion in vision or loudness changes in the auditory modality. The visual and auditory signals from the same object or event may be integrated and facilitate detection. We explored behavioural and electrophysiological correlates of congruent and incongruent audio-visual depth motion in conditions where auditory level changes, visual expansion, and visual disparity cues were manipulated. In Experiment 1 participants discriminated auditory motion direction whilst viewing looming or receding, 2D or 3D, visual stimuli. Responses were faster and more accurate for congruent than for incongruent audio-visual cues, and the congruency effect (i.e., difference between incongruent and congruent conditions) was larger for visual 3D cues compared to 2D cues. In Experiment 2, event-related potentials (ERPs) were collected during presentation of the 2D and 3D, looming and receding, audio-visual stimuli, while participants detected an infrequent deviant sound. Our main finding was that audio-visual congruity was affected by retinal disparity at an early processing stage (135 – 160 ms) over occipito-parietal scalp. Topographic analyses suggested that similar brain networks were activated for the 2D and 3D congruity effects, but that cortical responses were stronger in the 3D condition. Differences between congruent and incongruent conditions were observed between 140 – 200 ms, 220 – 280 ms, and 350 – 500 ms after stimulus onset

The multisensory function of the human primary visual cortex

It has been nearly 10 years since Ghazanfar and Schroeder (2006) proposed that the neocortex is essentially multisensory in nature. However, it is only recently that sufficient and hard evidence that supports this proposal has accrued. We review evidence that activity within the human primary visual cortex plays an active role in multisensory processes and directly impacts behavioural outcome. This evidence emerges from a full pallet of human brain imaging and brain mapping methods with which multisensory processes are quantitatively assessed by taking advantage of particular strengths of each technique as well as advances in signal analyses. Several general conclusions about multisensory processes in primary visual cortex of humans are supported relatively solidly. First, haemodynamic methods (fMRI/PET) show that there is both convergence and integration occurring within primary visual cortex. Second, primary visual cortex is involved in multisensory processes during early post-stimulus stages (as revealed by EEG/ERP/ERFs as well as TMS). Third, multisensory effects in primary visual cortex directly impact behaviour and perception, as revealed by correlational (EEG/ERPs/ERFs) as well as more causal measures (TMS/tACS). While the provocative claim of Ghazanfar and Schroeder (2006) that the whole of neocortex is multisensory in function has yet to be demonstrated, this can now be considered established in the case of the human primary visual cortex

The relationship between multisensory associative learning and multisensory integration

Integrating sensory information from multiple modalities leads to more precise and efficient perception and behaviour. The process of determining which sensory information should be perceptually bound is reliant on both low-level stimulus features, as well as multisensory associations learned throughout development based on the statistics of our environment. Here, we explored the relationship between multisensory associative learning and multisensory integration using encephalography (EEG) and behavioural measures. Sixty-one participants completed a three-phase study. First, participants were exposed to novel audiovisual shape-tone pairings with frequent and infrequent stimulus pairings and completed a target detection task. EEG recordings of the mismatch negativity (MMN) and P3 were calculated as neural indices of multisensory associative learning. Next, the same learned stimulus pairs were presented in audiovisual as well as unisensory auditory and visual modalities while both early (\u3c100 ms) and late neural indices of multisensory integration were recorded. Finally, participants completed an analogous behavioural speeded-response task, with behavioural indices of multisensory gain calculated using the Race Model. Significant relationships were found in fronto-central and occipital areas between neural measures of associative learning and both early and late indices of multisensory integration in frontal and centro-parietal areas, respectively. Participants who showed stronger indices of associative learning also exhibited stronger indices of multisensory integration of the stimuli they learned to associate. Furthermore, a significant relationship was found between neural index of early multisensory integration and behavioural indices of multisensory gain. These results provide insight into the neural underpinnings of how higher-order processes such as associative learning guide multisensory integration

Audiovisual Processing of Chinese Characters Elicits Suppression and Congruency Effects in MEG

Learning to associate written letters/characters with speech sounds is crucial for reading acquisition. Most previous studies have focused on audiovisual integration in alphabetic languages. Less is known about logographic languages such as Chinese characters, which map onto mostly syllable-based morphemes in the spoken language. Here we investigated how long-term exposure to native language affects the underlying neural mechanisms of audiovisual integration in a logographic language using magnetoencephalography (MEG). MEG sensor and source data from 12 adult native Chinese speakers and a control group of 13 adult Finnish speakers were analyzed for audiovisual suppression (bimodal responses vs. sum of unimodal responses) and congruency (bimodal incongruent responses vs. bimodal congruent responses) effects. The suppressive integration effect was found in the left angular and supramarginal gyri (205–365 ms), left inferior frontal and left temporal cortices (575–800 ms) in the Chinese group. The Finnish group showed a distinct suppression effect only in the right parietal and occipital cortices at a relatively early time window (285–460 ms). The congruency effect was only observed in the Chinese group in left inferior frontal and superior temporal cortex in a late time window (about 500–800 ms) probably related to modulatory feedback from multi-sensory regions and semantic processing. The audiovisual integration in a logographic language showed a clear resemblance to that in alphabetic languages in the left superior temporal cortex, but with activation specific to the logographic stimuli observed in the left inferior frontal cortex. The current MEG study indicated that learning of logographic languages has a large impact on the audiovisual integration of written characters with some distinct features compared to previous results on alphabetic languages

Multisensory Associative Learning and Multisensory Integration

Human multisensory experiences with the world rely on a combination of top-down and bottom-up influences, a process that changes throughout development. The present study explored the relationship between multisensory associative learning and multisensory integration using encephalography (EEG) and behavioural measures. While recording EEG activity, participants were exposed to novel pairings of non-sociolinguistic audiovisual stimuli of varying presentation probability while performing a detection task. The same stimuli were then used in another detection task, which was followed by an analogous behavioural speeded-response task, both of which kept probabilities equal and tested for multisensory integration. Significant relationships were found in fronto-central and occipital areas between late measures of associative learning and both early and late indices of multisensory integration in frontal and centro-parietal areas, respectively. Furthermore, a significant relationship was found between the behavioural and early neural index of multisensory integration. These results highlight the influence of higher-order processes, namely, learned associations on multisensory integration

Integration of spoken and written words in beginning readers: A topographic ERP study

Integrating visual and auditory language information is critical for reading. Suppression and congruency effects in audiovisual paradigms with letters and speech sounds have provided information about low-level mechanisms of grapheme-phoneme integration during reading. However, the central question about how such processes relate to reading entire words remains unexplored. Using ERPs, we investigated whether audiovisual integration occurs for words already in beginning readers, and if so, whether this integration is reflected by differences in map strength or topography (aim 1); and moreover, whether such integration is associated with reading fluency (aim 2). A 128-channel EEG was recorded while 69 monolingual (Swiss)-German speaking first-graders performed a detection task with rare targets. Stimuli were presented in blocks either auditorily (A), visually (V) or audiovisually (matching: AVM; nonmatching: AVN). Corresponding ERPs were computed, and unimodal ERPs summated (A + V = sumAV). We applied TANOVAs to identify time windows with significant integration effects: suppression (sumAV-AVM) and congruency (AVN-AVM). They were further characterized using GFP and 3D-centroid analyses, and significant effects were correlated with reading fluency. The results suggest that audiovisual suppression effects occur for familiar German and unfamiliar English words, whereas audiovisual congruency effects can be found only for familiar German words, probably due to lexical-semantic processes involved. Moreover, congruency effects were characterized by topographic differences, indicating that different sources are active during processing of congruent compared to incongruent audiovisual words. Furthermore, no clear associations between audiovisual integration and reading fluency were found. The degree to which such associations develop in beginning readers remains open to further investigation

Investigating multisensory integration in human early visual and auditory areas with intracranial electrophysiological recordings: insights and perspectives

Cross-modal processing and multisensory integration (MSI) can be observed at early stages of sensory processing in the cortex. However, the neurophysiological mechanisms underlying these processes and how they vary across sensory systems remain elusive. The aim of this study was to investigate how cross-modal processing and MSI are reflected in power and phase of oscillatory neuronal activity at different temporal scales in different sensory cortices. To this goal, we recorded stereo-electroencephalographic (SEEG) responses from early visual (calcarine and pericalcarine) and auditory (Heschl’s gyrus and planum temporale) regions in patients with drug-resistant epilepsy while performing an audio-visual oddball task. To Investigate crossmodal processing and MSI in the power domain of oscillatory activity, we explored a wide range of frequency bands (theta/alpha band: 5-13Hz; beta band: 13-30 Hz; gamma band: 30-80 Hz; high-gamma band: 80-200 Hz) during the first 150 ms post-stimulus onset. Differently, to investigate crossmodal processing and MSI in the phase domain of oscillatory activity, we explored a narrow range of frequency bands (theta/alpha band: 5-13Hz; beta band: 13-30 Hz; gamma band: 30-80 Hz) during the first 300 ms post-stimulus onset. In the power domain, we showed that cross-modal processing occurs mainly in the high-gamma band (80-200Hz) in both cortices. However, we evidenced that the way MSI is expressed across modalities differs considerably: in the visual cortex, MSI relies mainly on the beta band, however it is also evident, to a lesser extent, in the gamma and high-gamma band, while the auditory cortex reveals widespread MSI in the high-gamma band and, to a lesser extent, across the gamma band and the other investigated frequency bands. In the phase domain, we showed that cross-modal processing is differently expressed across modalities: in the auditory cortex it induces an increased phase concentration index (PCI) in ongoing oscillatory activity across all the investigated frequency bands, while, in the visual cortex, it induces an increased PCI particularly evident in the theta/alpha band with few or no effect respectively in the gamma and beta band. Importantly in both cortices, the most part of the COIs showing increased PCI, were not accompanied by a concomitant increase in power. These results indicate that in both auditory and visual cortex, cross-modal processing induces a pure phase resetting of the oscillatory activity. During MSI processing we observed, in both cortices, a stronger increase in PCI, in comparison to the intramodal processing, in the theta/alpha band and in the gamma band. Our results confirm the presence of cross-modal information representations at neuronal populations level and conform to a model where the cross-modal input induces phase-locked modulation of the ongoing oscillations. Importantly, our data showed that the way MSI is expressed in power modulations differs between the investigated sensory cortices suggesting the presence of different types of neurophysiological interactions during this process. These results are discussed in the framework of the current literature