Multisensory integration in speech processing: Neural mechanisms of cross-modal aftereffects

Traditionally, perceptual neuroscience has focused on unimodal information processing. This is true also for investigations of speech processing, where the auditory modality was the natural focus of interest. Given the complexity of neuronal processing, this was a logical step, considering that the field was still in its infancy. However, it is clear that this restriction does not do justice to the way we perceive the world around us in everyday interactions. Very rarely is sensory information confined to one modality. Instead, we are constantly confronted with a stream of input to several or all senses and already in infancy, we match facial movements with their corresponding sounds (Campbell et al. 2001; Kuhl and Meltzoff 1982). Moreover, the information that is processed by our individual senses does not stay separated. Rather, the different channels interact and influence each other, affecting perceptual interpretations and constructions (Calvert 2001). Consequently, in the last 15–20 years, the perspective in cognitive science and perceptual neuroscience has shifted to include investigations of such multimodal integrative phenomena. Facilitating cross-modal effects have consistently been demonstrated behaviorally (Shimojo and Shams 2001). When multisensory input is congruent (e.g., semantically and/or temporally) it typically lowers detection thresholds (Frassinetti et al. 2002), shortens reaction times (Forster et al. 2002; Schröger and Widmann 1998), and decreases saccadic eye movement latencies (Hughes et al. 1994) as compared to unimodal exposure. When incongruent input is (artificially) added in a second modality, this usually has opposite consequences (Sekuler et al. 1997)

Robust single trial identification of conscious percepts triggered by sensory events of variable saliency

The neural correlates of visual awareness are elusive because of its fleeting nature. Here we have addressed this issue by using single trial statistical "brain reading" of neurophysiological event related (ERP) signatures of conscious perception of visual attributes with different levels of saliency. Behavioral reports were taken at every trial in 4 experiments addressing conscious access to color, luminance, and local phase offset cues. We found that single trial neurophysiological signatures of target presence can be observed around 300 ms at central parietal sites. Such signatures are significantly related with conscious perception, and their probability is related to sensory saliency levels. These findings identify a general neural correlate of conscious perception at the single trial level, since conscious perception can be decoded as such independently of stimulus salience and fluctuations of threshold levels. This approach can be generalized to successfully detect target presence in other individuals.This work was supported by the Portuguese Foundation for Science and Technology, grants ID: COMPETE, PEst-C/SAU/UI3282/2011, CENTRO-07-ST24-FEDER-00205, FP7-HEALTH-2013-INNOVATION-1 - 602186 - BRAINTRAIN, PTDC/SAU-NEU/68483/2006, PIC/IC/82986/2007, PTDC/SAU-ORG/118380 (FCT), and a scholarship to the first author, SFRH/BD/64383/2009, Bial Foundation 132 & 133/2013. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript