Distracted But Still Listening: How Cognitive and Perceptual Load Affect Multisensory Integration

Abstract

There is an ongoing debate about whether multisensory integration (MSI) occurs automatically and before attention is allocated or if multisensory integration occurs after attention is allocated and is subject to top-down attentional control. A recent study investigated this by manipulating the availability of attentional resources in a dual-task paradigm, finding that when the demands of the secondary task increased, susceptibility to the Sound-Induced-Flash-Illusion, a non-speech audiovisual illusion, also increased (Michail & Keil, 2018). This was taken as evidence for the role of post-attentive, top-down control during MSI. In the present study, we extend this logic to explore the role of attentional processes in audiovisual speech integration. Participants completed an audio and audiovisual speech-in-noise task at a fixed signal-to-noise ratio interleaved with an n-back task (0, 1, or 2) to apply cognitive load. We found that as cognitive load increased, speech recognition accuracy decreased in audio and audiovisual stimuli; however, there was no effect of cognitive load on audiovisual enhancement. One limitation of this first experiment was that it did not investigate whether an effect of cognitive load would only be observed under low perceptual load conditions. To address this question, we conducted a follow-up experiment with our dual-task paradigm that manipulated cognitive load via an n-back task and perceptual load via the signal-to-noise ratio of our speech stimuli. We found that there was no interaction between cognitive load, perceptual load, and speech modality, suggesting that cognitive load has no effect on integration regardless of perceptual load. Contrasting prior findings, our results suggest that audiovisual integration of speech occurs in an automatic, pre-attentive manner. Our results underscore the importance of using ecologically valid stimuli in assessing how integration occurs