Modulating fusion in the McGurk effect by binding processes and contextual noise

International audienceIn a series of experiments we showed that the McGurk effect may be modulated by context: applying incoherent auditory and visual material before an audiovisual target made of an audio "ba" and a video "ga" significantly decreases the McGurk effect. We interpreted this as showing the existence of an audiovisual "binding" stage controlling the fusion process. Incoherence would produce "unbinding" and result in decreasing the weight of the visual input in the fusion process. In this study, we further explore this binding stage around two experiments. Firstly we test the "rebinding" process, by presenting a short period of either coherent material or silence after the incoherent "unbinding" context. We show that coherence provides "rebinding", resulting in a recovery of the McGurk effect. In contrary, silence provides no rebinding and hence "freezes" the unbinding process, resulting in no recovery of the McGurk effect. Capitalizing on this result, in a second experiment including an incoherent unbinding context followed by a coherent rebinding context before the target, we add noise all over the contextual period, though not in the McGurk target. It appears that noise uniformly increases the rate of McGurk responses compared to the silent condition. This suggests that contextual noise increases the weight of the visual input in fusion, even if there is no noise within the target stimulus where fusion is applied. We conclude on the role of audiovisual coherence and noise in the binding process, in the framework of audiovisual speech scene analysis and the cocktail party effect

Analyse de scènes de parole multisensorielle : Mise en évidence et caractérisation d’un processus de liage audiovisuel préalable à la fusion.

This thesis is focused on audiovisual speech, with the aim of better understanding the mechanisms enabling to link auditory and visual speech signals in the human brain.In a first theoretical part were view key issues concerning audiovisual speech and the underlying models of integration of auditory and visual inputs. Then we discuss the question of the existence of an early audio-visual integration stage and we formulate our main hypothesis in which we posit the existence of an “audiovisual binding process”, prior to audio- visual fusion.Based on this assumption the experimental work is organized in two parts. Firstly we provide evidence in favor of the existence of a binding process by developing and gradually refining a new experimental paradigm. Then we attempt to characterize this process by setting up a number of experiments studying the influence of various critical parameters. We finish with a discussion of our results and relate them to present questions about cognitive perceptual processing of audiovisual speech.Dans le cadre de cette thèse centrée sur la perception de la parole audiovisuelle, le but de notre étude est de mieux comprendre les mécanismes qui permettent de lier les signaux auditifs et visuels de la parole. Une première partie théorique permet dépasser en revue les principales questions portant sur la parole audiovisuelle et sur les principaux modèles d’intégration des inputs auditif et visuels. Puis nous abordons la question de l’existence d’un possible niveau précoce d’intégration audio-visuelle qui nous conduit à formuler notre hypothèse principale d’existence d’un processus de« liage audiovisuel », qui constituerait une étape préalable à la fusion audio-visuelle. En nous basant sur cette hypothèse nous abordons la phase expérimentale de notre travail, organisée en deux parties. D’abord nous entreprenons de démontrer l’existence du processus du liage en utilisant et en affinant peu à peu un nouveau paradigme expérimental. Puis nous traitons de la caractérisation de ce processus en mettant en place des expériences qui étudient l’influence de différents paramètres critiques. Nous terminons ce manuscrit par une discussion sur nos résultats et leur positionnement dans la littérature actuelle