Data and simulations about audiovisual asynchrony and predictability in speech perception

International audienceSince a paper by Chandrasekaran et al. (2009), an increasing number of neuroscience papers capitalize on the assumption that visual speech would be typically 150 ms ahead of auditory speech. It happens that the estimation of audiovisual asynchrony by Chandrasekaran et al. is valid only in very specific cases, for isolated CV syllables or at the beginning of a speech utterance. We present simple audiovisual data on plosive-vowel syllables (pa, ta, ka, ba, da, ga, ma, na) showing that audiovisual synchrony is actually rather precise when syllables are chained in sequences, as they are typically in most parts of a natural speech utterance. Then we discuss on the way the natural coordination between sound and image (combining cases of lead and lag of the visual input) is reflected in the so-called temporal integration window for audiovisual speech perception (van Wassenhove et al., 2007). We conclude by a computational proposal about predictive coding in such sequences, showing that the visual input may actually provide and enhance predictions even if it is quite synchronous with the auditory input

Auditory dominance in motor-sensory temporal recalibration

Perception of synchrony between one’s own action (e.g. a finger tap) and the sensory feedback thereof (e.g. a flash or click) can be shifted after exposure to an induced delay (temporal recalibration effect, TRE). It remains elusive, however, whether the same mechanism underlies motor-visual (MV) and motor-auditory (MA) TRE. We examined this by measuring crosstalk between MV- and MA-delayed feedbacks. During an exposure phase, participants pressed a mouse at a constant pace while receiving visual or auditory feedback that was either delayed (+150 ms) or subjectively synchronous (+50 ms). During a post-test, participants then tried to tap in sync with visual or auditory pacers. TRE manifested itself as a compensatory shift in the tap–pacer asynchrony (a larger anticipation error after exposure to delayed feedback). In experiment 1, MA and MV feedback were either both synchronous (MV-sync and MA-sync) or both delayed (MV-delay and MA-delay), whereas in experiment 2, different delays were mixed across alternating trials (MV-sync and MA-delay or MV-delay and MA-sync). Exposure to consistent delays induced equally large TREs for auditory and visual pacers with similar build-up courses. However, with mixed delays, we found that synchronized sounds erased MV-TRE, but synchronized flashes did not erase MA-TRE. These results suggest that similar mechanisms underlie MA- and MV-TRE, but that auditory feedback is more potent than visual feedback to induce a rearrangement of motor-sensory timing

Behavioral and Neural Indices of Perceiving Multisensory Action Outcomes

Distinct perception for voluntary vs. externally-generated action outcomes has been demonstrated in timing and intensity domains. First, time interval between an action and its outcome is perceived shorter. Second, sensory stimuli triggered by one’s own action is judged as less intense than similar stimuli triggered externally. The differential perception of voluntary action outcomes has been attributed to efference copy-related predictive mechanisms, and has been studied extensively using behavioral and imaging methods. However, although voluntary movements in the real world produce feedback in multiple modalities, previous experiments mostly investigated unimodal action outcomes. Therefore, the perception of multisensory inputs associated with our own actions remains to be explored. The aim of this dissertation was to fill this gap by investigating the behavioral and neural correlates of multisensory action outcomes. In Study 1, synchrony perception for multisensory outcomes triggered by voluntary vs. externally-generated movements was assessed. Study 1.1 showed increased perception of simultaneity for audiovisual stimulus pairs around the time of action. Study 1.2 revealed a similar effect also when the movement was externally-generated, underlining the importance of causal relations between events in shaping time perception. Interestingly, the slopes of the psychometric functions in the voluntary condition were significantly steeper than the slopes in the externally-generated condition, suggesting a role of action-related predictive mechanisms in making synchrony perception more precise. Study 2 investigated the neural correlates of perceiving unimodal vs. bimodal inputs triggered by voluntary button presses compared with passive viewing of identical stimuli. Results showed BOLD suppression for voluntary action outcomes in comparison to passive viewing of the same stimuli. This BOLD suppression effect was independent of the to-be-attended modality and the number of modalities presented. The cerebellum was found to be recruited more during bimodal trials and when a delay was detected. These findings support action-related predictive processing of voluntary action outcomes, demonstrating it also for multisensory action outcomes. The findings also indicate the cerebellum’s role in error-related action outcome processing, and the influence of the additional sensory modality on error-related activity in the cerebellum. Study 3 investigated neural correlates of perceiving unimodal vs. bimodal action outcomes by focusing on efference copy-related predictive processing in a naturalistic experimental set- up. Results extend findings of Study 2 regarding the predictive processing of multisensory action outcomes to a naturalistic context, and support the role of the cerebellum in error- related action outcome processing. Importantly, activity in the cerebellum was modulated by the additional modality, highlighting the role of multisensory processing in shaping motor- sensory interactions. Together, findings of these studies strengthen existing evidence on the distinctive perception for voluntary action outcomes, extending it to multisensory action outcomes, and to a realistic context. Implications of this line of research extend to revealing mechanisms behind agency deficits frequently observed in schizophrenia, as well as to the development of intervention techniques targeting the rehabilitation of patients with spinal cord injury or stroke

Degraded visual and auditory input individually impair audiovisual emotion recognition from speech-like stimuli, but no evidence for an exacerbated effect from combined degradation

Emotion recognition requires optimal integration of the multisensory signals from vision and hearing. A sensory loss in either or both modalities can lead to changes in integration and related perceptual strategies. To investigate potential acute effects of combined impairments due to sensory information loss only, we degraded the visual and auditory information in audiovisual video-recordings, and presented these to a group of healthy young volunteers. These degradations intended to approximate some aspects of vision and hearing impairment in simulation. Other aspects, related to advanced age, potential health issues, but also long-term adaptation and cognitive compensation strategies, were not included in the simulations. Besides accuracy of emotion recognition, eye movements were recorded to capture perceptual strategies. Our data show that emotion recognition performance decreases when degraded visual and auditory information are presented in isolation, but simultaneously degrading both modalities does not exacerbate these isolated effects. Moreover, degrading the visual information strongly impacts recognition performance and on viewing behavior. In contrast, degrading auditory information alongside normal or degraded video had little (additional) effect on performance or gaze. Nevertheless, our results hold promise for visually impaired individuals, because the addition of any audio to any video greatly facilitates performance, even though adding audio does not completely compensate for the negative effects of video degradation. Additionally, observers modified their viewing behavior to degraded video in order to maximize their performance. Therefore, optimizing the hearing of visually impaired individuals and teaching them such optimized viewing behavior could be worthwhile endeavors for improving emotion recognition

The benefit received from visual information when listening to clear and degraded speech in background noise

In order to improve speech understanding in background noise, visual information is used to enhance the incoming auditory signal. This enhancement is known as the visual speech benefit. Variation in the amount of visual speech benefit that is received by participants is the focus of this research project and is examined for both clear and vocoded speech. Vocoded speech simulates the type of speech experienced by cochlear-implant users. Experiments 1 and 2 examined variation in the amount of visual speech benefit gained if the type of background noise in the speech test changed. The key results from Experiment 1 and 2 were that the visual information provided was not enough to enhance speech understanding for particularly unintelligible speech. Experiment 3 assessed change to levels of visual speech benefit if the target talker in the stimuli changed. Significant differences in intelligibility between talkers was found. The amount of visual speech benefit increased as the audio intelligibility of the target talker decreased in clear speech. Overall, therefore, it is important that consideration is given to the levels of intelligibility provided by the stimuli used in speech perception testing as this may change outcomes. Experiments 4 and 5 examined individual differences that may predict the amount of visual speech benefit gained. In Experiment 4, the significant predictors of the amount of visual speech benefit gained in clear speech were general speech perception ability, ability to detect audio and visual synchrony, and tendency towards autistic traits. The results of Experiment 5 showed that general speech perception ability and time spent looking at the mouth area measured using eye-tracking were significant predictors of the amount of visual speech benefit gained in clear speech. Individual differences between participants may therefore predict differences in speech perception and should also be considered when testing speech perception

Keeping track of emotions:audiovisual integration for emotion recognition and compensation for sensory degradations captured by perceptual strategies

The majority of emotional expressions are multimodal and dynamic in nature. Emotion recognition, therefore, requires integration of these multimodal signals. Sensory impairments likely affect emotion recognition, but although sensory impairments are common in older adults, it is unknown how they affect emotion recognition. As more people reach old age, accompanied by an increase in the prevalence of sensory impairments, it is urgent to comprehensively understand audiovisual integration, especially in older individuals. My thesis sought to create a basic understanding of audiovisual integration for emotion recognition and study how audiovisual interactions change with simulated sensory impairments. A secondary aim was to understand how age affects these outcomes. To systematically address these aims, I examined how well observers recognize emotions, presented via videos, and how emotion recognition accuracy and perceptual strategies, assessed via eye-tracking, vary under changing availability and reliability of the visual and auditory information. The research presented in my thesis shows that audiovisual integration and compensation abilities remain intact with age, despite a general decline in recognition accuracy. Compensation for degraded audio is possible by relying more on visual signals, but not vice versa. Older observers adapt their perceptual strategies in a different, perhaps less efficient, manner than younger observers. Importantly, I demonstrate that it is crucial to use additional measurements besides recognition accuracy in order to understand audiovisual integration mechanisms. Measurements such as eye-tracking allow examining whether the reliance on visual and auditory signals alters with age and (simulated) sensory impairments, even when lacking a change in accuracy

Timing and Time Perception: Procedures, Measures, and Applications

Timing and Time Perception: Procedures, Measures, and Applications is a one-of-a-kind, collective effort to present the most utilized and known methods on timing and time perception. Specifically, it covers methods and analysis on circadian timing, synchrony perception, reaction/response time, time estimation, and alternative methods for clinical/developmental research. The book includes experimental protocols, programming code, and sample results and the content ranges from very introductory to more advanced so as to cover the needs of both junior and senior researchers. We hope that this will be the first step in future efforts to document experimental methods and analysis both in a theoretical and in a practical manner

Time Distortions in Mind

Time Distortions in Mind brings together current research on temporal processing in clinical populations to elucidate the interdependence between perturbations in timing and disturbances in the mind and brain. For the student, the scientist, and the stepping-stone for further research. Readership: An excellent reference for the student and the scientist interested in aspects of temporal processing and abnormal psychology