Mirror Adaptation in Sensory-Motor Simultaneity

Background: When one watches a sports game, one may feel her/his own muscles moving in synchrony with the player's. Such parallels between observed actions of others and one's own has been well supported in the latest progress in neuroscience, and coined “mirror system.” It is likely that due to such phenomena, we are able to learn motor skills just by observing an expert's performance. Yet it is unknown whether such indirect learning occurs only at higher cognitive levels, or also at basic sensorimotor levels where sensorimotor delay is compensated and the timing of sensory feedback is constantly calibrated. Methodology/Principal Findings: Here, we show that the subject's passive observation of an actor manipulating a computer mouse with delayed auditory feedback led to shifts in subjective simultaneity of self mouse manipulation and auditory stimulus in the observing subjects. Likewise, self adaptation to the delayed feedback modulated the simultaneity judgment of the other subjects manipulating a mouse and an auditory stimulus. Meanwhile, subjective simultaneity of a simple visual disc and the auditory stimulus (flash test) was not affected by observation of an actor nor self-adaptation. Conclusions/Significance: The lack of shift in the flash test for both conditions indicates that the recalibration transfer is specific to the action domain, and is not due to a general sensory adaptation. This points to the involvement of a system for the temporal monitoring of actions, one that processes both one's own actions and those of others

Effect before cause: supramodal recalibration of sensorimotor timing.

YesBackground: Our motor actions normally generate sensory events, but how do we know which events were self generated and which have external causes? Here we use temporal adaptation to investigate the processing stage and generality of our sensorimotor timing estimates. Methodology/Principal Findings: Adaptation to artificially-induced delays between action and event can produce a startling percept¿upon removal of the delay it feels as if the sensory event precedes its causative action. This temporal recalibration of action and event occurs in a quantitatively similar manner across the sensory modalities. Critically, it is robust to the replacement of one sense during the adaptation phase with another sense during the test judgment. Conclusions/Significance: Our findings suggest a high-level, supramodal recalibration mechanism. The effects are well described by a simple model which attempts to preserve the expected synchrony between action and event, but only when causality indicates it is reasonable to do so. We further demonstrate that this model successfully characterises related adaptation data from outside the sensorimotor domain

Behavioral and Neural Indices of Perceiving Multisensory Action Outcomes

Distinct perception for voluntary vs. externally-generated action outcomes has been demonstrated in timing and intensity domains. First, time interval between an action and its outcome is perceived shorter. Second, sensory stimuli triggered by one’s own action is judged as less intense than similar stimuli triggered externally. The differential perception of voluntary action outcomes has been attributed to efference copy-related predictive mechanisms, and has been studied extensively using behavioral and imaging methods. However, although voluntary movements in the real world produce feedback in multiple modalities, previous experiments mostly investigated unimodal action outcomes. Therefore, the perception of multisensory inputs associated with our own actions remains to be explored. The aim of this dissertation was to fill this gap by investigating the behavioral and neural correlates of multisensory action outcomes. In Study 1, synchrony perception for multisensory outcomes triggered by voluntary vs. externally-generated movements was assessed. Study 1.1 showed increased perception of simultaneity for audiovisual stimulus pairs around the time of action. Study 1.2 revealed a similar effect also when the movement was externally-generated, underlining the importance of causal relations between events in shaping time perception. Interestingly, the slopes of the psychometric functions in the voluntary condition were significantly steeper than the slopes in the externally-generated condition, suggesting a role of action-related predictive mechanisms in making synchrony perception more precise. Study 2 investigated the neural correlates of perceiving unimodal vs. bimodal inputs triggered by voluntary button presses compared with passive viewing of identical stimuli. Results showed BOLD suppression for voluntary action outcomes in comparison to passive viewing of the same stimuli. This BOLD suppression effect was independent of the to-be-attended modality and the number of modalities presented. The cerebellum was found to be recruited more during bimodal trials and when a delay was detected. These findings support action-related predictive processing of voluntary action outcomes, demonstrating it also for multisensory action outcomes. The findings also indicate the cerebellum’s role in error-related action outcome processing, and the influence of the additional sensory modality on error-related activity in the cerebellum. Study 3 investigated neural correlates of perceiving unimodal vs. bimodal action outcomes by focusing on efference copy-related predictive processing in a naturalistic experimental set- up. Results extend findings of Study 2 regarding the predictive processing of multisensory action outcomes to a naturalistic context, and support the role of the cerebellum in error- related action outcome processing. Importantly, activity in the cerebellum was modulated by the additional modality, highlighting the role of multisensory processing in shaping motor- sensory interactions. Together, findings of these studies strengthen existing evidence on the distinctive perception for voluntary action outcomes, extending it to multisensory action outcomes, and to a realistic context. Implications of this line of research extend to revealing mechanisms behind agency deficits frequently observed in schizophrenia, as well as to the development of intervention techniques targeting the rehabilitation of patients with spinal cord injury or stroke

Adaptation to motor-visual and motor-auditory temporal lags transfer across modalities

Previous research has shown that the timing of a sensor-motor event is recalibrated after a brief exposure to a delayed feedback of a voluntary action (Stetson et al. 2006). Here, we examined whether it is the sensory or motor event that is shifted in time. We compared lag adaption for action-feedback in visuo-motor pairs and audio-motor pairs using an adaptation-test paradigm. Participants were exposed to a constant lag (50 or 150 ms) between their voluntary action (finger tap) and its sensory feedback (flash or tone pip) during an adaptation period (~3 min). Immediately after that, they performed a temporal order judgment (TOJ) task about the tap-feedback test stimulus pairings. The modality of the feedback stimulus was either the same as the adapted one (within-modal) or different (cross-modal). The results showed that the point of subjective simultaneity (PSS) was uniformly shifted in the direction of the exposed lag within and across modalities (motor-visual, motor-auditory). This suggests that the TRE of sensor-motor events is mainly caused by a shift in the motor component

IMPULSE moment-by-moment test:An implicit measure of affective responses to audiovisual televised or digital advertisements

IMPULSE is a novel method for detecting affective responses to dynamic audiovisual content. It is an implicit reaction time test that is carried out while an audiovisual clip (e.g., a television commercial) plays in the background and measures feelings that are congruent or incongruent with the content of the clip. The results of three experiments illustrate the following four advantages of IMPULSE over self-reported and biometric methods: (1) being less susceptible to typical confounds associated with explicit measures, (2) being easier to measure deep-seated and often nonconscious emotions, (3) being better able to detect a broad range of emotions and feelings, and (4) being more efficient to implement as an online method.Published versio

Alternation of Sound Location Induces Visual Motion Perception of a Static Object

Background: Audition provides important cues with regard to stimulus motion although vision may provide the most salient information. It has been reported that a sound of fixed intensity tends to be judged as decreasing in intensity after adaptation to looming visual stimuli or as increasing in intensity after adaptation to receding visual stimuli. This audiovisual interaction in motion aftereffects indicates that there are multimodal contributions to motion perception at early levels of sensory processing. However, there has been no report that sounds can induce the perception of visual motion. Methodology/Principal Findings: A visual stimulus blinking at a fixed location was perceived to be moving laterally when the flash onset was synchronized to an alternating left-right sound source. This illusory visual motion was strengthened with an increasing retinal eccentricity (2.5 deg to 20 deg) and occurred more frequently when the onsets of the audio and visual stimuli were synchronized. Conclusions/Significance: We clearly demonstrated that the alternation of sound location induces illusory visual motion when vision cannot provide accurate spatial information. The present findings strongly suggest that the neural representations of auditory and visual motion processing can bias each other, which yields the best estimates of externa

Dynamic Double Flash Illusion: Auditory Triggered Replay of Illusory Visual Expansion

In the original double flash illusion, a visual flash (e.g., a sharp-edged disk, or uniformly filled circle) presented with two short auditory tones (beeps) is often followed by an illusory flash. The illusory flash has been previously shown to be triggered by the second auditory beep. The current study extends the double flash illusion by showing that this paradigm can not only create the illusory repeat of an on-off flash, but also trigger an illusory expansion (and in some cases a subsequent contraction) that is induced by the flash of a circular brightness gradient (gradient disk) to replay as well. The perception of the dynamic double flash illusion further supports the interpretation of the illusory flash (in the double flash illusion) as similar in its spatial and temporal properties to the perception of the real visual flash, likely by replicating the neural processes underlying the illusory expansion of the real flash. We show further that if a gradient disk (generating an illusory expansion) and a sharp-edged disk are presented simultaneously side by side with two sequential beeps, often only one visual stimulus or the other will be perceived to double flash. This indicates selectivity in auditory–visual binding, suggesting the usefulness of this paradigm as a psychophysical tool for investigating crossmodal binding phenomena

What does not happen: quantifying embodied engagement using NIMI and self-adaptors

Previous research into the quantification of embodied intellectual and emotional engagement using non-verbal movement parameters has not yielded consistent results across different studies. Our research introduces NIMI (Non-Instrumental Movement Inhibition) as an alternative parameter. We propose that the absence of certain types of possible movements can be a more holistic proxy for cognitive engagement with media (in seated persons) than searching for the presence of other movements. Rather than analyzing total movement as an indicator of engagement, our research team distinguishes between instrumental movements (i.e. physical movement serving a direct purpose in the given situation) and non-instrumental movements, and investigates them in the context of the narrative rhythm of the stimulus. We demonstrate that NIMI occurs by showing viewers’ movement levels entrained (i.e. synchronised) to the repeating narrative rhythm of a timed computer-presented quiz. Finally, we discuss the role of objective metrics of engagement in future context-aware analysis of human behaviour in audience research, interactive media and responsive system and interface design