The effect of action on perceptual feature binding

Color-motion asynchrony (CMA) refers to an apparent lag of direction of motion when a dynamic stimulus changes both color and direction at the same time. The subjective order of simultaneous events, however, is not only perceptual but also subject to illusions during voluntary actions. Self-initiated actions, for example, seem to precede their sensory outcomes following an adaptation to a delay between the action and the sensory feedback. Here, we demonstrate that the extent of the apparent asynchrony can be substantially reduced when direction change is induced by a voluntary key press following a delay adaptation regime. We also show that the reduced color-motion asynchrony effect size following a motor-sensory recalibration is not a result of a change in the onset of perceived direction change relative to that of the color. This is particularly important as it implies, for the first time in the literature, that voluntary action is not only important in forming action-sensory outcome associations but may also act as a binding factor between the two perceptual features of a sensory event

How voluntary actions modulate time perception

Distortions of time perception are generally explained either by variations in the rate of pacing signals of an “internal clock”, or by lag-adaptation mechanisms that recalibrate the perceived time of one event relative to another. This study compares these accounts directly for one temporal illusion: the subjective compression of the interval between voluntary actions and their effects, known as ‘intentional binding’. Participants discriminated whether two cutaneous stimuli presented after voluntary or passive movements were simultaneous or successive. In other trials, they judged the temporal interval between their movement and an ensuing tone. Temporal discrimination was impaired following voluntary movements compared to passive movements early in the action-tone interval. In a control experiment, active movements without subsequent tones produced no impairment in temporal discrimination. These results suggest that voluntary actions transiently slow down an internal clock during the action-effect interval. This in turn leads to intentional binding, and links the effects of voluntary actions to the self

The perceived timing of events across different sensory modalities. A psychophysical investigation of multisensory time perception in humans.

The experiments reported within this thesis use psychophysical techniques to examine the factors which determine perceived multisensory timing in humans. Chapters 1 and 2 describe anatomical and psychophysical features of temporal processing, respectively, whilst Chapter 3 introduces the reader to psychophysical methods. Chapter 4 examines the relationship between two measures of sensory latency, reaction time (RT) and crossmodal temporal order judgment (TOJ). Despite task and attentional manipulations the two measures do not correlate, suggesting that they measure some fundamentally different aspect(s) of temporal perception. Chapter 5 examines the effects of adaptation to asynchronous stimulus pairs on perceived audiovisual (AV), audiotactile (AT) and visuotactile (VT) temporal order. Significant temporal shifts are recorded in all three conditions. Evidence is also presented showing that crossmodal TOJs are intransitive. Chapter 6 shows that concurrent adaptation to two sets of asynchronous AV stimulus pairs causes perceived AV temporal order to recalibrate at two locations simultaneously, and that AV asynchrony adaptation effects are significantly affected by observers¿ attention during adaptation. Finally, Chapter 7 shows that when observers are accustomed to a physical delay between motor actions and sensory events, an event presented at a reduced delay appears to precede the causative motor action. The data are well-described by a simple model based on a strong prior assumption of physical synchrony between motor actions and their sensory consequences

Mental and sensorimotor extrapolation fare better than motion extrapolation in the offset condition

Evidence for motion extrapolation at motion offset is scarce. In contrast, there is abundant evidence that subjects mentally extrapolate the future trajectory of weak motion signals at motion offset. Further, pointing movements overshoot at motion offset. We believe that mental and sensorimotor extrapolation is sufficient to solve the problem of perceptual latencies. Both present the advantage of being much more flexible than motion extrapolatio

Behavioral and Neural Indices of Perceiving Multisensory Action Outcomes

Distinct perception for voluntary vs. externally-generated action outcomes has been demonstrated in timing and intensity domains. First, time interval between an action and its outcome is perceived shorter. Second, sensory stimuli triggered by one’s own action is judged as less intense than similar stimuli triggered externally. The differential perception of voluntary action outcomes has been attributed to efference copy-related predictive mechanisms, and has been studied extensively using behavioral and imaging methods. However, although voluntary movements in the real world produce feedback in multiple modalities, previous experiments mostly investigated unimodal action outcomes. Therefore, the perception of multisensory inputs associated with our own actions remains to be explored. The aim of this dissertation was to fill this gap by investigating the behavioral and neural correlates of multisensory action outcomes. In Study 1, synchrony perception for multisensory outcomes triggered by voluntary vs. externally-generated movements was assessed. Study 1.1 showed increased perception of simultaneity for audiovisual stimulus pairs around the time of action. Study 1.2 revealed a similar effect also when the movement was externally-generated, underlining the importance of causal relations between events in shaping time perception. Interestingly, the slopes of the psychometric functions in the voluntary condition were significantly steeper than the slopes in the externally-generated condition, suggesting a role of action-related predictive mechanisms in making synchrony perception more precise. Study 2 investigated the neural correlates of perceiving unimodal vs. bimodal inputs triggered by voluntary button presses compared with passive viewing of identical stimuli. Results showed BOLD suppression for voluntary action outcomes in comparison to passive viewing of the same stimuli. This BOLD suppression effect was independent of the to-be-attended modality and the number of modalities presented. The cerebellum was found to be recruited more during bimodal trials and when a delay was detected. These findings support action-related predictive processing of voluntary action outcomes, demonstrating it also for multisensory action outcomes. The findings also indicate the cerebellum’s role in error-related action outcome processing, and the influence of the additional sensory modality on error-related activity in the cerebellum. Study 3 investigated neural correlates of perceiving unimodal vs. bimodal action outcomes by focusing on efference copy-related predictive processing in a naturalistic experimental set- up. Results extend findings of Study 2 regarding the predictive processing of multisensory action outcomes to a naturalistic context, and support the role of the cerebellum in error- related action outcome processing. Importantly, activity in the cerebellum was modulated by the additional modality, highlighting the role of multisensory processing in shaping motor- sensory interactions. Together, findings of these studies strengthen existing evidence on the distinctive perception for voluntary action outcomes, extending it to multisensory action outcomes, and to a realistic context. Implications of this line of research extend to revealing mechanisms behind agency deficits frequently observed in schizophrenia, as well as to the development of intervention techniques targeting the rehabilitation of patients with spinal cord injury or stroke

Involvement of cerebellar Purkinje cells in adaptive locomotion of larval zebrafish

Animals need to adapt their behavior to survive and be successful in a constantly changing environment. Behavioral adaptations can be evoked by two mechanisms: feedback control and internal models. A feedback controller compares current sensory state with desired state and generates a motor output that minimizes their difference; such simple controllers can produce adaptive behavior without changing their own intrinsic parameters. In the central nervous system, however, feedback control is limited by long temporal delays associated with sensory processing required to estimate current sensory state. To overcome this limitation, internal models learn previous sensory-motor history to update parameters of motor control in a predictive manner. In the present study, I use multiple perturbations in visual feedback to show that larval zebrafish acutely adapt their swimming behavior to these perturbations. These acute behavioral changes do not affect an initial stereotyped ballistic portion of the swimming bouts that lasts ~ 220 ms and are unaffected by a pharmaco-genetic ablation of Purkinje cells – the major locus of internal models, suggesting that acute adaptation results from a feedback control mechanism. I support this hypothesis by modelling a simple feedback controller that is based on temporal integration of sensory evidence. The controller is able to closely reproduce all observed aspects of acute adaptation. The main assumption of the model: existence of temporal sensory integration in the larval zebrafish brain is supported by whole-brain functional imaging. On the other hand, during long-term adaptation, larval zebrafish gradually change their behavior, including their swimming kinematics during the ballistic period. In contrast with the acute short-term changes, these behavioral alterations are cerebellum-dependent. In conclusion, adaptive locomotion in larval zebrafish should be understood as a feedback control system whose intrinsic parameters can be modified by cerebellar output.Tiere müssen ihr Verhalten anpassen, um in einem sich verändernden Lebensraum zu überleben. Diese Anpassungen im Verhalten können durch zwei Mechanismen hervorgerufen werden: Rückkopplung und interne Modelle. Ein Regelkreis basierend auf sensorischer Rückkopplung vergleicht den Ist-Zustand mit dem Soll-Zustand und generiert dementsprechend ein Motorsignal, das die Differenz der Zustände minimiert. Solche einfachen Regler können ohne die Veränderung von internen Parametern adaptives Verhalten hervorrufen. Im zentralen Nervensystem sind jedoch diese Regelkreise limitiert durch lange zeitliche Verzögerungen aufgrund der Berechnung des sensorischen Ist-Zustandes. Um diese Einschränkungen zu umgehen, können interne Modelle verwendet werden, die mit Hilfe vorangegangener sensomotorischer Übersetzungen Motorparameter vorhersagen. In dieser Studie verwende ich mehrere Veränderungen in der visuellen Rückkopplung, um zu zeigen, dass Zebrafischlarven ihr Schwimmverhalten aufgrund dieser Veränderungen akut anpassen. Diese akuten Anpassungen im Verhalten beeinflussen allerdings nicht die stereotypen, ballistischen Eigenschaften einzelner Schwimm-Ereignisse („bouts“), die ca. 220 ms andauern. Des Weiteren sind diese beeinträchtigt durch pharmakologisch-genetische Ablationen von Purkinje-Zellen, welche einen Hauptort interner Modelle bilden, was darauf hinweist, dass akute Anpassungen von klassischen Rückkopplungsmechanismen gesteuert werden. Zur Unterstützung dieser Hypothese stelle ich ein Modell eines einfachen Rückkopplungsreglers vor, welches auf der zeitlichen Integration sensorischer Informationen basiert. Dieses Modell kann fast alle beobachteten Aspekte akuter Verhaltensanpassung reproduzieren. Die Hauptannahme des Modells, dass eine zeitliche Integration sensorischer Information im Gehirn von Zebrafischlarven existiert, wird durch funktionelle Bildgebung des gesamten Gehirns unterstützt. Dem gegenüber steht, dass Zebrafischlarven ihr Verhalten während einer Langzeitadaptionsstudie, inklusive der Kinematik ihres Schwimmverhaltens, graduell anpassen. Im Kontrast zu akuten Verhaltensanpassungen sind diese Verhaltensänderungen abhängig vom Kleinhirn. Zusammenfassend kann adaptives Verhalten während der Fortbewegung in Zebrafischlarven als Rückkopplungssystem verstanden werden, dessen intrinsische Parameter durch Signale des Kleinhirns modifiziert werden

Interfacce cervello-computer per la comunicazione aumentativa: algoritmi asincroni e adattativi e validazione con utenti finali

This thesis aimed at addressing some of the issues that, at the state of the art, avoid the P300-based brain computer interface (BCI) systems to move from research laboratories to end users’ home. An innovative asynchronous classifier has been defined and validated. It relies on the introduction of a set of thresholds in the classifier, and such thresholds have been assessed considering the distributions of score values relating to target, non-target stimuli and epochs of voluntary no-control. With the asynchronous classifier, a P300-based BCI system can adapt its speed to the current state of the user and can automatically suspend the control when the user diverts his attention from the stimulation interface. Since EEG signals are non-stationary and show inherent variability, in order to make long-term use of BCI possible, it is important to track changes in ongoing EEG activity and to adapt BCI model parameters accordingly. To this aim, the asynchronous classifier has been subsequently improved by introducing a self-calibration algorithm for the continuous and unsupervised recalibration of the subjective control parameters. Finally an index for the online monitoring of the EEG quality has been defined and validated in order to detect potential problems and system failures. This thesis ends with the description of a translational work involving end users (people with amyotrophic lateral sclerosis-ALS). Focusing on the concepts of the user centered design approach, the phases relating to the design, the development and the validation of an innovative assistive device have been described. The proposed assistive technology (AT) has been specifically designed to meet the needs of people with ALS during the different phases of the disease (i.e. the degree of motor abilities impairment). Indeed, the AT can be accessed with several input devices either conventional (mouse, touchscreen) or alterative (switches, headtracker) up to a P300-based BCI.Questa tesi affronta alcune delle problematiche che, allo stato dell'arte, limitano l'usabilità delle interfacce cervello computer (Brain Computer Interface - BCI) al di fuori del contesto sperimentale. E' stato inizialmente definito e validato un classificatore asincrono. Quest'ultimo basa il suo funzionamento sull'inserimento di un set di soglie all'interno del classificatore. Queste soglie vengono definite considerando le distribuzioni dei valori di score relativi agli stimoli target e non-target e alle epoche EEG in cui il soggetto non intendeva effettuare nessuna selezione (no-control). Con il classificatore asincrono, un BCI basato su potenziali P300 può adattare la sua velocità allo stato corrente dell'utente e sospendere automaticamente il controllo quando l'utente non presta attenzione alla stimolazione. Dal momento che i segnali EEG sono non-stazionari e mostrano una variabilità intrinseca, al fine di rendere possibile l'utilizzo dei sistemi BCI sul lungo periodo, è importante rilevare i cambiamenti dell'attività EEG e adattare di conseguenza i parametri del classificatore. A questo scopo, il classificatore asincrono è stato successivamente migliorato introducendo un algoritmo di autocalibrazione per la continua e non supervisionata ricalibrazione dei parametri di controllo soggettivi. Infine è stato definito e validato un indice per monitorare on-line la qualità del segnale EEG, in modo da rilevare potenziali problemi e malfunzionamenti del sistema. Questa tesi si conclude con la descrizione di un lavoro che ha coinvolto gli utenti finali (persone affette da sclerosi laterale amiotrofica-SLA). In particolare, basandosi sui principi dell’user-centered design, sono state descritte le fasi relative alla progettazione, sviluppo e validazione di una tecnologia assistiva (TA) innovativa. La TA è stata specificamente progettata per rispondere alla esigenze delle persone affetta da SLA durante le diverse fasi della malattia. Infatti, la TA proposta può essere utilizzata sia mediante dispositivi d’input tradizionali (mouse, tastiera) che alternativi (bottoni, headtracker) fino ad arrivare ad un BCI basato su potenziali P300

Fusion of virtual reality and brain-machine interfaces for the assessment and rehabilitation of patients with spinal cord injury

La presente tesis está centrada en la utilización de nuevas tecnologías (Interfaces Cerebro-Máquina y Realidad Virtual). En la primera parte de la tesis se describe la definición y la aplicación de un conjunto de métricas para evaluar el estado funcional de los pacientes con lesión medular en el contexto de un sistema de realidad virtual para la rehabilitación de los miembros superiores. El objetivo de este primer estudio es demostrar que la realidad virtual puede utilizarse, en combinación con sensores inerciales para rehabilitar y evaluar simultáneamente. 15 pacientes con lesión medular llevaron a cabo 3 sesiones con el sistema de realidad virtual Toyra y se aplicó el conjunto definido de métricas a las grabaciones obtenidas con los sensores inerciales. Se encontraron correlaciones entre algunas de las métricas definidas y algunas de las escalas clínicas utilizadas con frecuencia en el contexto de la rehabilitación. En la segunda parte de la tesis se ha combinado una retroalimentación virtual con un estimulador eléctrico funcional (en adelante FES, por sus siglas en inglés Functional Electrical Stimulator), ambos controlados por un Interfaz Cerebro-Máquina (BMI por sus siglas en inglés Brain-Machine Interface), para desarrollar un nuevo tipo de enfoque terapéutico para los pacientes. El sistema ha sido utilizado por 4 pacientes con lesión medular que intentaron mover sus manos. Esta intención desencadenó simultáneamente el FES y la retroalimentación virtual, cerrando la mano de los pacientes y mostrándoles una fuente adicional de retroalimentación para complementar la terapia. Este trabajo es, de acuerdo al estado del arte revisado, el primero que integra BMI, FES y realidad virtual como terapia para pacientes con lesión medular. Se han obtenido resultados clínicos prometedores por 4 pacientes con lesión medular después de realizar 5 sesiones de terapia con el sistema, mostrando buenos niveles de precisión en las diferentes sesiones (79,13% en promedio). En la tercera parte de la tesis se ha definido una nueva métrica para estudiar los cambios de conectividad cerebral en los pacientes con lesión medular, que incluye información de las interacciones neuronales entre diferentes áreas. El objetivo de este estudio ha sido extraer información clínicamente relevante de la actividad del EEG cuando se realizan terapias basadas en BMI

Too Late! Influence of Temporal Delay on the Neural Processing of One’s Own Incidental and Intentional Action-Induced Sounds

The influence of delayed auditory feedback on action evaluation and execution of real-life action-induced sounds apart from language and music is still poorly understood. Here, we examined how a temporal delay impacted the behavioral evaluation and neural representation of hurdling and tap-dancing actions in a functional magnetic resonance imaging (fMRI) experiment, postulating that effects of delay diverge between the two, as we create action-induced sounds intentionally in tap dancing, but incidentally in hurdling. Based on previous findings, we expected that conditions differ regarding the engagement of the supplementary motor area (SMA), posterior superior temporal gyrus (pSTG), and primary auditory cortex (A1). Participants were videotaped during a 9-week training of hurdling and tap dancing; in the fMRI scanner, they were presented with point-light videos of their own training videos, including the original or the slightly delayed sound, and had to evaluate how well they performed on each single trial. For the undelayed conditions, we replicated A1 attenuation and enhanced pSTG and SMA engagement for tap dancing (intentionally generated sounds) vs. hurdling (incidentally generated sounds). Delayed auditory feedback did not negatively influence behavioral rating scores in general. Blood-oxygen-level-dependent (BOLD) response transiently increased and then adapted to repeated presentation of point-light videos with delayed sound in pSTG. This region also showed a significantly stronger correlation with the SMA under delayed feedback. Notably, SMA activation increased more for delayed feedback in the tap-dancing condition, covarying with higher rating scores. Findings suggest that action evaluation is more strongly based on top–down predictions from SMA when sounds of intentional action are distorted

Sex-Differences in the Early Detection of Dementia Risk Using a Cognitive-motor Integration Task

Cognitive-motor integration (CMI) involves concurrent thought and action which requires the interaction of large brain networks. Our research objectives were to examine the effect that dementia risk has on the ability to integrate rules into action and to investigate sex-related differences in this rule-based motor performance. Given that early-stage dementia involves neural network dysfunction, problems with CMI may prove useful for early dementia detection. Males and females at high- and low-dementia risk were tested on increasingly spatially-dissociated visuomotor tasks. We observed significantly greater endpoint error scores and corrective path lengths in females compared to males in the most complex CMI condition. These data suggest that underlying brain networks controlling simultaneous thought and action differ between the sexes, and that dementia risk may affect female CMI performance to a greater extent. Thus, sex-related differences must be taken into account when assessing CMI performance as a means to examine dementia risk-related functional abilities