Credit assignment in multiple goal embodied visuomotor behavior

The intrinsic complexity of the brain can lead one to set aside issues related to its relationships with the body, but the field of embodied cognition emphasizes that understanding brain function at the system level requires one to address the role of the brain-body interface. It has only recently been appreciated that this interface performs huge amounts of computation that does not have to be repeated by the brain, and thus affords the brain great simplifications in its representations. In effect the brain’s abstract states can refer to coded representations of the world created by the body. But even if the brain can communicate with the world through abstractions, the severe speed limitations in its neural circuitry mean that vast amounts of indexing must be performed during development so that appropriate behavioral responses can be rapidly accessed. One way this could happen would be if the brain used a decomposition whereby behavioral primitives could be quickly accessed and combined. This realization motivates our study of independent sensorimotor task solvers, which we call modules, in directing behavior. The issue we focus on herein is how an embodied agent can learn to calibrate such individual visuomotor modules while pursuing multiple goals. The biologically plausible standard for module programming is that of reinforcement given during exploration of the environment. However this formulation contains a substantial issue when sensorimotor modules are used in combination: The credit for their overall performance must be divided amongst them. We show that this problem can be solved and that diverse task combinations are beneficial in learning and not a complication, as usually assumed. Our simulations show that fast algorithms are available that allot credit correctly and are insensitive to measurement noise

Hand eye coordination in surgery

The coordination of the hand in response to visual target selection has always been regarded as an essential quality in a range of professional activities. This quality has thus far been elusive to objective scientific measurements, and is usually engulfed in the overall performance of the individuals. Parallels can be drawn to surgery, especially Minimally Invasive Surgery (MIS), where the physical constraints imposed by the arrangements of the instruments and visualisation methods require certain coordination skills that are unprecedented. With the current paradigm shift towards early specialisation in surgical training and shortened focused training time, selection process should identify trainees with the highest potentials in certain specific skills. Although significant effort has been made in objective assessment of surgical skills, it is only currently possible to measure surgeons’ abilities at the time of assessment. It has been particularly difficult to quantify specific details of hand-eye coordination and assess innate ability of future skills development. The purpose of this thesis is to examine hand-eye coordination in laboratory-based simulations, with a particular emphasis on details that are important to MIS. In order to understand the challenges of visuomotor coordination, movement trajectory errors have been used to provide an insight into the innate coordinate mapping of the brain. In MIS, novel spatial transformations, due to a combination of distorted endoscopic image projections and the “fulcrum” effect of the instruments, accentuate movement generation errors. Obvious differences in the quality of movement trajectories have been observed between novices and experts in MIS, however, this is difficult to measure quantitatively. A Hidden Markov Model (HMM) is used in this thesis to reveal the underlying characteristic movement details of a particular MIS manoeuvre and how such features are exaggerated by the introduction of rotation in the endoscopic camera. The proposed method has demonstrated the feasibility of measuring movement trajectory quality by machine learning techniques without prior arbitrary classification of expertise. Experimental results have highlighted these changes in novice laparoscopic surgeons, even after a short period of training. The intricate relationship between the hands and the eyes changes when learning a skilled visuomotor task has been previously studied. Reactive eye movement, when visual input is used primarily as a feedback mechanism for error correction, implies difficulties in hand-eye coordination. As the brain learns to adapt to this new coordinate map, eye movements then become predictive of the action generated. The concept of measuring this spatiotemporal relationship is introduced as a measure of hand-eye coordination in MIS, by comparing the Target Distance Function (TDF) between the eye fixation and the instrument tip position on the laparoscopic screen. Further validation of this concept using high fidelity experimental tasks is presented, where higher cognitive influence and multiple target selection increase the complexity of the data analysis. To this end, Granger-causality is presented as a measure of the predictability of the instrument movement with the eye fixation pattern. Partial Directed Coherence (PDC), a frequency-domain variation of Granger-causality, is used for the first time to measure hand-eye coordination. Experimental results are used to establish the strengths and potential pitfalls of the technique. To further enhance the accuracy of this measurement, a modified Jensen-Shannon Divergence (JSD) measure has been developed for enhancing the signal matching algorithm and trajectory segmentations. The proposed framework incorporates high frequency noise filtering, which represents non-purposeful hand and eye movements. The accuracy of the technique has been demonstrated by quantitative measurement of multiple laparoscopic tasks by expert and novice surgeons. Experimental results supporting visual search behavioural theory are presented, as this underpins the target selection process immediately prior to visual motor action generation. The effects of specialisation and experience on visual search patterns are also examined. Finally, pilot results from functional brain imaging are presented, where the Posterior Parietal Cortical (PPC) activation is measured using optical spectroscopy techniques. PPC has been demonstrated to involve in the calculation of the coordinate transformations between the visual and motor systems, which establishes the possibilities of exciting future studies in hand-eye coordination

Concurrent adaptation to opposing visual displacements during an alternating movement.

It has been suggested that, during tasks in which subjects are exposed to a visual rotation of cursor feedback, alternating bimanual adaptation to opposing rotations is as rapid as unimanual adaptation to a single rotation (Bock et al. in Exp Brain Res 162:513–519, 2005). However, that experiment did not test strict alternation of the limbs but short alternate blocks of trials. We have therefore tested adaptation under alternate left/right hand movement with opposing rotations. It was clear that the left and right hand, within the alternating conditions, learnt to adapt to the opposing displacements at a similar rate suggesting that two adaptive states were formed concurrently. We suggest that the separate limbs are used as contextual cues to switch between the relevant adaptive states. However, we found that during online correction the alternating conditions had a significantly slower rate of adaptation in comparison to the unimanual conditions. Control conditions indicate that the results are not directly due the alternation between limbs or to the constant switching of vision between the two eyes. The negative interference may originate from the requirement to dissociate the visual information of these two alternating displacements to allow online control of the two arms

L’influence de l'anticipation sur les modulations de puissance dans la bande de fréquence bêta durant la préparation du mouvement et L'effet de la variance dans les rétroactions sensorielles sur la rétention à court terme

La production du mouvement est un aspect primordial de la vie qui permet aux organismes vivants d'interagir avec l'environnement. En ce sens, pour être efficaces, tous les mouvements doivent être planifiés et mis à jour en fonction de la complexité et de la variabilité de l'environnement. Des chercheurs du domaine du contrôle moteur ont étudié de manière approfondie les processus de planification et d’adaptation motrice. Puisque les processus de planification et d'adaptation motrice sont influencés par la variabilité de l'environnement, le présent mémoire cherche à fournir une compréhension plus profonde de ces deux processus moteurs à cet égard. La première contribution scientifique présentée ici tire parti du fait que les temps de réaction (TR) sont réduits lorsqu'il est possible d'anticiper l’objectif moteur, afin de déterminer si les modulations de TR associées à l'anticipation spatiale et temporelle sont sous-tendues par une activité préparatoire similaire. Cela a été fait en utilisant l'électroencéphalographie (EEG) de surface pour analyser l'activité oscillatoire dans la bande de fréquence bêta (13 - 30 Hz) au cours de la période de planification du mouvement. Les résultats ont révélé que l'anticipation temporelle était associée à la désynchronisation de la bande bêta au-dessus des régions sensorimotrices controlatérales à la main effectrice, en particulier autour du moment prévu de l'apparition de la cible. L’ampleur de ces modulations était corrélée aux modulations de TR à travers les participants. En revanche, l'anticipation spatiale a augmenté de manière sélective la puissance de la bande bêta au-dessus des régions pariéto-occipitales bilatérales pendant toute la période de planification. Ces résultats suggèrent des états de préparation distinct en fonction de l’anticipation temporelle et spatiale. D’un autre côté, le deuxième projet traite de la façon dont la variabilité de la rétroaction sensorielle interfère avec la rétention à court terme dans l’étude de l’adaptation motrice. Plus précisément, une tâche d'adaptation visuomotrice a été utilisée au cours de laquelle la variance des rotations a été manipulée de manière paramétrique à travers trois groupes, et ce, tout au long de la période d’acquisition. Par la suite, la rétention de cette nouvelle relation visuomotrice a été évaluée. Les résultats ont révélé que, même si le processus d'adaptation était robuste à la manipulation de la variance, la rétention à court terme était altérée par des plus hauts niveaux de variance. Finalement, la discussion a d'abord cherché à intégrer ces deux contributions en revisitant l'interprétation des résultats sous un angle centré sur l'incertitude et en fournissant un aperçu des potentielles représentations internes de l'incertitude susceptibles de sous-tendre les résultats expérimentaux observés. Par la suite, une partie de la discussion a été réservée à la manière dont le champ du contrôle moteur migre de plus en plus vers l’utilisation de tâches et d’approches expérimentales plus complexes, mais écologiques aux dépends des tâches simples, mais quelque peu dénaturées que l’on retrouve dans les laboratoires du domaine. La discussion a été couronnée par une brève proposition allant dans ce sens.Abstract: Motor behavior is a paramount aspect of life that enables the living to interact with the environment through the production of movement. In order to be efficient, movements need to be planned and updated according to the complexity and the ever-changing nature of the environment. Motor control experts have extensively investigated the planning and adaptation processes. Since both motor planning and motor adaptation processes are influenced by variability in the environment, the present thesis seeks to provide a deeper understanding of both these motor processes in this regard. More specifically, the first scientific contribution presented herein leverages the fact that reaction times (RTs) are reduced when the anticipation of the motor goal is possible to elucidate whether the RT modulations associated with temporal and spatial anticipation are subtended by similar preparatory activity. This was done by using scalp electroencephalography (EEG) to analyze the oscillatory activity in the beta frequency band (13 – 30 Hz) during the planning period. Results revealed that temporal anticipation was associated with beta-band desynchronization over contralateral sensorimotor regions, specifically around the expected moment of target onset, the magnitude of which was correlated with RT modulations across participants. In contrast, spatial anticipation selectively increased beta-band power over bilateral parieto-occipital regions during the entire planning period, suggesting that distinct states of preparation are incurred by temporal and spatial anticipation. Additionally, the second project addressed how variance in the sensory feedback interferes with short-term retention of motor adaptation. Specifically, a visuomotor adaptation task was used during which the variance of exposed rotation was parametrically manipulated across three groups, and retention of the adapted visuomotor relationship was assessed. Results revealed that, although the adaptation process was robust to the manipulation of variance, the short-term retention was impaired. The discussion first sought to integrate these two projects by revisiting the interpretation of both projects under the scope of uncertainty and by providing an overview of the internal representation of uncertainty that might subtend the experimental results. Subsequently, a part of the discussion was reserved to allude how the motor control field is transitioning from laboratory-based tasks to more naturalistic paradigms by using approaches to move motor control research toward real-world conditions. The discussion culminates with a brief scientific proposal along those lines

kPAM 2.0: Feedback Control for Category-Level Robotic Manipulation

In this paper, we explore generalizable, perception-to-action robotic manipulation for precise, contact-rich tasks. In particular, we contribute a framework for closed-loop robotic manipulation that automatically handles a category of objects, despite potentially unseen object instances and significant intra-category variations in shape, size and appearance. Previous approaches typically build a feedback loop on top of a real-time 6-DOF pose estimator. However, representing an object with a parameterized transformation from a fixed geometric template does not capture large intra-category shape variation. Hence we adopt the keypoint-based object representation proposed in kPAM for category-level pick-and-place, and extend it to closed-loop manipulation policies with contact-rich tasks. We first augment keypoints with local orientation information. Using the oriented keypoints, we propose a novel object-centric action representation in terms of regulating the linear/angular velocity or force/torque of these oriented keypoints. This formulation is surprisingly versatile -- we demonstrate that it can accomplish contact-rich manipulation tasks that require precision and dexterity for a category of objects with different shapes, sizes and appearances, such as peg-hole insertion for pegs and holes with significant shape variation and tight clearance. With the proposed object and action representation, our framework is also agnostic to the robot grasp pose and initial object configuration, making it flexible for integration and deployment.Comment: IEEE Robotics and Automation Letter. The video demo is on https://sites.google.com/view/kpam2

Flexible Cognitive Strategies during Motor Learning

Visuomotor rotation tasks have proven to be a powerful tool to study adaptation of the motor system. While adaptation in such tasks is seemingly automatic and incremental, participants may gain knowledge of the perturbation and invoke a compensatory strategy. When provided with an explicit strategy to counteract a rotation, participants are initially very accurate, even without on-line feedback. Surprisingly, with further testing, the angle of their reaching movements drifts in the direction of the strategy, producing an increase in endpoint errors. This drift is attributed to the gradual adaptation of an internal model that operates independently from the strategy, even at the cost of task accuracy. Here we identify constraints that influence this process, allowing us to explore models of the interaction between strategic and implicit changes during visuomotor adaptation. When the adaptation phase was extended, participants eventually modified their strategy to offset the rise in endpoint errors. Moreover, when we removed visual markers that provided external landmarks to support a strategy, the degree of drift was sharply attenuated. These effects are accounted for by a setpoint state-space model in which a strategy is flexibly adjusted to offset performance errors arising from the implicit adaptation of an internal model. More generally, these results suggest that strategic processes may operate in many studies of visuomotor adaptation, with participants arriving at a synergy between a strategic plan and the effects of sensorimotor adaptation

Sensorimotor Differences in Autism Spectrum Disorder: An evaluation of potential mechanisms.

This thesis examined the aetiology of sensorimotor impairments in Autism Spectrum Disorder: a neurodevelopmental condition that affects an individual’s socio-behavioural preferences, personal independence, and quality of life. Issues relating to clumsiness and movement coordination are common features of autism that contribute to wide-ranging daily living difficulties. However, these characteristics are relatively understudied and there is an absence of evidence-based practical interventions. To pave the way for new, scientifically-focused programmes, a series of studies investigated the mechanistic underpinnings of sensorimotor differences in autism. Following a targeted review of previous research, study one explored links between autistic-like traits and numerous conceptually-significant movement control functions. Eye-tracking analyses were integrated with force transducers and motion capture technology to examine how participants interacted with uncertain lifting objects. Upon identifying a link between autistic-like traits and context-sensitive predictive action control, study two replicated these procedures with a sample of clinically-diagnosed participants. Results illustrated that autistic people are able to use predictions to guide object interactions, but that uncertainty-related adjustments in sensorimotor integration are atypical. Such findings were advanced within a novel virtual-reality paradigm in study three, which systematically manipulated environmental uncertainty during naturalistic interception actions. Here, data supported proposals that precision weighting functions are aberrant in autistic people, and suggested that these individuals have difficulties with processing volatile sensory information. These difficulties were not alleviated by the experimental provision of explicit contextual cues in study four. Together, these studies implicate the role of implicit neuromodulatory mechanisms that regulate dynamic sensorimotor behaviours. Results support the development of evidence-based programmes that ‘make the world more predictable’ for autistic people, with various theoretical and practical implications presented. Possible applications of these findings are discussed in relation to recent multi-disciplinary research and conceptual advances in the field, which could help improve daily living skills and functional quality of life.Economic and Social Research Council (ESRC

Extrinsic and intrinsic dynamics in visuomotor tracking

PhD ThesisHumans typically produce 2–3 submovements per second when tracking slow targets. This intermittency is altered by the addition of delays in sensory feedback suggesting that it is governed by extrinsic properties of the control loop. However, the motor cortex also exhibits an intrinsic rhythmicity at 2–3 Hz, which might influence the temporal structure of movements. This thesis examines how the interplay between extrinsic and intrinsic dynamics shapes the kinematics of tracking behaviour. I found that the dependence of submovement frequencies on extrinsic delays could be reproduced by a simple feedback controller model. This model predicted that submovements reflect frequencies at which visuomotor noise is exacerbated, and this was confirmed by perturbation experiments. However, these experiments also revealed a 2-3 Hz band-pass filtering of feedback responses irrespective of extrinsic delay. Further experimental evidence suggested this filter did not reflect properties of either visuomotor noise, the feedforward pathway, or visual processing. However, the filter exhibited features consistent with a state estimator required for optimal feedback control (OFC) in the presence of visual and motor noise. Finally, I sought evidence that this filter was implemented by motor cortical circuits. Multichannel local field potentials (LFPs) in the motor cortex of macaque monkeys were strongly correlated with submovements, at frequencies which depended on extrinsic delay. However, the dynamics of LFP cycles during submovements were independent of delay, and matched instead the properties of the state estimator in the OFC model. In summary, by combining human behavioural studies, computational modelling and monkey electrophysiology, I show how movement intermittency can be explained by the interplay of both extrinsic and intrinsic dynamics within an OFC framework. Moreover, I suggest that motor cortical rhythmicity reflects recurrent circuitry that combines sensory feedback with an internal dynamical model to form optimal estimates of required motor corrections.Indonesian Endowment Fund for Education (Lembaga Pengelola Dana Pendidikan Republik Indonesia) for supporting my PhD studies. This work would also not have been possible without the financial support of the Wellcome Trust, and the Medical Research Council