Neural substrates of reward, error, and effort processing underlying adaptive motor behaviour

Human motor control is highly adaptive to new tasks and changing environments. Motor adaptation relies on multiple dissociable processes that function to increase attainment of reward and to reduce sensory error and physical effort as costs. This thesis tests the hypothesis that fronto-striatal and dopaminergic neural systems contribute to specific aspects of motor adaptation that occur through reinforcement of rewarding actions. Behavioral tasks were designed to isolate learning in response to feedback conveying information about reward, error, and physical effort. We also measured behavioral effects of savings and anterograde interference, by which memories from previous motor learning can facilitate or impair subsequent learning. Electroencephalography (EEG) was used to record neural event-related potentials (ERPs) elicited by task-related feedback. We measured the feedback-related negativity/ reward positivity (FRN/RP), a midfrontal component of ERP responses to feedback stimuli that correlates with neural activity throughout fronto-striatal circuits. Levodopa, a dopamine precursor, was used to manipulate dopamine release in healthy volunteers, as it has been shown to impair reward-based learning in various cognitive tasks. We first determined that medial frontal feedback processing indexed by the FRN/RP is a specific neural correlate of reward prediction error during motor adaptation, and that the FRN/RP is not elicited by sensory error. Next, we found that levodopa did not affect either the FRN/RP, reward-based motor adaptation, savings, or anterograde interference. Finally, we determined that medial frontal activity indexed by the FRN/RP does not respond to physical effort as a cost that discounts the value of reward. However, effort increased neural sensitivity to reinforcement outcomes in activity measured by a midfrontal ERP component that was spatially and temporally distinct from the FRN/RP. These findings suggest that mid-frontal feedback processing measured by the FRN/RP may play a specific role in reward-based motor learning that is distinct from error- and effort-based learning processes. Our findings also indicate that reinforcement learning mechanisms that contribute to motor adaptation do not depend on the same dopaminergic processes that are impaired by levodopa in cognitive learning tasks

The gradient of the reinforcement landscape influences sensorimotor learning

© 2019 Cashaback et al. Consideration of previous successes and failures is essential to mastering a motor skill. Much of what we know about how humans and animals learn from such reinforcement feedback comes from experiments that involve sampling from a small number of discrete actions. Yet, it is less understood how we learn through reinforcement feedback when sampling from a continuous set of possible actions. Navigating a continuous set of possible actions likely requires using gradient information to maximize success. Here we addressed how humans adapt the aim of their hand when experiencing reinforcement feedback that was associated with a continuous set of possible actions. Specifically, we manipulated the change in the probability of reward given a change in motor action-the reinforcement gradient-to study its influence on learning. We found that participants learned faster when exposed to a steep gradient compared to a shallow gradient. Further, when initially positioned between a steep and a shallow gradient that rose in opposite directions, participants were more likely to ascend the steep gradient. We introduce a model that captures our results and several features of motor learning. Taken together, our work suggests that the sensorimotor system relies on temporally recent and spatially local gradient information to drive learning

Control of movement EEG correlates of physical effort and reward processing during reinforcement learning

Copyright © 2020 the American Physiological Society. Effort-based decision making is often modeled using subjective value, a function of reward discounted by effort. We asked whether EEG event-related potential (ERP) correlates of reward processing are also modulated by physical effort. Human participants performed a task in which they were required to accurately produce target levels of muscle activation to receive rewards. Quadriceps muscle activation was recorded with electromyography (EMG) during isometric knee extension. On a given trial, the target muscle activation required either low or high effort. The effort was determined probabilistically according to a binary choice, such that the responses were associated with 20% and 80% probability of high effort. This contingency could only be learned through experience, and it reversed periodically. Binary reinforcement feedback depended on accurately producing the target muscle activity. Participants adaptively avoided effort by switching responses more frequently after choices that resulted in hard effort. Feedback after participants’ choices that revealed the resulting effort requirement did not elicit modulation of the feedback-related negativity/reward positivity (FRN/RP). However, the neural response to reinforcement outcome after effort production was increased by preceding physical effort. Source decomposition revealed separable early and late positive deflections contributing to the ERP. The main effect of reward outcome, characteristic of the FRN/RP, loaded onto the earlier component, whereas the reward effort interaction was observed only in the later positivity, which resembled the P300. Thus, retrospective effort modulates reward processing. This may explain paradoxical behavioral findings whereby rewards requiring more effort to obtain can become more powerful reinforcers

Neural signatures of reward and sensory error feedback processing in motor learning

© 2019 the American Physiological Society. At least two distinct processes have been identified by which motor commands are adapted according to movement-related feedback: reward-based learning and sensory error-based learning. In sensory error-based learning, mappings between sensory targets and motor commands are recalibrated according to sensory error feedback. In reward-based learning, motor commands are associated with subjective value, such that successful actions are reinforced. We designed two tasks to isolate reward- and sensory error-based motor adaptation, and we used electroencephalography in humans to identify and dissociate the neural correlates of reward and sensory error feedback processing. We designed a visuomotor rotation task to isolate sensory error-based learning that was induced by altered visual feedback of hand position. In a reward learning task, we isolated reward-based learning induced by binary reward feedback that was decoupled from the visual target. A fronto-central eventrelated potential called the feedback-related negativity (FRN) was elicited specifically by binary reward feedback but not sensory error feedback. A more posterior component called the P300 was evoked by feedback in both tasks. In the visuomotor rotation task, P300 amplitude was increased by sensory error induced by perturbed visual feedback and was correlated with learning rate. In the reward learning task, P300 amplitude was increased by reward relative to nonreward and by surprise regardless of feedback valence. We propose that during motor adaptation the FRN specifically reflects a reward-based learning signal whereas the P300 reflects feedback processing that is related to adaptation more generally. NEW & NOTEWORTHY We studied the event-related potentials evoked by feedback stimuli during motor adaptation tasks that isolate reward- and sensory error-based learning mechanisms. We found that the feedback-related negativity was specifically elicited by binary reward feedback, whereas the P300 was observed in both tasks. These results reveal neural processes associated with different learning mechanisms and elucidate which classes of errors, from a computational standpoint, elicit the feedback-related negativity and P300

Effects of speed on smooth pursuit eye movements.

<p>Shown are means and standard deviations for slow and fast speed and results of repeated-measures ANOVA with speed as factor.</p

Experimental timeline and set-up.

<p>A. Trial timeline of computer-based DVA test. Each trial starts with peripheral fixation, followed by step-ramp stimulus motion; target is a Landolt-C ring. Observers performed a judgment about the location of the gap in the “C” (one of 4 locations) at the end of each trial via button press. B. Set-up for computerized dynamic-object DVA test and eye tracking. C. Set-up for clinical static-object DVA test.</p

Distinct eye movement patterns enhance dynamic visual acuity

<div><p>Dynamic visual acuity (DVA) is the ability to resolve fine spatial detail in dynamic objects during head fixation, or in static objects during head or body rotation. This ability is important for many activities such as ball sports, and a close relation has been shown between DVA and sports expertise. DVA tasks involve eye movements, yet, it is unclear which aspects of eye movements contribute to successful performance. Here we examined the relation between DVA and the kinematics of smooth pursuit and saccadic eye movements in a cohort of 23 varsity baseball players. In a computerized dynamic-object DVA test, observers reported the location of the gap in a small Landolt-C ring moving at various speeds while eye movements were recorded. Smooth pursuit kinematics—eye latency, acceleration, velocity gain, position error—and the direction and amplitude of saccadic eye movements were linked to perceptual performance. Results reveal that distinct eye movement patterns—minimizing eye position error, tracking smoothly, and inhibiting reverse saccades—were related to dynamic visual acuity. The close link between eye movement quality and DVA performance has important implications for the development of perceptual training programs to improve DVA.</p></div

Perceptual acuity results.

<p>A. Proportion correct in dynamic visual acuity task as a function of gap size for n = 23 observers; n = 11 were junior players (open symbols) and n = 12 were senior players (filled symbols). Speed is denoted by color, seniority by line type; error bars are standard errors of the mean. B. Relation between static visual acuity (binocular ETDRS) and dynamic-object DVA at slow and fast speed. Lines are best fit linear regressions. C. Relation between static-object and dynamic-object DVA in degrees of visual angle.</p