Average pattern of compensation of each group to the skewed lateral shift probability distribution for A) separate bins of trials and B) averaged across the last 400 trials.

<p>In <b>A)</b>, all experimental trials are represented in the line graph (bins 1–50), where each point represents the average of 10 trials. The circles at bin 0 represent the average of trials 1–3 and are displayed to show the similarity between groups immediately after perturbation onset. In both <b>A)</b> and <b>B)</b>, the upper dashed line represents the optimal compensation (location to aim the hand) that maximizes the probability of hitting the target , based on the movement variability of the Reinforcement group. The lower dashed line represents the optimal compensation (location to aim the hand) that minimizes squared error . These findings suggest that the sensorimotor system heavily weights error feedback over reinforcement feedback when both forms of feedback are available. Error bars represent ±1 standard error of the mean. * p < 0.05.</p

<i>Experiments 1 and 2</i>: Participants held the handle of a robotic arm with their right hand.

<p>A semi-silvered mirror reflected the image (visual targets, visual feedback) from an LCD screen (not shown) onto a horizontal plane aligned with the shoulder. Participants made forward reaches from a home position, attempted to move through a visual target and stopped once their hand passed over a horizontal line that disappeared when crossed. Error (visual) feedback and reinforcement (target expands, pleasant sound and monetary reward) feedback was laterally shifted relative to true hand position. The magnitude of any particular lateral shift was drawn from a skewed probability distribution. Participants had to compensate for the lateral shifts to hit the target. Compensation represents how laterally displaced their hand was relative to the displayed target. <i>Experiment 1</i>: Laterally shifted error feedback was flashed halfway through each reach as a single dot (<i>ς</i>_0<i>mm</i>), a medium cloud of dots (<i>ς</i>_15<i>mm</i>), a large cloud of dots (<i>ς</i>_30<i>mm</i>), or withheld (<i>ς</i>_∞). The cursor and hand (not visible) paths shown above illustrate compensation that depended on the amount of visual uncertainty (<i>ς</i>_0<i>mm</i> and <i>ς</i>_15<i>mm</i> conditions shown). In the single dot (<i>ς</i>_0<i>mm</i>) condition, participants received additional feedback (error or error + reinforcement) at the target. <i>Experiment 2</i>: Participants were provided with error feedback and or reinforcement feedback only at the target.</p

Dissociating error-based and reinforcement-based loss functions during sensorimotor learning - Fig 3

<p>Average pattern of compensation (filled circles) in Experiment 1 to different magnitudes of lateral shift (<i>x</i>-axes) and visual uncertainty (separate lines) of participants receiving <b>A)</b> error feedback laterally shifted by skewed-right probability distribution, <b>B)</b> error feedback laterally shifted by a skewed-left probability distribution (note: these data are ‘flipped’ to visually align with the other groups), and <b>C)</b> both reinforcement and error feedback laterally shifted by a skewed-right probability distribution. A darker shade of blue signifies greater visual uncertainty. <b>D)</b> The best-fit power loss-function exponent (<i>α</i>^<i>opt</i>) of a Bayesian model that, in addition to characterizing how error was minimized, was also a sensitive metric to whether participants were influenced by reinforcement feedback (see <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1005623#sec009" target="_blank">Methods</a>). An exponent of 2.0 corresponds to minimizing squared error (upper dashed line), while an exponent of 1.0 corresponds to minimizing absolute error (lower dashed line). We found no significant differences in either the compensation (<i>p</i> = 0.956) or <i>α</i>^<i>opt</i> (<i>p</i> = 0.187) between groups. These findings suggest that all groups minimized approximately squared error, and that the sensorimotor system heavily weights error feedback over reinforcement feedback when both forms of feedback are available. Error bars represent ±1 standard error of the mean.</p

Group comparisons, p-values and effect sizes (), were robust to whether the last 100, 200, 300 and 400 trials were averaged together.

<p>Group comparisons, p-values and effect sizes (), were robust to whether the last 100, 200, 300 and 400 trials were averaged together.</p

Pattern of compensation of a typical participant in the: A) reinforcement group, B) reinforcement + error group, and C) error group.

<p>All experimental trials are represented in the line graph (bins 1–50), where each point represents the average of 10 trials. The circles at bin 0 represent the average of the first three trials and show each participant’s behaviour immediately after perturbation onset, which was similar across groups (see <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1005623#pcbi.1005623.g006" target="_blank">Fig 6</a>). For each group, the upper dashed line represents the optimal compensation (location to aim the hand) that maximizes the probability of hitting the target []. The lower dashed line represents the optimal compensation (location to aim the hand) that minimizes squared error []. It can be seen that the Reinforcement participant had a pattern of compensation that on average maximized target hits. Conversely, both the Error participant and the Reinforcement + Error participant learned a compensation that on average minimized approximately squared error. This behavior was consistent across participants. Error bars represent ±1 standard deviation.</p

METHODS from Reinforcement-based processes actively regulate motor exploration along redundant solution manifolds

From a baby’s babbling to a songbird practising a new tune, exploration is critical to motor learning. A hallmark of exploration is the emergence of random walk behaviour along solution manifolds, where successive motor actions are not independent but rather become serially dependent. Such exploratory random walk behaviour is ubiquitous across species neural firing, gait patterns and reaching behaviour. The past work has suggested that exploratory random walk behaviour arises from an accumulation of movement variability and a lack of error-based corrections. Here, we test a fundamentally different idea—that reinforcement-based processes regulate random walk behaviour to promote continual motor exploration to maximize success. Across three human reaching experiments, we manipulated the size of both the visually displayed target and an unseen reward zone, as well as the probability of reinforcement feedback. Our empirical and modelling results parsimoniously support the notion that exploratory random walk behaviour emerges by utilizing knowledge of movement variability to update intended reach aim towards recently reinforced motor actions. This mechanism leads to active and continuous exploration of the solution manifold, currently thought by prominent theories to arise passively. The ability to continually explore muscle, joint and task redundant solution manifolds is beneficial while acting in uncertain environments, during motor development or when recovering from a neurological disorder to discover and learn new motor actions