The gradient of the reinforcement landscape influences sensorimotor learning

© 2019 Cashaback et al. Consideration of previous successes and failures is essential to mastering a motor skill. Much of what we know about how humans and animals learn from such reinforcement feedback comes from experiments that involve sampling from a small number of discrete actions. Yet, it is less understood how we learn through reinforcement feedback when sampling from a continuous set of possible actions. Navigating a continuous set of possible actions likely requires using gradient information to maximize success. Here we addressed how humans adapt the aim of their hand when experiencing reinforcement feedback that was associated with a continuous set of possible actions. Specifically, we manipulated the change in the probability of reward given a change in motor action-the reinforcement gradient-to study its influence on learning. We found that participants learned faster when exposed to a steep gradient compared to a shallow gradient. Further, when initially positioned between a steep and a shallow gradient that rose in opposite directions, participants were more likely to ascend the steep gradient. We introduce a model that captures our results and several features of motor learning. Taken together, our work suggests that the sensorimotor system relies on temporally recent and spatially local gradient information to drive learning

Both fast and slow learning processes contribute to savings following sensorimotor adaptation

© 2019 the American Physiological Society. Recent work suggests that the rate of learning in sensorimotor adaptation is likely not fixed, but rather can change based on previous experience. One example is savings, a commonly observed phenomenon whereby the relearning of a motor skill is faster than the initial learning. Sensorimotor adaptation is thought to be driven by sensory prediction errors, which are the result of a mismatch between predicted and actual sensory consequences. It has been proposed that during motor adaptation the generation of sensory prediction errors engages two processes (fast and slow) that differ in learning and retention rates. We tested the idea that a history of errors would influence both the fast and slow processes during savings. Participants were asked to perform the same force field adaptation task twice in succession. We found that adaptation to the force field a second time led to increases in estimated learning rates for both fast and slow processes. While it has been proposed that savings is explained by an increase in learning rate for the fast process, here we observed that the slow process also contributes to savings. Our work suggests that fast and slow adaptation processes are both responsive to a history of error and both contribute to savings. NEW & NOTEWORTHY We studied the underlying mechanisms of savings during motor adaptation. Using a two-state model to represent fast and slow processes that contribute to motor adaptation, we found that a history of error modulates performance in both processes. While previous research has attributed savings to only changes in the fast process, we demonstrated that an increase in both processes is needed to account for the measured behavioral data

Failure induces task-irrelevant exploration during a stencil task

During reward-based motor tasks, performance failure leads to an increase in movement variability along task-relevant dimensions. These increases in movement variability are indicative of exploratory behaviour in search of a better, more successful motor action. It is unclear whether failure also induces exploration along task-irrelevant dimensions that do not influence performance. In this study, we ask whether participants would explore the task-irrelevant dimension while they performed a stencil task. With a stylus, participants applied downward, normal force that influenced whether they received reward (task-relevant) as they simultaneously made erasing-like movement patterns along the tablet that did not influence performance (task-irrelevant). In this task, the movement pattern was analyzed as the distribution of movement directions within a movement. The results showed significant exploration of task-relevant force and task-irrelevant movement patterns. We conclude that failure can induce additional movement variability along a task-irrelevant dimension

Reinforcement-based processes actively regulate motor exploration along redundant solution manifolds

From a baby's babbling to a songbird practising a new tune, exploration is critical to motor learning. A hallmark of exploration is the emergence of random walk behaviour along solution manifolds, where successive motor actions are not independent but rather become serially dependent. Such exploratory random walk behaviour is ubiquitous across species' neural firing, gait patterns and reaching behaviour. The past work has suggested that exploratory random walk behaviour arises from an accumulation of movement variability and a lack of error-based corrections. Here, we test a fundamentally different idea-that reinforcement-based processes regulate random walk behaviour to promote continual motor exploration to maximize success. Across three human reaching experiments, we manipulated the size of both the visually displayed target and an unseen reward zone, as well as the probability of reinforcement feedback. Our empirical and modelling results parsimoniously support the notion that exploratory random walk behaviour emerges by utilizing knowledge of movement variability to update intended reach aim towards recently reinforced motor actions. This mechanism leads to active and continuous exploration of the solution manifold, currently thought by prominent theories to arise passively. The ability to continually explore muscle, joint and task redundant solution manifolds is beneficial while acting in uncertain environments, during motor development or when recovering from a neurological disorder to discover and learn new motor actions.</p