Disentangling the involvement of primary motor cortex in value-based reinforcement learning and value-based decision making.

When one makes the decision to act in the physical world, the neural activity in primary motor cortex (M1) encodes the competition between potential action choices. Traditional approaches have viewed this activity as reflecting the unfolding of the outcome of a decision process taking place upstream. However, a recently emerging theoretical framework posits that the motor neural structures directly contribute to the decision process. We recently tested this hypothesis (Zenon et al., 2015, Brain Stimulation) by using continuous theta burst stimulation (cTBS) to alter activity in M1 while participants performed a task that required them to select between two fingers in the right hand based on the color of a stimulus (green or red, explicit instruction). Importantly, this finger choice was biased such that, to earn more money, the subjects also had to take into account the shape of the stimulus (circle or square, undisclosed manipulation). So the motor response depended, on the one hand, on a perceptual decision process, interpreting the color of the stimulus according to instructed rules and, on the other hand, on a value-based decision process relying on reinforcement learning. Interestingly, cTBS over M1 modified the extent to which the value-based process influenced the subjects' decisions whereas it had no effect on their ability to make a choice based on perceptual evidence. Importantly, in that study, cTBS was applied at the very beginning of the experiment, before the subjects had learned the task. Hence, we cannot tell from that work whether the effect of M1 cTBS was due to an alteration of value-based reinforcement learning or of value-based decision making, which takes place once learning is complete. Here, we present a study in which we intend to use the same task but with cTBS applied at different times in order to assess the contribution of M1 to the two value-based processes (learning and decision making). More precisely, the experiment will extend over three sessions, each occurring at 24-hours interval. Each experimental session will consist of six blocks, each lasting about 4 minutes. Pilot data suggest that the value-based process begins to effectively shape the subject decisions in the middle of the second session. Given this, cTBS over M1 will be applied either at the beginning of the first session (before learning) or at the beginning of the third session (after learning). This procedure will allow us to disentangle the involvement of M1 in value-based reinforcement learning and value-based decision making

CHOReOS Middleware Specification (D3.1)

This deliverable specifies the main concepts of the CHOReOS middleware architecture. Starting from the Future Internet (FI) challenges for scalability, heterogeneity, mobility, awareness, and adaptation that have been investigated in prior work done in WP1, we introduce the aforementioned concepts to deal with the requirements derived from the FI challenges. In particular, we propose an extensible and scalable service discovery approach for the organization and discovery of services that relies on multiple service discovery protocols. Moreover, we introduce an extensible and scalable approach, based on the service bus paradigm, for service access that features the integration and adaptation of multiple interaction protocols. Furthermore, we propose solutions that enable the execution of FI service compositions that range from compositions of choreographed services, developed according to the CHOReOS development process, to massive compositions of things. Finally, we detail the Cloud & Grid middleware facilities that support the overall middleware and the choreographies that are built on it, via a unified API that provides access to multiple cloud infrastructures (e.g., Amazon EC2, HP Open Cirrus, private clouds)

Mechanisms underlying reinforcement learning of motor skills

Motor learning allows animals, including human beings, to acquire skills that are essential for efficient interactions with the environment. This ability to learn new motor skills is of great practical relevance for daily-life activities (such as when learning to drive), but also for motor rehabilitation after a lesion of the nervous system (such as a stroke). For a long time, motor learning has been mainly conceptualized as a process allowing to iteratively correct movements based on sensory information (e.g., visual, somatosensory). Importantly though, in the last years, there has been an increased appreciation that motor learning also results from other mechanisms including reinforcement learning, a process through which appropriate actions are selected through outcome-based feedback (e.g., success or failure). As such, recent evidence shows that reinforcement feedback and mo-tivation can be beneficial for motor learning both in healthy individuals and neurological populations. Despite the potential importance of these findings to improve current rehabil-itation protocols, the mechanisms underlying reinforcement-related improvements in mo-tor learning remain largely unexplored. This PhD aimed at providing deeper mechanistic understanding of reinforcement learning of motor skills through behavioral analyses, neuroimaging and non-invasive brain stimulation. In Study 1, I found that enhancing motivation (by providing monetary reward for good performance) during a motor training can lead to persistent improvements in performance that are not obtained with reinforce-ment feedback only, and are related to an increased regulation of motor variability based on previous outcomes. In Study 2, I investigated the effect of reward timing (i.e., the delay between the end of movement execution and reward receipt) on motor learning and found that delaying reward by only a few seconds could strongly influence motor learning dynamics and consolidation. Finally, in Study 3, I investigated the causal role of the stria-tum in reinforcement motor learning. Here, I show, by combining an innovative non-invasive deep brain stimulation approach called transcranial electric temporal interfer-ence stimulation and neuroimaging, that a specific mechanism relying on striatal high gamma oscillations is causally involved in reinforcement learning of motor skills. Overall, this work characterizes key mechanisms underlying the effect of reinforcement on motor learning, paving the way towards the incorporation of optimized reinforcements in motor rehabilitation protocols.(MED - Sciences médicales) -- UCL, 202

Selecting and Executing Actions for Rewards

n/

Advanced TMS approaches to probe corticospinal excitability during action preparation

The motor system displays strong changes in neural activity during action preparation. In the past decades, several techniques, including transcranial magnetic stimulation (TMS), electroencephalography (EEG) and functional magnetic resonance imaging (fMRI), have allowed us to gain insights into the functional role of such preparatory activity in humans. More recently, new TMS tools have been proposed to study the mechanistic principles underlying the changes in corticospinal excitability during action preparation. The aim of the present review is to provide a comprehensive description of these advanced methods and to discuss the new knowledge they give access to, relative to other existing approaches. We start with a brief synthesis of the work that has been achieved so far using classic TMS protocols during action preparation, such as the so-called single-pulse and paired-pulse techniques. We then highlight three new approaches that recently arose in the field of action preparation, including (1) the exploitation of TMS current direction, known as directional TMS, which enables investigating different subsets of neurons in the primary motor cortex, (2) the use of paired-pulse TMS to study the suppressive influence of the cerebellum on corticospinal excitability and (3) the development of a double-coil TMS approach, which facilitates the study of bilateral changes in corticospinal excitability. The aim of the present article is twofold: we seek to provide a comprehensive description of these advanced TMS tools and to discuss their bearings for the field of action preparation with respect to more traditional TMS approaches, as well as to neuroimaging techniques such as EEG or fMRI. Finally, we point out perspectives for fundamental and clinical research that arise from the combination of these methods, widening the horizon of possibilities for the investigation of the human motor system, both in health and disease

Beyond motor noise: considering other causes of impaired reinforcement learning in cerebellar patients.

n/

Reward timing matters in motor learning.

Reward timing, that is, the delay after which reward is delivered following an action is known to strongly influence reinforcement learning. Here, we asked if reward timing could also modulate how people learn and consolidate new motor skills. In 60 healthy participants, we found that delaying reward delivery by a few seconds influenced motor learning. Indeed, training with a short reward delay (1 s) induced continuous improvements in performance, whereas a long reward delay (6 s) led to initially high learning rates that were followed by an early plateau in the learning curve and a lower performance at the end of training. Participants who learned the skill with a long reward delay also exhibited reduced overnight memory consolidation. Overall, our data show that reward timing affects the dynamics and consolidation of motor learning, a finding that could be exploited in future rehabilitation programs

Motor training strengthens corticospinal suppression during movement preparation.

Training can improve motor skills and modify neural activity at rest and during movement execution. Learning-related modulations may also concern motor preparation but the neural correlates and the potential behavioral relevance of such adjustments remain unclear. In humans, preparatory processes have been largely investigated using transcranial magnetic stimulation (TMS) with several studies reporting decreased corticospinal excitability (CSE) relative to a baseline measure at rest; a phenomenon called preparatory suppression. Here, we investigated the effect of motor training on such preparatory suppression, in relation to resting CSE, in humans. We trained participants to initiate quick movements in an instructed-delay reaction time (RT) task and used TMS to investigate changes in CSE over the practice blocks. Training on the task speeded up RTs, with no repercussion on error rates. Training also increased resting CSE. Most interestingly, we found that CSE during action preparation did not mirror the training-related increase observed at rest. Rather, compared to the rising baseline, the degree of preparatory suppression strengthened with practice. This training-related change in preparatory suppression (but not the changes in baseline CSE) predicted RT gains: the subjects showing a greater strengthening of preparatory suppression were also those exhibiting larger gains in RTs. Finally, such relationship between RTs and preparatory suppression was also evident at the single-trial level, though only in the non-selected effector: RTs were generally faster in trials where preparatory suppression was deeper. These findings suggest training induces changes in motor preparatory processes, that are linked to an enhanced ability to initiate fast movements

Contribution of the primary motor cortex to action value encoding during motor decisions

In daily life, action selection and decision-making are constantly biased by implicit value signals. When choosing a place for dinner on a Friday night, the estimated value of each restaurant will not only depend on a deliberate inspection of the menus but also on less conscious cues such as the atmosphere in the restaurant. Despite the crucial impact of implicit value cues on daily living choices, the exact contribution of the primary motor cortex to the encoding of this source of information remains obscure. In this talk, I will present the results of a recently published study (Derosiere et al., 2017, NeuroImage) in which we show that the human primary motor cortex is involved in the encoding of implicit value information during motor decisions. A thorough description of the dynamics of this encoding during reinforcement learning and decision-making will be provided