Expected reward modulates encoding-related theta activity before an event

Oscillatory brain activity in the theta frequency range (4–8 Hz) before the onset of an event has been shown to affect the likelihood of successfully encoding the event into memory. Recent work has also indicated that frontal theta activity might be modulated by reward, but it is not clear how reward expectancy, anticipatory theta activity, and memory formation might be related. Here, we used scalp electroencephalography (EEG) to assess the relationship between these factors. EEG was recorded from healthy adults while they memorized a series of words. Each word was preceded by a cue that indicated whether a high or low monetary reward would be earned if the word was successfully remembered in a later recognition test. Frontal theta power between the presentation of the reward cue and the onset of a word was predictive of later memory for the word, but only in the high reward condition. No theta differences were observed before word onset following low reward cues. The magnitude of prestimulus encoding-related theta activity in the high reward condition was correlated with the number of high reward words that were later confidently recognized. These findings provide strong evidence for a link between reward expectancy, theta activity, and memory encoding. Theta activity before event onset seems to be especially important for the encoding of motivationally significant stimuli. One possibility is that dopaminergic activity during reward anticipation mediates frontal theta activity related to memory

State anxiety alters the neural oscillatory correlates of predictions and prediction errors during reward-based learning

Anxiety influences how the brain estimates and responds to uncertainty. The consequences of these processes on behaviour have been described in theoretical and empirical studies, yet the associated neural correlates remain unclear. Rhythm-based accounts of Bayesian predictive coding propose that predictions in generative models of perception are represented in alpha (8–12 Hz) and beta oscillations (13–30 Hz). Updates to predictions are driven by prediction errors weighted by precision (inverse variance), and are encoded in gamma oscillations (>30 Hz) and associated with suppression of beta activity. We tested whether state anxiety alters the neural oscillatory activity associated with predictions and precision-weighted prediction errors (pwPE) during learning. Healthy human participants performed a probabilistic reward-based learning task in a volatile environment. In our previous work, we described learning behaviour in this task using a hierarchical Bayesian model, revealing more precise (biased) beliefs about the tendency of the reward contingency in state anxiety, consistent with reduced learning in this group. The model provided trajectories of predictions and pwPEs for the current study, allowing us to assess their parametric effects on the time-frequency representations of EEG data. Using convolution modelling for oscillatory responses, we found that, relative to a control group, state anxiety increased beta activity in frontal and sensorimotor regions during processing of pwPE, and in fronto-parietal regions during encoding of predictions. No effects of state anxiety on gamma modulation were found. Our findings expand prior evidence on the oscillatory representations of predictions and pwPEs into the reward-based learning domain. The results suggest that state anxiety modulates beta-band oscillatory correlates of pwPE and predictions in generative models, providing insights into the neural processes associated with biased belief updating and poorer learning

Feedback information and the reward positivity

The reward positivity is a component of the event-related brain potential (ERP) sensitive to neural mechanisms of reward processing. Multiple studies have demonstrated that reward positivity amplitude indices a reward prediction error signal that is fundamental to theories of reinforcement learning. However, whether this ERP component is also sensitive to richer forms of performance information important for supervised learning is less clear. To investigate this question, we recorded the electroencephalogram from participants engaged in a time estimation task in which the type of error information conveyed by feedback stimuli was systematically varied across conditions. Consistent with our predictions, we found that reward positivity amplitude decreased in relation to increasing information content of the feedback, and that reward positivity amplitude was unrelated to trial-to-trial behavioral adjustments in task performance. By contrast, a series of exploratory analyses revealed frontal-central and posterior ERP components immediately following the reward positivity that related to these processes. Taken in the context of the wider literature, these results suggest that the reward positivity is produced by a neural mechanism that motivates task performance, whereas the later ERP components apply the feedback information according to principles of supervised learning

Rhythmic Action Synchronizes Memory Replay During Reinforcement Learning

Our cognitive abilities - learning from the past, sensing the current environment, planning into the future, executing an action, and infusing value into an experience - all rely on precisely timed and widespread electrical communications across neural networks. The brain’s hippocampal formation receives multimodal input, forges episodic associations, and predicts future state. Oscillating electrical bursts originating from the hippocampus, termed ‘sharp-wave ripples’ (SWR), often contain patterns of previously expressed neural spike sequences, and are necessary for certain forms of learning and memory. The discharge of SWR-replay resonates in remote parts of the brain and displays specific characteristics depending on a subject’s state of awareness and sensory context. In the sleep state, when motoric repertoire is limited, waves of breathing synchronize neural activity in several regions of the brain, including SWRs of the hippocampus. During active sensation of the awake state, cyclic licking dynamically entrains taste-reward networks in subcortical and cortical areas throughout learning. However, the neural correlates linking oromotor movements in the active learning state to the memory system of the hippocampal formation have not yet been established. Given the recurrence of SWR-replay during rhythmic ingestion of reinforcement learning and the hierarchical coupling of orofacial behaviors, we hypothesized that repeated licking could provide the oscillatory framework to synchronize memory reactivation during active learning. We approach this question with new technology development to track licking events at a reward port (P-event) during behavior on a spatial alternation task. Additionally, we developed a modular brain implant to simultaneously record from hippocampal area CA1 and medial entorhinal cortex (MEC) - interconnected brain regions that are crucial to episodic memory processing. Along with the co-modulation of individual neurons by licking and SWRs, we provide the first evidence that SWRs detected in dorsal CA1 synchronize with the phase of P-event cycle during learning. Furthermore, we confirmed that SWRs occurring during licking bouts contain neural reactivation of active navigation and trigger enhanced ripple-frequency power in downstream MEC. These results connect movement with memory and may assist in addressing abnormal ingestion behaviors that negatively affect mental or physical healt

Transcranial Direct Corrent stimulation (tDCS) of the anterior prefrontal cortex (aPFC) modulates reinforcement learning and decision-making under uncertainty: A doubleblind crossover study

Reinforcement learning refers to the ability to acquire information from the outcomes of prior choices (i.e. positive and negative) in order to make predictions on the effect of future decision and adapt the behaviour basing on past experiences. The anterior prefrontal cortex (aPFC) is considered to play a key role in the representation of event value, reinforcement learning and decision-making. However, a causal evidence of the involvement of this area in these processes has not been provided yet. The aim of the study was to test the role of the orbitofrontal cortex in feedback processing, reinforcement learning and decision-making under uncertainly. Eighteen healthy individuals underwent three sessions of tDCS over the prefrontal pole (anodal, cathodal, sham) during a probabilistic learning (PL) task. In the PL task, participants were invited to learn the covert probabilistic stimulusoutcome association from positive and negative feedbacks in order to choose the best option. Afterwards, a probabilistic selection (PS) task was delivered to assess decisions based on the stimulus-reward associations acquired in the PL task. During cathodal tDCS, accuracy in the PL task was reduced and participants were less prone to maintain their choice after positive feedback or to change it after a negative one (i.e., winstay and lose-shift behavior). In addition, anodal tDCS affected the subsequent PS task by reducing the ability to choose the best alternative during hard probabilistic decisions. In conclusion, the present study suggests a causal role of aPFC in feedback trial-by-trial behavioral adaptation and decision-making under uncertainty

Theta phase synchronization between the human hippocampus and prefrontal cortex increases during encoding of unexpected information: A case study

Events that violate predictions are thought to not only modulate activity within the hippocampus and PFC but also enhance communication between the two regions. Scalp and intracranial EEG studies have shown that oscillations in the theta frequency band are enhanced during processing of contextually unexpected information. Some theories suggest that the hippocampus and PFC interact during processing of unexpected events, and it is possible that theta oscillations may mediate these interactions. Here, we had the rare opportunity to conduct simultaneous electrophysiological recordings from the human hippocampus and PFC from two patients undergoing presurgical evaluation for pharmacoresistant epilepsy. Recordings were conducted during a task that involved encoding of contextually expected and unexpected visual stimuli. Across both patients, hippocampal–prefrontal theta phase synchronization was significantly higher during encoding of contextually unexpected study items, relative to contextually expected study items. Furthermore, the hippocampal–prefrontal theta phase synchronization was larger for contextually unexpected items that were later remembered compared with later forgotten items. Moreover, we did not find increased theta synchronization between the PFC and rhinal cortex, suggesting that the observed effects were specific to prefrontal–hippocampal interactions. Our findings are consistent with the idea that theta oscillations orchestrate communication between the hippocampus and PFC in support of enhanced encoding of contextually deviant information

Principal components analysis of reward prediction errors in a reinforcement learning task

Models of reinforcement learning represent reward and punishment in terms of reward prediction errors (RPEs), quantitative signed terms describing the degree to which outcomes are better than expected (positive RPEs) or worse (negative RPEs). An electrophysiological component known as feedback related negativity (FRN) occurs at frontocentral sites 240-340 ms after feedback on whether a reward or punishment is obtained, and has been claimed to neurally encode an RPE. An outstanding question however, is whether the FRN is sensitive to the size of both positive RPEs and negative RPEs. Previous attempts to answer this question have examined the simple effects of RPE size for positive RPEs and negative RPEs separately. However, this methodology can be compromised by overlap from components coding for unsigned prediction error size, or "salience", which are sensitive to the absolute size of a prediction error but not its valence. In our study, positive and negative RPEs were parametrically modulated using both reward likelihood and magnitude, with principal components analysis used to separate out overlying components. This revealed a single RPE encoding component responsive to the size of positive RPEs, peaking at similar to 330ms, and occupying the delta frequency band. Other components responsive to unsigned prediction error size were shown, but no component sensitive to negative RPE size was found. (C) 2015 Elsevier Inc. All rights reserved

Unraveling the influence of trial-based motivational changes on performance monitoring stages in a flanker task

Performance monitoring (PM) is a vital component of adaptive behavior and known to be influenced by motivation. We examined effects of potential gain (PG) and loss avoidance (LA) on neural correlates of PM at different processing stages, using a task with trial-based changes in these motivational contexts. Findings suggest more attention is allocated to the PG context, with higher amplitudes for respective correlates of stimulus and feedback processing. The PG context favored rapid responses, while the LA context emphasized accurate responses. Lower response thresholds in the PG context after correct responses derived from a drift–diffusion model also indicate a more approach-oriented response style in the PG context. This cognitive shift is mirrored in neural correlates: negative feedback in the PG context elicited a higher feedback-related negativity (FRN) and higher theta power, whereas positive feedback in the LA context elicited higher P3a and P3b amplitudes, as well as higher theta power. There was no effect of motivational context on response-locked brain activity. Given the similar frequency of negative feedback in both contexts, the elevated FRN and theta power in PG trials cannot be attributed to variations in reward prediction error. The observed variations in the FRN indicate that the effect of outcome valence is modulated by motivational salience