Principal components analysis of reward prediction errors in a reinforcement learning task

Models of reinforcement learning represent reward and punishment in terms of reward prediction errors (RPEs), quantitative signed terms describing the degree to which outcomes are better than expected (positive RPEs) or worse (negative RPEs). An electrophysiological component known as feedback related negativity (FRN) occurs at frontocentral sites 240-340 ms after feedback on whether a reward or punishment is obtained, and has been claimed to neurally encode an RPE. An outstanding question however, is whether the FRN is sensitive to the size of both positive RPEs and negative RPEs. Previous attempts to answer this question have examined the simple effects of RPE size for positive RPEs and negative RPEs separately. However, this methodology can be compromised by overlap from components coding for unsigned prediction error size, or "salience", which are sensitive to the absolute size of a prediction error but not its valence. In our study, positive and negative RPEs were parametrically modulated using both reward likelihood and magnitude, with principal components analysis used to separate out overlying components. This revealed a single RPE encoding component responsive to the size of positive RPEs, peaking at similar to 330ms, and occupying the delta frequency band. Other components responsive to unsigned prediction error size were shown, but no component sensitive to negative RPE size was found. (C) 2015 Elsevier Inc. All rights reserved

Feedback information and the reward positivity

The reward positivity is a component of the event-related brain potential (ERP) sensitive to neural mechanisms of reward processing. Multiple studies have demonstrated that reward positivity amplitude indices a reward prediction error signal that is fundamental to theories of reinforcement learning. However, whether this ERP component is also sensitive to richer forms of performance information important for supervised learning is less clear. To investigate this question, we recorded the electroencephalogram from participants engaged in a time estimation task in which the type of error information conveyed by feedback stimuli was systematically varied across conditions. Consistent with our predictions, we found that reward positivity amplitude decreased in relation to increasing information content of the feedback, and that reward positivity amplitude was unrelated to trial-to-trial behavioral adjustments in task performance. By contrast, a series of exploratory analyses revealed frontal-central and posterior ERP components immediately following the reward positivity that related to these processes. Taken in the context of the wider literature, these results suggest that the reward positivity is produced by a neural mechanism that motivates task performance, whereas the later ERP components apply the feedback information according to principles of supervised learning

Frontal midline theta and N200 amplitude reflect complementary information about expectancy and outcome evaluation

Feedback ERN (fERN) and frontal midline theta have both been proposed to index a dopamine-like reinforcement learning signal in anterior cingulate cortex (ACC). We investigated these proposals by comparing fERN amplitude and theta power with respect to their sensitivities to outcome valence and probability in a previously collected EEG dataset. Bayesian model comparison revealed a dissociation between the two measures, with fERN amplitude mainly sensitive to valence and theta power mainly sensitive to probability. Further, fERN amplitude was highly correlated with the portion of theta power that is consistent in phase across trials (i.e., evoked theta power). These results suggest that although both measures provide valuable information about cognitive function of frontal midline cortex, fERN amplitude is specifically sensitive to dopamine reinforcement learning signals whereas theta power reflects the ACC response to unexpected events

The feedback correct-related positivity : sensitivity of the event-related brain potential to unexpected positive feedback

The N200 and the feedback error-related negativity (fERN) are two components of the event-related brain potential (ERP) that share similar scalp distributions, time courses, morphologies, and functional dependencies, which raises the question as to whether they are actually the same phenomenon. To investigate this issue, we recorded the ERP from participants engaged in two tasks that independently elicited the N200 and fERN. Our results indicate that they are, in fact, the same ERP component and further suggest that positive feedback elicits a positive-going deflection in the time range of the fERN. Taken together, these results indicate that negative feedback elicits a common N200 and that modulation of fERN amplitude results from the superposition on correct trials of a positive-going deflection that we term the feedback correct-related positivity

Remembering Forward: Neural Correlates of Memory and Prediction in Human Motor Adaptation

We used functional MR imaging (FMRI), a robotic manipulandum and systems identification techniques to examine neural correlates of predictive compensation for spring-like loads during goal-directed wrist movements in neurologically-intact humans. Although load changed unpredictably from one trial to the next, subjects nevertheless used sensorimotor memories from recent movements to predict and compensate upcoming loads. Prediction enabled subjects to adapt performance so that the task was accomplished with minimum effort. Population analyses of functional images revealed a distributed, bilateral network of cortical and subcortical activity supporting predictive load compensation during visual target capture. Cortical regions – including prefrontal, parietal and hippocampal cortices – exhibited trial-by-trial fluctuations in BOLD signal consistent with the storage and recall of sensorimotor memories or “states” important for spatial working memory. Bilateral activations in associative regions of the striatum demonstrated temporal correlation with the magnitude of kinematic performance error (a signal that could drive reward-optimizing reinforcement learning and the prospective scaling of previously learned motor programs). BOLD signal correlations with load prediction were observed in the cerebellar cortex and red nuclei (consistent with the idea that these structures generate adaptive fusimotor signals facilitating cancelation of expected proprioceptive feedback, as required for conditional feedback adjustments to ongoing motor commands and feedback error learning). Analysis of single subject images revealed that predictive activity was at least as likely to be observed in more than one of these neural systems as in just one. We conclude therefore that motor adaptation is mediated by predictive compensations supported by multiple, distributed, cortical and subcortical structures

High temporal discounters overvalue immediate rewards rather than undervalue future rewards : an event-related brain potential study

Impulsivity is characterized in part by heightened sensitivity to immediate relative to future rewards. Although previous research has suggested that "high discounters" in intertemporal choice tasks tend to prefer immediate over future rewards because they devalue the latter, it remains possible that they instead overvalue immediate rewards. To investigate this question, we recorded the reward positivity, a component of the event-related brain potential (ERP) associated with reward processing, with participants engaged in a task in which they received both immediate and future rewards and nonrewards. The participants also completed a temporal discounting task without ERP recording. We found that immediate but not future rewards elicited the reward positivity. High discounters also produced larger reward positivities to immediate rewards than did low discounters, indicating that high discounters relatively overvalued immediate rewards. These findings suggest that high discounters may be more motivated than low discounters to work for monetary rewards, irrespective of the time of arrival of the incentives

Reward positivity elicited by predictive cues

A recent theory holds that a component of the human event-related brain potential called the reward positivity reflects a reward prediction error signal. We investigated this idea in gambling-like task in which, on each trial, a visual stimulus predicted a subsequent rewarding or nonrewarding outcome with 80% probability. Consistent with earlier results, we found that the reward positivity was larger to unexpected than to expected outcomes. In addition, we found that the predictive cues also elicited a reward positivity, as proposed by the theory. These results indicate that the reward positivity reflects the initial assessment of whether a trial will end in success or failure and the reappraisal of that information once the outcome actually occurs. NeuroReport 22:249-252 (C) 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins