Overlapping Prediction Errors in Dorsal Striatum During Instrumental Learning With Juice and Money Reward in the Human Brain

Prediction error signals have been reported in human imaging studies in target areas of dopamine neurons such as ventral and dorsal striatum during learning with many different types of reinforcers. However, a key question that has yet to be addressed is whether prediction error signals recruit distinct or overlapping regions of striatum and elsewhere during learning with different types of reward. To address this, we scanned 17 healthy subjects with functional magnetic resonance imaging while they chose actions to obtain either a pleasant juice reward (1 ml apple juice), or a monetary gain (5 cents) and applied a computational reinforcement learning model to subjects' behavioral and imaging data. Evidence for an overlapping prediction error signal during learning with juice and money rewards was found in a region of dorsal striatum (caudate nucleus), while prediction error signals in a subregion of ventral striatum were significantly stronger during learning with money but not juice reward. These results provide evidence for partially overlapping reward prediction signals for different types of appetitive reinforcers within the striatum, a finding with important implications for understanding the nature of associative encoding in the striatum as a function of reinforcer type

Neural Prediction Errors Reveal a Risk-Sensitive Reinforcement-Learning Process in the Human Brain

Humans and animals are exquisitely, though idiosyncratically, sensitive to risk or variance in the outcomes of their actions. Economic, psychological, and neural aspects of this are well studied when information about risk is provided explicitly. However, we must normally learn about outcomes from experience, through trial and error. Traditional models of such reinforcement learning focus on learning about the mean reward value of cues and ignore higher order moments such as variance. We used fMRI to test whether the neural correlates of human reinforcement learning are sensitive to experienced risk. Our analysis focused on anatomically delineated regions of a priori interest in the nucleus accumbens, where blood oxygenation level-dependent (BOLD) signals have been suggested as correlating with quantities derived from reinforcement learning. We first provide unbiased evidence that the raw BOLD signal in these regions corresponds closely to a reward prediction error. We then derive from this signal the learned values of cues that predict rewards of equal mean but different variance and show that these values are indeed modulated by experienced risk. Moreover, a close neurometric–psychometric coupling exists between the fluctuations of the experience-based evaluations of risky options that we measured neurally and the fluctuations in behavioral risk aversion. This suggests that risk sensitivity is integral to human learning, illuminating economic models of choice, neuroscientific models of affective learning, and the workings of the underlying neural mechanisms

Neural Correlates of Opponent Processes for Financial Gains and Losses

Objective: Functional imaging studies offer alternative explanations for the neural correlates of monetary gain and loss related brain activity, and their opponents, omission of gains and losses. One possible explanation based on the psychology of opponent process theory suggests that successful avoidance of an aversive outcome is itself rewarding, and hence activates brain regions involved in reward processing. In order to test this hypothesis, we compared brain activation for successful avoidance of losses and receipt of monetary gains. Additionally, the brain regions involved in processing of frustrative neutral outcomes and actual losses were compared in order to test whether these two representations are coded in common or distinct brain regions. Methods: Using a 3 Tesla functional magnetic resonance imaging machine, fifteen healthy volunteers between the ages 22 to 28 were scanned for blood oxygen level dependent signal changes while they were performing a probabilistic learning task, wherein each trial a participant chose one of the two available options in order to win or avoid losing money. Results: The results confirmed, previous findings showing that medial frontal cortex and ventral striatum show significant activation (p<0.001) not only for monetary gains but also for successful avoidance of losses. A similar activation pattern was also observed for monetary losses and avoidance of gains in the medial frontal cortex, and posterior cingulate cortex, however, there was increased activation in amygdala specific to monetary losses (p<0.001). Further, subtraction analysis showed that regardless of the type of loss (i.e., frustrative neutral outcomes) posterior insula showed increased activation. Conclusion: This study provides evidence for a significant overlap not only between gains and losses, but also between their opponents. The results suggested that the overlapping activity pattern in the medial frontal cortex could be explained by a more abstract function of medial frontal cortex, such as outcome evaluation or performance monitoring, which possibly does not differentiate between winning and losing monetary outcomes.Peer reviewedFinal Published versio

Neural Correlates of Approach and Avoidance Learning in Behavioral Inhibition

Behavioral inhibition is a temperamental trait characterized in infancy and early childhood by a tendency to withdraw from novel or familiar stimuli. Recent neuroimaging research indicates that BI individuals have atypical neural responses to information regarding reward and punishment in the striatum and amygdala--regions of the brain that receive information about salient stimuli and use it to guide motivated behavior. Activation to rewarding and punishing stimuli in these regions follows a "prediction error" pattern. My research examines whether behaviorally inhibited young adults display atypical prediction error responses, and whether these responses are specific to rewarding or aversive events. Prediction error signals are theorized to be critical for approach and avoidance learning, and a second study examined probabilistic approach and avoidance learning in the same sample, examining differences in approach and avoidance learning between behaviorally inhibited and non-inhibited individuals, and the relation between learning and neural prediction error signals to reward and punishment

Dissociating the Role of the Orbitofrontal Cortex and the Striatum in the Computation of Goal Values and Prediction Errors

To make sound economic decisions, the brain needs to compute several different value-related signals. These include goal values that measure the predicted reward that results from the outcome generated by each of the actions under consideration, decision values that measure the net value of taking the different actions, and prediction errors that measure deviations from individuals' previous reward expectations. We used functional magnetic resonance imaging and a novel decision-making paradigm to dissociate the neural basis of these three computations. Our results show that they are supported by different neural substrates: goal values are correlated with activity in the medial orbitofrontal cortex, decision values are correlated with activity in the central orbitofrontal cortex, and prediction errors are correlated with activity in the ventral striatum

Dopaminergic drug treatment remediates exaggerated cingulate prediction error responses in obsessive-compulsive disorder

Abstract: Rationale: Patients with obsessive-compulsive disorder (OCD) have been found to show exaggerated error responses and prediction error learning signals in a variety of EEG and fMRI tasks, with data converging on the anterior cingulate cortex as a key locus of dysfunction. Considerable evidence has linked prediction error processing to dopaminergic function. Objective: In this study, we investigate potential dopaminergic dysfunction during reward processing in the context of OCD. Methods: We studied OCD patients (n = 18) and controls (n = 18) whilst they learned probabilistic associations between abstract stimuli and monetary rewards in the fMRI scanner involving administration (on separate visits) of a dopamine receptor agonist, pramipexole 0.5 mg; a dopamine receptor antagonist, amisulpride 400 mg; and placebo. We fitted a Q-learning computational model to fMRI prediction error responses; group differences were examined in anterior cingulate and nucleus accumbens regions of interest. Results: There were no significant group, drug, or interaction effects in the number of correct choices; computational modeling suggested a marginally significant difference in learning rates between groups (p = 0.089, partial ƞ2 = 0.1). In the imaging results, there was a significant interaction of group by drug (p = 0.013, partial ƞ2 = 0.13). OCD patients showed abnormally strong cingulate signaling of prediction errors during omission of an expected reward, with unexpected reduction by both pramipexole and amisulpride (p = 0.014, partial ƞ2 = 0.26, 1-β error probability = 0.94). Exaggerated cingulate prediction error signaling to omitted reward in placebo was related to trait subjective difficulty in self-regulating behavior in OCD. Conclusions: Our data support cingulate dysfunction during reward processing in OCD, and bidirectional remediation by dopaminergic modulation, suggesting that exaggerated cingulate error signals in OCD may be of dopaminergic origin. The results help to illuminate the mechanisms through which dopamine receptor antagonists achieve therapeutic benefit in OCD. Further research is needed to disentangle the different functions of dopamine receptor agonists and antagonists during bidirectional modulation of cingulate activation

Increased ventral striatal volume in college-aged binge drinkers

BACKGROUND Binge drinking is a serious public health issue associated with cognitive, physiological, and anatomical differences from healthy individuals. No studies, however, have reported subcortical grey matter differences in this population. To address this, we compared the grey matter volumes of college-age binge drinkers and healthy controls, focusing on the ventral striatum, hippocampus and amygdala. METHOD T1-weighted images of 19 binge drinkers and 19 healthy volunteers were analyzed using voxel-based morphometry. Structural data were also covaried with Alcohol Use Disorders Identification Test (AUDIT) scores. Cluster-extent threshold and small volume corrections were both used to analyze imaging data. RESULTS Binge drinkers had significantly larger ventral striatal grey matter volumes compared to controls. There were no between group differences in hippocampal or amygdalar volume. Ventral striatal, amygdalar, and hippocampal volumes were also negatively related to AUDIT scores across groups. CONCLUSIONS Our findings stand in contrast to the lower ventral striatal volume previously observed in more severe forms of alcohol use disorders, suggesting that college-age binge drinkers may represent a distinct population from those groups. These findings may instead represent early sequelae, compensatory effects of repeated binge and withdrawal, or an endophenotypic risk factor