Human Dorsal Striatal Activity during Choice Discriminates Reinforcement Learning Behavior from the Gambler’s Fallacy

Reinforcement learning theory has generated substantial interest in neurobiology, particularly because of the resemblance between phasic dopamine and reward prediction errors. Actor–critic theories have been adapted to account for the functions of the striatum, with parts of the dorsal striatum equated to the actor. Here, we specifically test whether the human dorsal striatum—as predicted by an actor–critic instantiation—is used on a trial-to-trial basis at the time of choice to choose in accordance with reinforcement learning theory, as opposed to a competing strategy: the gambler's fallacy. Using a partial-brain functional magnetic resonance imaging scanning protocol focused on the striatum and other ventral brain areas, we found that the dorsal striatum is more active when choosing consistent with reinforcement learning compared with the competing strategy. Moreover, an overlapping area of dorsal striatum along with the ventral striatum was found to be correlated with reward prediction errors at the time of outcome, as predicted by the actor–critic framework. These findings suggest that the same region of dorsal striatum involved in learning stimulus–response associations may contribute to the control of behavior during choice, thereby using those learned associations. Intriguingly, neither reinforcement learning nor the gambler's fallacy conformed to the optimal choice strategy on the specific decision-making task we used. Thus, the dorsal striatum may contribute to the control of behavior according to reinforcement learning even when the prescriptions of such an algorithm are suboptimal in terms of maximizing future rewards

Overlapping Prediction Errors in Dorsal Striatum During Instrumental Learning With Juice and Money Reward in the Human Brain

Prediction error signals have been reported in human imaging studies in target areas of dopamine neurons such as ventral and dorsal striatum during learning with many different types of reinforcers. However, a key question that has yet to be addressed is whether prediction error signals recruit distinct or overlapping regions of striatum and elsewhere during learning with different types of reward. To address this, we scanned 17 healthy subjects with functional magnetic resonance imaging while they chose actions to obtain either a pleasant juice reward (1 ml apple juice), or a monetary gain (5 cents) and applied a computational reinforcement learning model to subjects' behavioral and imaging data. Evidence for an overlapping prediction error signal during learning with juice and money rewards was found in a region of dorsal striatum (caudate nucleus), while prediction error signals in a subregion of ventral striatum were significantly stronger during learning with money but not juice reward. These results provide evidence for partially overlapping reward prediction signals for different types of appetitive reinforcers within the striatum, a finding with important implications for understanding the nature of associative encoding in the striatum as a function of reinforcer type

Contextual novelty changes reward representations in the striatum

Reward representation in ventral striatum is boosted by perceptual novelty, although the mechanism of this effect remains elusive. Animal studies indicate a functional loop (Lisman and Grace, 2005) that includes hippocampus, ventral striatum, and midbrain as being important in regulating salience attribution within the context of novel stimuli. According to this model, reward responses in ventral striatum or midbrain should be enhanced in the context of novelty even if reward and novelty constitute unrelated, independent events. Using fMRI, we show that trials with reward-predictive cues and subsequent outcomes elicit higher responses in the striatum if preceded by an unrelated novel picture, indicating that reward representation is enhanced in the context of novelty. Notably, this effect was observed solely when reward occurrence, and hence reward-related salience, was low. These findings support a view that contextual novelty enhances neural responses underlying reward representation in the striatum and concur with the effects of novelty processing as predicted by the model of Lisman and Grace (2005)

The neural basis of unwanted thoughts during resting state.

Human beings are constantly engaged in thought. Sometimes thoughts occur repetitively and can become distressing. Up to now the neural bases of these intrusive or unwanted thoughts is largely unexplored. To study the neural correlates of unwanted thoughts, we acquired resting-state fMRI data of 41 female healthy subjects and assessed the self-reported amount of unwanted thoughts during measurement. We analyzed local connectivity by means of regional homogeneity (ReHo) and functional connectivity of a seed region. More unwanted thoughts (state) were associated with lower ReHo in right dorsolateral prefrontal cortex (DLPFC) and higher ReHo in left striatum (putamen). Additional seed-based analysis revealed higher functional connectivity of the left striatum with left inferior frontal gyrus (IFG) in participants reporting more unwanted thoughts. The state-dependent higher connectivty in left striatum was positively correlated with rumination assessed with a dedicated questionnaire focussing on trait aspects. Unwanted thoughts are associated with activity in the fronto-striatal brain circuitry. The reduction of local connectivity in DLPFC could reflect deficiencies in thought suppression processes, whereas the hightened activity in left striatum could imply an imbalance of gating mechanisms housed in basal ganglia. Its functional connectivity to left IFG is discussed as the result of thought-related speech processes

Dystonia: sparse synapses for D2 receptors in striatum of a DYT1 knock-out mouse model

Dystonia pathophysiology has been partly linked to downregulation and dysfunction of dopamine D2 receptors in striatum. We aimed to investigate the possible morpho-structural correlates of D2 receptor downregulation in the striatum of a DYT1 Tor1a mouse model. Adult control Tor1a+/+ and mutant Tor1a+/− mice were used. The brains were perfused and free-floating sections of basal ganglia were incubated with polyclonal anti-D2 antibody, followed by secondary immune-fluorescent antibody. Confocal microscopy was used to detect immune-fluorescent signals. The same primary antibody was used to evaluate D2 receptor expression by western blot. The D2 receptor immune-fluorescence appeared circumscribed in small disks (~0.3–0.5 μm diameter), likely representing D2 synapse aggregates, densely distributed in the striatum of Tor1a+/+ mice. In the Tor1a+/− mice the D2 aggregates were significantly smaller (μm2 2.4 ± SE 0.16, compared to μm2 6.73 ± SE 3.41 in Tor1a+/+) and sparse, with ~30% less number per microscopic field, value correspondent to the amount of reduced D2 expression in western blotting analysis. In DYT1 mutant mice the sparse and small D2 synapses in the striatum may be insufficient to “gate” the amount of presynaptic dopamine release diffusing in peri-synaptic space, and this consequently may result in a timing and spatially larger nonselective sphere of influence of dopamine action

Levothyroxine effects on depressive symptoms and limbic glucose metabolism in bipolar disorder: a randomized, placebo-controlled positron emission tomography study.

Adding supraphysiologic doses of levothyroxine (L-T4) to standard treatment for bipolar depression shows promise, but the mechanisms underlying clinical improvement are unknown. In a previous pilot study, L-T4 treatment reduced depression scores and activity within the anterior limbic network. Here we extended this work in a randomized, double-blind, placebo-controlled study of patients with bipolar depression. Cerebral glucose metabolism was assessed with positron emission tomography and [F-18]fluorodeoxyglucose before and after 6 weeks of treatment with L-T4 (n=15) or placebo (n=10) in 12 volumes of interest (VOIs): the bilateral thalamus, amygdala, hippocampus, dorsal striatum and ventral striatum, and midline cerebellar vermis and subgenual cingulate cortex. Radioactivity in the VOIs, normalized to whole-brain radioactivity was taken as a surrogate index of glucose metabolism, and markers of thyroid function were assayed. Changes in brain activity and their association with clinical response were assessed using statistical parametric mapping. Adjunctive L-T4 treatment produced a significant decline in depression scores during the 6-week treatment. In patients treated with L-T4, we found a significant decrease in regional activity at P<0.05 after Bonferroni correction in the left thalamus, right amygdala, right hippocampus, left ventral striatum and the right dorsal striatum. Decreases in the left thalamus, left dorsal striatum and the subgenual cingulate were correlated with a reduction in depression scores (P<0.05 after Bonferroni correction). Placebo treatment was associated with a significant decrease in activity only in the right amygdala, and no region had a change in activity that was correlated with change in depression scores. The groups differed significantly in the relationship between the changes in depression scores and in activity in the thalamus bilaterally and the left ventral striatum. The findings provide evidence that administration of supraphysiologic thyroid hormone improves depressive symptoms in patients with bipolar disorder by modulating function in components of the anterior limbic network

Cigarette Use and Striatal Dopamine D2/3 Receptors: Possible Role in the Link between Smoking and Nicotine Dependence.

BackgroundCigarette smoking induces dopamine release in the striatum, and smoking- or nicotine-induced ventral striatal dopamine release is correlated with nicotine dependence. Smokers also exhibit lower dopamine D2/3 receptor availability in the dorsal striatum than nonsmokers. Negative correlations of striatal dopamine D2/3 receptor availability with smoking exposure and nicotine dependence, therefore, might be expected but have not been tested.MethodsTwenty smokers had positron emission tomography scans with [18F]fallypride to measure dopamine D2/3 receptor availability in ventral and dorsal regions of the striatum and provided self-report measures of recent and lifetime smoking and of nicotine dependence.ResultsAs reported before, lifetime smoking was correlated with nicotine dependence. New findings were that ventral striatal dopamine D2/3 receptor availability was negatively correlated with recent and lifetime smoking and also with nicotine dependence.ConclusionThe results suggest an effect of smoking on ventral striatal D2/3 dopamine receptors that may contribute to nicotine dependence

The Effects of Acute Stress Exposure on Neural Correlates of Pavlovian Conditioning with Monetary Gains and Losses

Pavlovian conditioning involves the association of an inherently neutral stimulus with an appetitive or aversive outcome, such that the neutral stimulus itself acquires reinforcing properties. Across species, this type of learning has been shown to involve subcortical brain regions such as the striatum and the amygdala. It is less clear, however, how the neural circuitry involved in the acquisition of Pavlovian contingencies in humans, particularly in the striatum, is affected by acute stress. In the current study, we investigate the effect of acute stress exposure on Pavlovian conditioning using monetary reinforcers. Participants underwent a partial reinforcement conditioning procedure in which neutral stimuli were paired with high and low magnitude monetary gains and losses. A between-subjects design was used, such that half of the participants were exposed to cold stress while the remaining participants were exposed to a no stress control procedure. Cortisol measurements and subjective ratings were used as measures of stress. We observed an interaction between stress, valence, and magnitude in the ventral striatum, with the peak in the putamen. More specifically, the stress group exhibited an increased sensitivity to magnitude in the gain domain. This effect was driven by those participants who experienced a larger increase in circulating cortisol levels in response to the stress manipulation. Taken together, these results suggest that acute stress can lead to individual differences in circulating cortisol levels which influence the striatum during Pavlovian conditioning with monetary reinforcers