Striatum in stimulus-response learning via feedback and in decision making.

Cognitive deficits are recognized in Parkinson\u27s disease. Understanding cognitive functions mediated by the striatum can clarify some of these impairments and inform treatment strategies. The dorsal striatum, a region impaired in Parkinson\u27s disease, has been implicated in stimulus-response learning. However, most investigations combine acquisition of associations between stimuli, responses, or outcomes (i.e., learning) and expression of learning through response selection and decision enactment, confounding these separate processes. Using neuroimaging, we provide evidence that dorsal striatum does not mediate stimulus-response learning from feedback but rather underlies decision making once associations between stimuli and responses are learned. In the experiment, 11 males and 5 females (mean age 22) learned to associate abstract images to specific button-press responses through feedback in Session 1. In Session 2, they were asked to provide responses learned in Session 1. Feedback was omitted, precluding further feedback-based learning in this session. Using functional magnetic resonance imaging, dorsal striatum activation in healthy young participants was observed at the time of response selection and not during feedback, when greatest learning presumably occurs. Moreover, dorsal striatum activity increased across the duration of Session 1, peaking after most associations were well learned, and was significant during Session 2 where no feedback was provided, and therefore no feedback-based learning occurred. Preferential ventral striatum activity occurred during feedback and was maximal early in Session 1. Taken together, the results suggest that the ventral striatum underlies learning associations between stimuli and responses via feedback whereas the dorsal striatum mediates enacting decisions

Neural Dynamics Underlying Impaired Autonomic and Conditioned Responses Following Amygdala and Orbitofrontal Lesions

A neural model is presented that explains how outcome-specific learning modulates affect, decision-making and Pavlovian conditioned approach responses. The model addresses how brain regions responsible for affective learning and habit learning interact, and answers a central question: What are the relative contributions of the amygdala and orbitofrontal cortex to emotion and behavior? In the model, the amygdala calculates outcome value while the orbitofrontal cortex influences attention and conditioned responding by assigning value information to stimuli. Model simulations replicate autonomic, electrophysiological, and behavioral data associated with three tasks commonly used to assay these phenomena: Food consumption, Pavlovian conditioning, and visual discrimination. Interactions of the basal ganglia and amygdala with sensory and orbitofrontal cortices enable the model to replicate the complex pattern of spared and impaired behavioral and emotional capacities seen following lesions of the amygdala and orbitofrontal cortex.National Science Foundation (SBE-0354378; IIS-97-20333); Office of Naval Research (N00014-01-1-0624); Defense Advanced Research Projects Agency and the Office of Naval Research (N00014-95-1-0409); National Institutes of Health (R29-DC02952

Human Dorsal Striatal Activity during Choice Discriminates Reinforcement Learning Behavior from the Gambler’s Fallacy

Reinforcement learning theory has generated substantial interest in neurobiology, particularly because of the resemblance between phasic dopamine and reward prediction errors. Actor–critic theories have been adapted to account for the functions of the striatum, with parts of the dorsal striatum equated to the actor. Here, we specifically test whether the human dorsal striatum—as predicted by an actor–critic instantiation—is used on a trial-to-trial basis at the time of choice to choose in accordance with reinforcement learning theory, as opposed to a competing strategy: the gambler's fallacy. Using a partial-brain functional magnetic resonance imaging scanning protocol focused on the striatum and other ventral brain areas, we found that the dorsal striatum is more active when choosing consistent with reinforcement learning compared with the competing strategy. Moreover, an overlapping area of dorsal striatum along with the ventral striatum was found to be correlated with reward prediction errors at the time of outcome, as predicted by the actor–critic framework. These findings suggest that the same region of dorsal striatum involved in learning stimulus–response associations may contribute to the control of behavior during choice, thereby using those learned associations. Intriguingly, neither reinforcement learning nor the gambler's fallacy conformed to the optimal choice strategy on the specific decision-making task we used. Thus, the dorsal striatum may contribute to the control of behavior according to reinforcement learning even when the prescriptions of such an algorithm are suboptimal in terms of maximizing future rewards

Dysfunctional Social Reinforcement Processing in Disruptive Behavior Disorders: An Functional Magnetic Resonance Imaging Study.

ObjectivePrior functional magnetic resonance imaging (fMRI) work has revealed that children/adolescents with disruptive behavior disorders (DBDs) show dysfunctional reward/non-reward processing of non-social reinforcements in the context of instrumental learning tasks. Neural responsiveness to social reinforcements during instrumental learning, despite the importance of this for socialization, has not yet been previously investigated.MethodsTwenty-nine healthy children/adolescents and 19 children/adolescents with DBDs performed the fMRI social/non-social reinforcement learning task. Participants responded to random fractal image stimuli and received social and non-social rewards/non-rewards according to their accuracy.ResultsChildren/adolescents with DBDs showed significantly reduced responses within the caudate and posterior cingulate cortex (PCC) to non-social (financial) rewards and social non-rewards (the distress of others). Connectivity analyses revealed that children/adolescents with DBDs have decreased positive functional connectivity between the ventral striatum (VST) and the ventromedial prefrontal cortex (vmPFC) seeds and the lateral frontal cortex in response to reward relative to non-reward, irrespective of its sociality. In addition, they showed decreased positive connectivity between the vmPFC seed and the amygdala in response to non-reward relative to reward.ConclusionThese data indicate compromised reinforcement processing of both non-social rewards and social non-rewards in children/adolescents with DBDs within core regions for instrumental learning and reinforcement-based decision- making (caudate and PCC). In addition, children/adolescents with DBDs show dysfunctional interactions between the VST, vmPFC, and lateral frontal cortex in response to rewarded instrumental actions potentially reflecting disruptions in attention to rewarded stimuli

How Laminar Frontal Cortex and Basal Ganglia Circuits Interact to Control Planned and Reactive Saccades

The basal ganglia and frontal cortex together allow animals to learn adaptive responses that acquire rewards when prepotent reflexive responses are insufficient. Anatomical studies show a rich pattern of interactions between the basal ganglia and distinct frontal cortical layers. Analysis of the laminar circuitry of the frontal cortex, together with its interactions with the basal ganglia, motor thalamus, superior colliculus, and inferotemporal and parietal cortices, provides new insight into how these brain regions interact to learn and perform complexly conditioned behaviors. A neural model whose cortical component represents the frontal eye fields captures these interacting circuits. Simulations of the neural model illustrate how it provides a functional explanation of the dynamics of 17 physiologically identified cell types found in these areas. The model predicts how action planning or priming (in cortical layers III and VI) is dissociated from execution (in layer V), how a cue may serve either as a movement target or as a discriminative cue to move elsewhere, and how the basal ganglia help choose among competing actions. The model simulates neurophysiological, anatomical, and behavioral data about how monkeys perform saccadic eye movement tasks, including fixation; single saccade, overlap, gap, and memory-guided saccades; anti-saccades; and parallel search among distractors.Defense Advanced Research Projects Agency and the Office of Naval Research (N00014-95-l-0409, N00014-92-J-1309, N00014-95-1-0657); National Science Foundation (IRI-97-20333)

Stress and Decision Making: Effects on Valuation, Learning, and Risk-taking

A wide range of stressful experiences can influence human decision making in complex ways beyond the simple predictions of a fight-or-flight model. Recent advances may provide insight into this complicated interaction, potentially in directions that could result in translational applications. Early research suggests that stress exposure influences basic neural circuits involved in reward processing and learning, while also biasing decisions toward habit and modulating our propensity to engage in risk-taking. That said, a substantial array of theoretical and methodological considerations in research on the topic challenge strong cross study comparisons necessary for the field to move forward. In this review we examine the multifaceted stress construct in the context of human decision making, emphasizing stress’ effect on valuation, learning, and risk-taking

Dopaminergic and Non-Dopaminergic Value Systems in Conditioning and Outcome-Specific Revaluation

Animals are motivated to choose environmental options that can best satisfy current needs. To explain such choices, this paper introduces the MOTIVATOR (Matching Objects To Internal Values Triggers Option Revaluations) neural model. MOTIVATOR describes cognitiveemotional interactions between higher-order sensory cortices and an evaluative neuraxis composed of the hypothalamus, amygdala, and orbitofrontal cortex. Given a conditioned stimulus (CS), the model amygdala and lateral hypothalamus interact to calculate the expected current value of the subjective outcome that the CS predicts, constrained by the current state of deprivation or satiation. The amygdala relays the expected value information to orbitofrontal cells that receive inputs from anterior inferotemporal cells, and medial orbitofrontal cells that receive inputs from rhinal cortex. The activations of these orbitofrontal cells code the subjective values of objects. These values guide behavioral choices. The model basal ganglia detect errors in CS-specific predictions of the value and timing of rewards. Excitatory inputs from the pedunculopontine nucleus interact with timed inhibitory inputs from model striosomes in the ventral striatum to regulate dopamine burst and dip responses from cells in the substantia nigra pars compacta and ventral tegmental area. Learning in cortical and striatal regions is strongly modulated by dopamine. The model is used to address tasks that examine food-specific satiety, Pavlovian conditioning, reinforcer devaluation, and simultaneous visual discrimination. Model simulations successfully reproduce discharge dynamics of known cell types, including signals that predict saccadic reaction times and CS-dependent changes in systolic blood pressure.Defense Advanced Research Projects Agency and the Office of Naval Research (N00014-95-1-0409); National Institutes of Health (R29-DC02952, R01-DC007683); National Science Foundation (IIS-97-20333, SBE-0354378); Office of Naval Research (N00014-01-1-0624

The Neural Mechanisms Underlying the Influence of Pavlovian Cues on Human Decision Making

In outcome-specific transfer, pavlovian cues that are predictive of specific outcomes bias action choice toward actions associated with those outcomes. This transfer occurs despite no explicit training of the instrumental actions in the presence of pavlovian cues. The neural substrates of this effect in humans are unknown. To address this, we scanned 23 human subjects with functional magnetic resonance imaging while they made choices between different liquid food rewards in the presence of pavlovian cues previously associated with one of these outcomes. We found behavioral evidence of outcome-specific transfer effects in our subjects, as well as differential blood oxygenation level-dependent activity in a region of ventrolateral putamen when subjects chose, respectively, actions consistent and inconsistent with the pavlovian-predicted outcome. Our results suggest that choosing an action incompatible with a pavlovian-predicted outcome might require the inhibition of feasible but nonselected action– outcome associations. The results of this study are relevant for understanding how marketing actions can affect consumer choice behavior as well as for how environmental cues can influence drug-seeking behavior in addiction

Event-related functional magnetic resonance imaging of reward-related brain circuitry in children and adolescents

BACKGROUND: Functional disturbances in reward-related brain systems are thought to play a role in the development of mood, impulse, and substance abuse disorders. Studies in non-human primates have identified brain regions, including the dorsal / ventral striatum and orbital-frontal cortex (OFC), in which neural activity is modulated by reward. Recent studies in adults have concurred with these findings by observing reward-contingent blood oxygen level dependant (BOLD) responses in these regions during functional magnetic resonance imaging (FMRI) paradigms. However no previous studies indicate whether comparable modulations of neural activity exist in the brain reward systems of children and adolescents. METHODS: We used event-related FMRI and a behavioral paradigm modeled on previous work in adults to study brain responses to monetary gains and losses in non-psychiatric children and adolescents as part of a program examining the neural substrates of anxiety and depression in youth. RESULTS: Regions and time-courses of reward-related activity were similar to those observed in adults with condition-dependent BOLD changes in the ventral striatum, lateral and medial OFC; specifically, these regions showed larger responses to positive than to negative feedback. CONCLUSIONS: These results provide further evidence for the value of event-related FMRI in examining reward systems of the brain, demonstrate the feasibility of this approach in children and adolescents, and establish a baseline from which to understand the pathophysiology of reward-related psychiatric disorders in youth