The Dopaminergic Midbrain Mediates an Effect of Average Reward on Pavlovian Vigor

Dopamine plays a key role in motivation. Phasic dopamine response reflects a reinforcement prediction error (RPE), whereas tonic dopamine activity is postulated to represent an average reward that mediates motivational vigor. However, it has been hard to find evidence concerning the neural encoding of average reward that is uncorrupted by influences of RPEs. We circumvented this difficulty in a novel visual search task where we measured participants' button pressing vigor in a context where information (underlying an RPE) about future average reward was provided well before the average reward itself. Despite no instrumental consequence, participants' pressing force increased for greater current average reward, consistent with a form of Pavlovian effect on motivational vigor. We recorded participants' brain activity during task performance with fMRI. Greater average reward was associated with enhanced activity in dopaminergic midbrain to a degree that correlated with the relationship between average reward and pressing vigor. Interestingly, an opposite pattern was observed in subgenual cingulate cortex, a region implicated in negative mood and motivational inhibition. These findings highlight a crucial role for dopaminergic midbrain in representing aspects of average reward and motivational vigor

Reductions in Mesolimbic Dopamine Signaling and Aversion: Implications for Relapse and Learned Avoidance

The ability to adjust behavior appropriately following an aversive experience is essential for survival, yet variability in this process contributes to a wide range of disorders, including drug addiction. It is clear that proper approach and avoidance is regulated, in part, by the activity of the mesolimbic dopamine system. While the importance of this system as a critical modulator of reward learning has been extensively characterized, its involvement in directing aversion-related behaviors and learning is still poorly understood. Recent studies have revealed that aversive stimuli and their predictors cause rapid reductions in nucleus accumbens (NAc) dopamine concentrations. Furthermore, a normally appetitive stimulus that is made aversive through association with cocaine also decreases dopamine, and the magnitude of the expressed aversion predicts drug-taking. However, whether the presentation of a drug cue that reduces dopamine, and evokes a negative affective state, can motivate relapse is unknown. Here we demonstrate that the presentation of an aversive drug cue both reduces dopamine and causes cocaine-seeking. This finding is provocative because drug seeking in reinstatement designs is typically associated with increased dopamine signaling. Using a combination of fast scan cyclic voltammetry (FSCV) and in vivo electrophysiology we subsequently show that the presence of an aversive drug cue abolishes the dopaminergic encoding of other drug cues and alters NAc neuronal activity patterns. Importantly, a subpopulation of neurons that subsequently encode aspects of drug-seeking behavior increase their baseline firing rates during this aversive experience. We then examine the mechanistic regulation of dopamine signaling by aversive stimuli under more natural conditions. Using FSCV and site-specific behavioral pharmacology we demonstrate that blockade of ventral tegmental area kappa opioid receptors attenuates aversion-induced reductions in dopamine, and prevents proper avoidance learning caused by punishment. By maintaining D2 receptor occupancy within the NAc during punishment, we demonstrate the requirement of aversion-induced reductions in dopamine for aversive learning. Together, these studies inform an evolving model of striatal physiology. Our findings emphasize a role for both increases and decreases in dopamine signaling that modulate behavior by promoting the stimulus-specific activity of distinct striatal output pathways. The continued interrogation of this model may offer novel targets for therapeutic development aimed at treating neurodegenerative disease and drug addiction

Dopamine Contributions to Motivational Vigor and Reinforcement Driven Learning.

Brain mechanisms for reinforcement learning and adaptive decision-making are widely accepted to critically involve the basal ganglia (BG) and the neurotransmitter dopamine (DA). DA is a key modulator of synaptic plasticity within the striatum, critically regulating neurophysiological adaptations for normal reinforcement driven learning, and maladaptive changes during disease conditions (e.g. drug addiction, Parkinson’s disease). Activity in midbrain DA cells are reported to encode errors in reward prediction, providing a learning signal to guide future behaviors. Yet, dopamine is also a key modulatory of motivation, invigorating current behavior. Prevailing theories of DA emphasize its role in either affecting current performance, or modulating reward-related learning. This thesis will present data aimed at resolving gaps in the literature for how DA makes simultaneous contributions to dissociable learning and motivational processes. Specifically, I argue that striatal DA fluctuations signal a single decision variable: a Value function (an ongoing estimate of discounted future rewards) that is used for motivational decision making ('Is It worth it?') and that abrupt deflections in this value function serve as temporal-difference reward prediction errors used for reinforcement/learning ("repeat action?”). These DA prediction errors may be causally involved in strengthening some, but not all, valuation mechanisms. Furthermore, DA activity on the midbrain-forebrain axis indicate a dissociation between DA cell bodies and their striatal terminals. I propose that this is an adaptive computational strategy, whereby DA targets tailor release to their own computational requirements, potentially converting an RPE-like spike signal into a motivational (value) message.PHDNeuroscienceUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/135768/1/hamidaa_1.pd

Structural integrity of the substantia nigra and subthalamic nucleus predicts flexibility of instrumental learning in older-age individuals

Flexible instrumental learning is required to harness the appropriate behaviors to obtain rewards and to avoid punishments. The precise contribution of dopaminergic midbrain regions (substantia nigra/ventral tegmental area [SN/VTA]) to this form of behavioral adaptation remains unclear. Normal aging is associated with a variable loss of dopamine neurons in the SN/VTA. We therefore tested the relationship between flexible instrumental learning and midbrain structural integrity. We compared task performance on a probabilistic monetary go/no-go task, involving trial and error learning of: "go to win," "no-go to win," "go to avoid losing," and "no-go to avoid losing" in 42 healthy older adults to previous behavioral data from 47 younger adults. Quantitative structural magnetization transfer images were obtained to index regional structural integrity. On average, both some younger and some older participants demonstrated a behavioral asymmetry whereby they were better at learning to act for reward ("go to win" > "no-go to win"), but better at learning not to act to avoid punishment ("no-go to avoid losing" > "go to avoid losing"). Older, but not younger, participants with greater structural integrity of the SN/VTA and the adjacent subthalamic nucleus could overcome this asymmetry. We show that interindividual variability among healthy older adults of the structural integrity within the SN/VTA and subthalamic nucleus relates to effective acquisition of competing instrumental responses

Dopamine neurons create Pavlovian conditioned stimuli with circuit-defined motivational properties.

Environmental cues, through Pavlovian learning, become conditioned stimuli that guide animals toward the acquisition of rewards (for example, food) that are necessary for survival. We tested the fundamental role of midbrain dopamine neurons in conferring predictive and motivational properties to cues, independent of external rewards. We found that brief phasic optogenetic excitation of dopamine neurons, when presented in temporal association with discrete sensory cues, was sufficient to instantiate those cues as conditioned stimuli that subsequently both evoked dopamine neuron activity on their own and elicited cue-locked conditioned behavior. Notably, we identified highly parcellated functions for dopamine neuron subpopulations projecting to different regions of striatum, revealing dissociable dopamine systems for the generation of incentive value and conditioned movement invigoration. Our results indicate that dopamine neurons orchestrate Pavlovian conditioning via functionally heterogeneous, circuit-specific motivational signals to create, gate, and shape cue-controlled behaviors

Dopamine, reward learning, and active inference

Temporal difference learning models propose phasic dopamine signaling encodes reward prediction errors that drive learning. This is supported by studies where optogenetic stimulation of dopamine neurons can stand in lieu of actual reward. Nevertheless, a large body of data also shows that dopamine is not necessary for learning, and that dopamine depletion primarily affects task performance. We offer a resolution to this paradox based on an hypothesis that dopamine encodes the precision of beliefs about alternative actions, and thus controls the outcome-sensitivity of behavior. We extend an active inference scheme for solving Markov decision processes to include learning, and show that simulated dopamine dynamics strongly resemble those actually observed during instrumental conditioning. Furthermore, simulated dopamine depletion impairs performance but spares learning, while simulated excitation of dopamine neurons drives reward learning, through aberrant inference about outcome states. Our formal approach provides a novel and parsimonious reconciliation of apparently divergent experimental findings

The Role of Ventral Tegmental Area and Nucleus Accumbens in the Kamin Blocking Effect

The overall aim of the research described in this thesis is to identify neural substrates underlying the Kamin blocking effect. This phenomenon is crucial for understanding of the neural mechanisms of associative learning. Kamin blocking refers to the finding that conditioned responding to a cue is attenuated when it is paired with a reinforcer in the presence of another cue which has previously been conditioned using that reinforcer. The blocking effect suggests that associative learning is driven by prediction errors, and is not based purely on temporal contiguity between events. In the context of appetitive classical conditioning, recent evidence suggests that the ventral tegmental area and the nucleus accumbens play a role in computing reward prediction error. The current study shows that blocking inhibition in the ventral tegmental area or inactivating the nucleus accumbens neurons during compound cue conditioning attenuates Kamin blocking. Inactivating the nucleus accumbens during single cue conditioning also attenuates Kamin blocking. Taken together, these findings suggest that inhibition in the ventral tegmental area, inhibitory output from the nucleus accumbens, and learning in the nucleus accumbens play crucial roles in the Kamin blocking effect. Previous studies show that dopamine transients track the theoretical reward prediction error during appetitive classical conditioning, and the reduction in the dopamine response evoked by the reward when it is expected has been suggested to play a role in the Kamin blocking effect. In support of this hypothesis, the current study also found that goal tracking rats, in which expected rewards have previously been shown to evoke a robust dopamine response, did not express the Kamin blocking effect. Conversely, sign trackers, in which expected rewards evoke a diminished dopamine response, expressed the blocking effect. These findings are discussed in relation to psychological theory of learning and the possible underlying neural mechanisms.Okinawa Institute of Science and Technology Graduate Universit

Dopaminergic and Non-Dopaminergic Value Systems in Conditioning and Outcome-Specific Revaluation

Animals are motivated to choose environmental options that can best satisfy current needs. To explain such choices, this paper introduces the MOTIVATOR (Matching Objects To Internal Values Triggers Option Revaluations) neural model. MOTIVATOR describes cognitiveemotional interactions between higher-order sensory cortices and an evaluative neuraxis composed of the hypothalamus, amygdala, and orbitofrontal cortex. Given a conditioned stimulus (CS), the model amygdala and lateral hypothalamus interact to calculate the expected current value of the subjective outcome that the CS predicts, constrained by the current state of deprivation or satiation. The amygdala relays the expected value information to orbitofrontal cells that receive inputs from anterior inferotemporal cells, and medial orbitofrontal cells that receive inputs from rhinal cortex. The activations of these orbitofrontal cells code the subjective values of objects. These values guide behavioral choices. The model basal ganglia detect errors in CS-specific predictions of the value and timing of rewards. Excitatory inputs from the pedunculopontine nucleus interact with timed inhibitory inputs from model striosomes in the ventral striatum to regulate dopamine burst and dip responses from cells in the substantia nigra pars compacta and ventral tegmental area. Learning in cortical and striatal regions is strongly modulated by dopamine. The model is used to address tasks that examine food-specific satiety, Pavlovian conditioning, reinforcer devaluation, and simultaneous visual discrimination. Model simulations successfully reproduce discharge dynamics of known cell types, including signals that predict saccadic reaction times and CS-dependent changes in systolic blood pressure.Defense Advanced Research Projects Agency and the Office of Naval Research (N00014-95-1-0409); National Institutes of Health (R29-DC02952, R01-DC007683); National Science Foundation (IIS-97-20333, SBE-0354378); Office of Naval Research (N00014-01-1-0624

Brief Optogenetic Inhibition of Dopamine Neurons Mimics Endogenous Negative Reward Prediction Errors

Correlative studies have strongly linked phasic changes in dopamine activity with reward prediction error signaling. But causal evidence that these brief changes in firing actually serve as error signals to drive associative learning is more tenuous. While there is direct evidence that brief increases can substitute for positive prediction errors, there is no comparable evidence that similarly brief pauses can substitute for negative prediction errors. Lacking such evidence, the effect of increases in firing could reflect novelty or salience, variables also correlated with dopamine activity. Here we provide such evidence, showing in a modified Pavlovian over-expectation task that brief pauses in the firing of dopamine neurons in rat ventral tegmental area at the time of reward are sufficient to mimic the effects of endogenous negative prediction errors. These results support the proposal that brief changes in the firing of dopamine neurons serve as full-fledged bidirectional prediction error signals

Dopamine Modulation of Choice Behavior Following Unexpected Reward Omission

Being able identify decreases in resource availability and alter motivated behavior accordingly is evolutionarily adaptive. Additionally, the neurobiological mechanisms that facilitate these basic foraging skills in animals are thought to be utilized in other forms of goal directed cognition in humans. To study how the brain mediates such behaviors, we adapt an operant behavioral task in which laboratory rats can earn food rewards from two distinct levers. We find that when one lever is extinguished, while the other lever continues to be reinforced, both male and female rats quickly identify this contingency change and develop a choice preference for the rewarded lever. Previous electrophysiology studies of putative midbrain dopamine (DA) neurons have revealed brief pauses in neuronal activity when an expected reward is omitted, which is thought to briefly decrease DA transmission in terminal regions, such as the nucleus accumbens (NAc). Additionally, decreases in DA transmission have been hypothesized to be signaled preferentially through D2 receptors. Other studies, however, have proposed that extra-cellular DA levels over longer periods of time may play a role in motivation and behavioral flexibility. To test these hypotheses, we employ one-minute sampling microdialysis and fast-scan cyclic voltammetry (FSCV) in the NAc. Microdialysis experiments reveal an increase in DA concentration, lasting multiple minutes, following the omission of an expected rewarded. These increases in DA concentration correlate to observed increases in motivational vigor and exploratory behaviors. In contrast, FSCV reveals brief decreases in DA transmission when the expected reward is omitted, consistent with previous electrophysiology studies. Furthermore, holding D2 receptor tone, through site-specific microinfusion of a D2-like agonist into the NAc, attenuates the behavioral preference for the rewarded option. Together, these experiments reveal dynamic changes in DA transmission over multiple time scales when an expected reward is omitted. Tonic increases in DA concentration may motivate the animal to employ alternate behavioral strategies, while the phasic decreases are likely involved in redirecting choice behavior away from the non-rewarded option. This series of experiments provides novel insight into the complex relationships between DA transmission and motivated behavior during negative changes in reward availability.PhDPsychologyUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/109040/1/stransky_1.pd