Dissociating the Role of the Orbitofrontal Cortex and the Striatum in the Computation of Goal Values and Prediction Errors

To make sound economic decisions, the brain needs to compute several different value-related signals. These include goal values that measure the predicted reward that results from the outcome generated by each of the actions under consideration, decision values that measure the net value of taking the different actions, and prediction errors that measure deviations from individuals' previous reward expectations. We used functional magnetic resonance imaging and a novel decision-making paradigm to dissociate the neural basis of these three computations. Our results show that they are supported by different neural substrates: goal values are correlated with activity in the medial orbitofrontal cortex, decision values are correlated with activity in the central orbitofrontal cortex, and prediction errors are correlated with activity in the ventral striatum

Choosing the lesser of two evils, the better of two goods: Specifying the roles of ventromedial prefrontal cortex and dorsal anterior cingulate in object choice

The ventromedial prefrontal cortex (vmPFC) and dorsal anterior cingulate cortices (ACd) are considered important for reward-based decision making. However, work distinguishing their individual functional contributions has only begun. One aspect of decision making that has received little attention is that making the right choice often translates to making the better choice. Thus, response choice often occurs in situations where both options are desirable (e.g., choosing between mousse au chocolat or crème caramel cheesecake from a menu) or, alternatively, in situations where both options are undesirable. Moreover, response choice is easier when the reinforcements associated with the objects are far apart, rather than close together, in value. We used functional magnetic resonance imaging to delineate the functional roles of the vmPFC and ACd by investigating these two aspects of decision making: (1) decision form (i.e., choosing between two objects to gain the greater reward or the lesser punishment), and (2) between-object reinforcement distance (i.e., the difference in reinforcements associated with the two objects). Blood oxygen level-dependent (BOLD) responses within the ACd and vmPFC were both related to decision form but differentially. Whereas ACd showed greater responses when deciding between objects to gain the lesser punishment, vmPFC showed greater responses when deciding between objects to gain the greater reward. Moreover, vmPFC was sensitive to reinforcement expectations associated with both the chosen and the forgone choice. In contrast, BOLD responses within ACd, but not vmPFC, related to between-object reinforcement distance, increasing as the distance between the reinforcements of the two objects decreased. These data are interpreted with reference to models of ACd and vmPFC functioning

Do Political and Economic Choices Rely on Common Neural Substrates? A Systematic Review of the Emerging Neuropolitics Literature

The methods of cognitive neuroscience are beginning to be applied to the study of political behavior. The neural substrates of value-based decision-making have been extensively examined in economic contexts; this might provide a powerful starting point for understanding political decision-making. Here, we asked to what extent the neuropolitics literature to date has used conceptual frameworks and experimental designs that make contact with the reward-related approaches that have dominated decision neuroscience. We then asked whether the studies of political behavior that can be considered in this light implicate the brain regions that have been associated with subjective value related to “economic” reward. We performed a systematic literature review to identify papers addressing the neural substrates of political behavior and extracted the fMRI studies reporting behavioral measures of subjective value as defined in decision neuroscience studies of reward. A minority of neuropolitics studies met these criteria and relatively few brain activation foci from these studies overlapped with regions where activity has been related to subjective value. These findings show modest influence of reward-focused decision neuroscience on neuropolitics research to date. Whether the neural substrates of subjective value identified in economic choice paradigms generalize to political choice thus remains an open question. We argue that systematically addressing the commonalities and differences in these two classes of value-based choice will be important in developing a more comprehensive model of the brain basis of human decision-making

The Neural Computations In The Caudate Nucleus For Reward-Biased Perceptual Decision-Making

Decision-making is a complex process in which our brain has to combine different sources of information, such as noisy sensory evidence and expected reward, in appropriate ways to obtain the outcome that satisfies the decision-maker. Despite various studies on perceptual decision-making and value-based decision making, it is still unclear how the brain combines sensory and reward information to make a complex decision. A prime candidate for mediating this process is the basal ganglia pathway. This pathway is known to make separate contributions to perceptual decisions based on the interpretation of uncertain sensory evidence and value-based decisions that select among outcome options. To begin to investigate what computations are performed by the brain, particularly in the basal ganglia, we trained monkeys to perform a reward-biased visual motion direction discrimination task and performed single-unit extracellular recordings in the caudate nucleus, the input station in the basal ganglia. Fitting the monkeys’ behaviors to a drift-diffusion model, we found that the monkeys used a rational heuristic to combine sensory and reward information. This heuristic is suboptimal but leads to good-enough outcomes. We also found that the monkeys’ reward biases were sensitive to the changes in the reward functions from session to session. This adaptive adjustment could be a possible reason underlying the individual variability in their decision strategies. By recording in the caudate nucleus, we found that it is involved in both the decision-formation and evaluation: before the monkey started accumulating sensory evidence, the caudate neurons represented the reward context that could be used to form a reward bias; during decision-formation, some caudate neurons jointly represented sensory evidence and reward information, which could facilitate the combining of sensory and reward information appropriately. After a decision is made, caudate nucleus represented both decision confidence and reward expectation, two evaluation-related quantities that influence the monkeys’ subsequent decision behaviors

Self-Optimizing and Pareto-Optimal Policies in General Environments based on Bayes-Mixtures

The problem of making sequential decisions in unknown probabilistic environments is studied. In cycle $t$ action $y_t$ results in perception $x_t$ and reward $r_t$, where all quantities in general may depend on the complete history. The perception $x_t$ and reward $r_t$ are sampled from the (reactive) environmental probability distribution $\mu$. This very general setting includes, but is not limited to, (partial observable, k-th order) Markov decision processes. Sequential decision theory tells us how to act in order to maximize the total expected reward, called value, if $\mu$ is known. Reinforcement learning is usually used if $\mu$ is unknown. In the Bayesian approach one defines a mixture distribution $\xi$ as a weighted sum of distributions \nu\in\M, where \M is any class of distributions including the true environment $\mu$. We show that the Bayes-optimal policy $p^\xi$ based on the mixture $\xi$ is self-optimizing in the sense that the average value converges asymptotically for all \mu\in\M to the optimal value achieved by the (infeasible) Bayes-optimal policy $p^\mu$ which knows $\mu$ in advance. We show that the necessary condition that \M admits self-optimizing policies at all, is also sufficient. No other structural assumptions are made on \M. As an example application, we discuss ergodic Markov decision processes, which allow for self-optimizing policies. Furthermore, we show that $p^\xi$ is Pareto-optimal in the sense that there is no other policy yielding higher or equal value in {\em all} environments \nu\in\M and a strictly higher value in at least one.Comment: 15 page

Dopamine restores reward prediction errors in old age.

Senescence affects the ability to utilize information about the likelihood of rewards for optimal decision-making. Using functional magnetic resonance imaging in humans, we found that healthy older adults had an abnormal signature of expected value, resulting in an incomplete reward prediction error (RPE) signal in the nucleus accumbens, a brain region that receives rich input projections from substantia nigra/ventral tegmental area (SN/VTA) dopaminergic neurons. Structural connectivity between SN/VTA and striatum, measured by diffusion tensor imaging, was tightly coupled to inter-individual differences in the expression of this expected reward value signal. The dopamine precursor levodopa (L-DOPA) increased the task-based learning rate and task performance in some older adults to the level of young adults. This drug effect was linked to restoration of a canonical neural RPE. Our results identify a neurochemical signature underlying abnormal reward processing in older adults and indicate that this can be modulated by L-DOPA

Primate Amygdala Neurons Simulate Decision Processes of Social Partners.

By observing their social partners, primates learn about reward values of objects. Here, we show that monkeys' amygdala neurons derive object values from observation and use these values to simulate a partner monkey's decision process. While monkeys alternated making reward-based choices, amygdala neurons encoded object-specific values learned from observation. Dynamic activities converted these values to representations of the recorded monkey's own choices. Surprisingly, the same activity patterns unfolded spontaneously before partner's choices in separate neurons, as if these neurons simulated the partner's decision-making. These "simulation neurons" encoded signatures of mutual-inhibitory decision computation, including value comparisons and value-to-choice conversions, resulting in accurate predictions of partner's choices. Population decoding identified differential contributions of amygdala subnuclei. Biophysical modeling of amygdala circuits showed that simulation neurons emerge naturally from convergence between object-value neurons and self-other neurons. By simulating decision computations during observation, these neurons could allow primates to reconstruct their social partners' mental states

Impulsivity and self-control during intertemporal decision making linked to the neural dynamics of reward value representation

A characteristic marker of impulsive decision making is the discounting of delayed rewards, demonstrated via choice preferences and choice-related brain activity. However, delay discounting may also arise from how subjective reward value is dynamically represented in the brain when anticipating an upcoming chosen reward. In the current study, brain activity was continuously monitored as human participants freely selected an immediate or delayed primary liquid reward and then waited for the specified delay before consuming it. The ventromedial prefrontal cortex (vmPFC) exhibited a characteristic pattern of activity dynamics during the delay period, as well as modulation during choice, that is consistent with the time-discounted coding of subjective value. The ventral striatum (VS) exhibited a similar activity pattern, but preferentially in impulsive individuals. A contrasting profile of delay-related and choice activation was observed in the anterior PFC (aPFC), but selectively in patient individuals. Functional connectivity analyses indicated that both vmPFC and aPFC exerted modulatory, but opposite, influences on VS activation. These results link behavioral impulsivity and self-control to dynamically evolving neural representations of future reward value, not just during choice, but also during postchoice delay periods

REPRESENTATION OF VALUE AND SALIENCE IN THE PRIMATE BRAIN

Most times, we choose goods and avoid evils. Sometimes, goods and evils choose us, which quickly grabs our attention. How do our brains represent our values so that we can choose? How do things we love or loathe automatically capture our attention? To investigate these two cognitive processes, we performed two experiments to study neuronal representations of value (worth) and salience (importance) in the primate brain. First, we tested whether neurons in LIP and amygdala encode the value or salience of choice-options early during value-based decision-making. To dissociate value from salience, we recorded neurons from LIP and the amygdala in monkeys making value-based decisions among options promising different reward-sizes and options threatening different penalty-sizes. Value increases with promised-reward but decreases with threatened-penalty. Salience increases with the intensity of both promises and threats. LIP neurons fired more for options promising more reward and also fired more for options threatening more penalty. Amygdala neurons fired more for options promising more reward but fired less for options threatening more penalty. Whereas LIP encoded salience early during decision-making, reflecting automatic capture of attention by the more important options, the amygdala encoded value early during the decision-making process. Second, we tested whether early reward-effects in neurons in LIP and amygdala depended on early automatic capture of visual attention by a more rewarding option. We dissociated cue-reward from cue-salience in LIP and amygdala using choice-options that could not acquire salience. The near-identical physical appearance of these precluded early automatic capture of attention by the large-reward option but not the monkeys’ ability to make optimal value-based decisions. LIP did not encode reward early during decision-making, only later and weakly. The number of LIP neurons with reward-effects was no greater than expected by chance. In amygdala, reward-effects were stronger and earlier than in LIP. Significant numbers of single amygdala neurons signaled the reward-size of the choice-option. Early reward-effects depend upon cue-salience in LIP but not the amygdala. The results from both experiments are consistent with the view that, early during value-based decision-making, neurons in LIP encode cue-salience whereas neurons in amygdala encode cue-value