It's all about timing : an electrophysiological examination of feedback-based learning with immediate and delayed feedback

Feedback regarding an individual's action can occur immediately or with a temporal delay. Processing of feedback that varies in its delivery time is proposed to engage different brain mechanisms. fMRI data implicate the striatum in the processing of immediate feedback, and the medial temporal lobe (MTL) in the processing of delayed feedback. The present study offers an electrophysiological examination of feedback processing in the context of timing, by studying the effects of feedback timing on the feedback-related negativity (FRN), a product of the midbrain dopamine system, and elucidating whether the N170 ERP component could capture MTL activation associated with the processing of delayed feedback. Participants completed a word-object paired association learning task; they received feedback 500 ms (immediate feedback condition) following a button press during the learning of two sets of 14 items, and at a delay of 6500 ms (delayed feedback condition) during the learning of the other two sets. The results indicated that while learning outcomes did not differ under the two timing conditions, Event Related Potential (ERPs) pointed to differential activation of the examined ERP components. FRN amplitude was found to be larger following the immediate feedback condition when compared with the delayed feedback condition, and sensitive to valence and learning only under the immediate feedback condition. Additionally, the amplitude of the N170 was found larger following the delayed feedback condition when compared with the immediate feedback condition. Taken together, the findings of the present study support the contention that the processing of delayed feedback involves a shift away from midbrain dopamine activation to the recruitment of the MTL

Encoding of Marginal Utility across Time in the Human Brain

Marginal utility theory prescribes the relationship between the objective property of the magnitude of rewards and their subjective value. Despite its pervasive influence, however, there is remarkably little direct empirical evidence for such a theory of value, let alone of its neurobiological basis. We show that human preferences in an intertemporal choice task are best described by a model that integrates marginally diminishing utility with temporal discounting. Using functional magnetic resonance imaging, we show that activity in the dorsal striatum encodes both the marginal utility of rewards, over and above that which can be described by their magnitude alone, and the discounting associated with increasing time. In addition, our data show that dorsal striatum may be involved in integrating subjective valuation systems inherent to time and magnitude, thereby providing an overall metric of value used to guide choice behavior. Furthermore, during choice, we show that anterior cingulate activity correlates with the degree of difficulty associated with dissonance between value and time. Our data support an integrative architecture for decision making, revealing the neural representation of distinct subcomponents of value that may contribute to impulsivity and decisiveness

Modelling the Effect of Dorsal Raphe Serotonin Neurons on Patience for Future Rewards

Serotonin is a neurotransmitter that is implicated in many basic human functions and behaviours and is closely associated with happiness, depression and reward processing. In particular it appears to be involved in suppressing responses to distracting stimuli while waiting for a delayed reward. Here we present a system level model of the limbic system which is able to generate a serotonin (5-hydroxytryptamine [5HT]) signal so that a simulated animal waits for a delayed reward. We propose that the 5HT signal is computed by a network involving the medial Orbital Frontal Cortex (mOFC), medial Pre Frontal Cortex (mPFC), Dorsal Raphe Nucleus (DRN)and the Nucleus Accumbens Core (NAcc). The serotonin signal encodes pre-reward liking, motivation throughout the trial and delayed reward waiting. We have successfully replicated the behaviour and dynamics of laboratory studies. With the help of this model we can predict that low levels of serotonin indirectly cause less encountered rewards because the animal gives up too early

Single Dose of a Dopamine Agonist Impairs Reinforcement Learning in Humans: Evidence from Event-related Potentials and Computational Modeling of Striatal-Cortical Function

Animal findings have highlighted the modulatory role of phasic dopamine (DA) signaling in incentive learning, particularly in the acquisition of reward-related behavior. In humans, these processes remain largely unknown. In a recent study, we demonstrated that a single low dose of a D2/D3 agonist (pramipexole) - assumed to activate DA autoreceptors and thus reduce phasic DA bursts - impaired reward learning in healthy subjects performing a probabilistic reward task. The purpose of this study was to extend these behavioral findings using event-related potentials and computational modeling. Compared with the placebo group, participants receiving pramipexole showed increased feedback-related negativity to probabilistic rewards and decreased activation in dorsal anterior cingulate regions previously implicated in integrating reinforcement history over time. Additionally, findings of blunted reward learning in participants receiving pramipexole were simulated by reduced presynaptic DA signaling in response to reward in a neural network model of striatal-cortical function. These preliminary findings offer important insights on the role of phasic DA signals on reinforcement learning in humans and provide initial evidence regarding the spatiotemporal dynamics of brain mechanisms underlying these processes.Psycholog

High temporal discounters overvalue immediate rewards rather than undervalue future rewards : an event-related brain potential study

Impulsivity is characterized in part by heightened sensitivity to immediate relative to future rewards. Although previous research has suggested that "high discounters" in intertemporal choice tasks tend to prefer immediate over future rewards because they devalue the latter, it remains possible that they instead overvalue immediate rewards. To investigate this question, we recorded the reward positivity, a component of the event-related brain potential (ERP) associated with reward processing, with participants engaged in a task in which they received both immediate and future rewards and nonrewards. The participants also completed a temporal discounting task without ERP recording. We found that immediate but not future rewards elicited the reward positivity. High discounters also produced larger reward positivities to immediate rewards than did low discounters, indicating that high discounters relatively overvalued immediate rewards. These findings suggest that high discounters may be more motivated than low discounters to work for monetary rewards, irrespective of the time of arrival of the incentives

Temporal Decision-Making: Insights from Cognitive Neuroscience

Decisions frequently have consequences that play out over time and these temporal factors can exert strong influences on behavior. For example, decision-makers exhibit delay discounting, behaving as though immediately consumable goods are more valuable than those available only after some delay. With the use of functional magnetic resonance imaging, we are now beginning to characterize the physiological bases of such behavior in humans and to link work on this topic from neuroscience, psychology, and economics. Here we review recent neurocognitive investigations of temporal decision-making and outline the theoretical picture that is beginning to take shape. Taken as a whole, this body of work illustrates the progress made in understanding temporal choice behavior. However, we also note several questions that remain unresolved and areas where future work is needed

Individual differences in delay discounting and nicotine self-administration in rats

Delay discounting—a behavioral measure of impulsivity defined as a tendency to prefer a small, immediate reward over a larger reward delayed in time—has been extensively linked with tobacco smoking. However, the causal direction of this relationship remains unclear. One possibility is that delay discounting may be a marker for an underlying vulnerability to nicotine reinforcement—a possibility which can be isolated using an animal model. In the current study, we investigated whether indifference points derived using an adjustable delay procedure of delay discounting predicted several indices of nicotine reinforcement in rats, including rate of acquisition of nicotine self-administration, break point reached on a progressive ratio schedule of reinforcement, or a shift in the dose-response curve. Stable indifference points were assessed for 63 male Sprague-Dawley rats, and extreme groups of highly impulsive (HI; n=15) and low impulsive (LI; n=11) rats were selected to self-administer nicotine. Rats responded by nose poking for infusions of 0.03 mg/kg nicotine during 1 hour daily sessions. After a 20 session acquisition period, rats completed 3 4-hour progressive ratio sessions, during which the response requirement was increased after each infusion earned. This was followed by 3 1-hour fixed ratio sessions at each of 3 nicotine doses, presented in ascending order (0.015, 0.03, and 0.09 mg/kg). All but one rat (HI group) acquired stable nicotine self-administration; however, no group differences in rate of acquisition were observed. HI and LI rats did not differ in their responses on a progressive ratio schedule or infusions earned at any dose of nicotine, although a significant dose-response effect was observed overall. Indifference points reassessed after self-administration were highly correlated with original indifference points, and mean indifference points for each group at the second assessment did not differ significantly from baseline assessment. These results suggest that delay discounting is a highly reliable measure, but may not be a predictive marker for increased vulnerability to nicotine self-administration in rats

Active inference, evidence accumulation, and the urn task

Deciding how much evidence to accumulate before making a decision is a problem we and other animals often face, but one that is not completely understood. This issue is particularly important because a tendency to sample less information (often known as reflection impulsivity) is a feature in several psychopathologies, such as psychosis. A formal understanding of information sampling may therefore clarify the computational anatomy of psychopathology. In this theoretical letter, we consider evidence accumulation in terms of active (Bayesian) inference using a generic model of Markov decision processes. Here, agents are equipped with beliefs about their own behavior--in this case, that they will make informed decisions. Normative decision making is then modeled using variational Bayes to minimize surprise about choice outcomes. Under this scheme, different facets of belief updating map naturally onto the functional anatomy of the brain (at least at a heuristic level). Of particular interest is the key role played by the expected precision of beliefs about control, which we have previously suggested may be encoded by dopaminergic neurons in the midbrain. We show that manipulating expected precision strongly affects how much information an agent characteristically samples, and thus provides a possible link between impulsivity and dopaminergic dysfunction. Our study therefore represents a step toward understanding evidence accumulation in terms of neurobiologically plausible Bayesian inference and may cast light on why this process is disordered in psychopathology