Encoding of Marginal Utility across Time in the Human Brain

Marginal utility theory prescribes the relationship between the objective property of the magnitude of rewards and their subjective value. Despite its pervasive influence, however, there is remarkably little direct empirical evidence for such a theory of value, let alone of its neurobiological basis. We show that human preferences in an intertemporal choice task are best described by a model that integrates marginally diminishing utility with temporal discounting. Using functional magnetic resonance imaging, we show that activity in the dorsal striatum encodes both the marginal utility of rewards, over and above that which can be described by their magnitude alone, and the discounting associated with increasing time. In addition, our data show that dorsal striatum may be involved in integrating subjective valuation systems inherent to time and magnitude, thereby providing an overall metric of value used to guide choice behavior. Furthermore, during choice, we show that anterior cingulate activity correlates with the degree of difficulty associated with dissonance between value and time. Our data support an integrative architecture for decision making, revealing the neural representation of distinct subcomponents of value that may contribute to impulsivity and decisiveness

Discounting of reward sequences: a test of competing formal models of hyperbolic discounting

Humans are known to discount future rewards hyperbolically in time. Nevertheless, a formal recursive model of hyperbolic discounting has been elusive until recently, with the introduction of the hyperbolically discounted temporal difference (HDTD) model. Prior to that, models of learning (especially reinforcement learning) have relied on exponential discounting, which generally provides poorer fits to behavioral data. Recently, it has been shown that hyperbolic discounting can also be approximated by a summed distribution of exponentially discounted values, instantiated in the μAgents model. The HDTD model and the μAgents model differ in one key respect, namely how they treat sequences of rewards. The μAgents model is a particular implementation of a Parallel discounting model, which values sequences based on the summed value of the individual rewards whereas the HDTD model contains a non-linear interaction. To discriminate among these models, we observed how subjects discounted a sequence of three rewards, and then we tested how well each candidate model fit the subject data. The results show that the Parallel model generally provides a better fit to the human data

Impulsivity and self-control during intertemporal decision making linked to the neural dynamics of reward value representation

A characteristic marker of impulsive decision making is the discounting of delayed rewards, demonstrated via choice preferences and choice-related brain activity. However, delay discounting may also arise from how subjective reward value is dynamically represented in the brain when anticipating an upcoming chosen reward. In the current study, brain activity was continuously monitored as human participants freely selected an immediate or delayed primary liquid reward and then waited for the specified delay before consuming it. The ventromedial prefrontal cortex (vmPFC) exhibited a characteristic pattern of activity dynamics during the delay period, as well as modulation during choice, that is consistent with the time-discounted coding of subjective value. The ventral striatum (VS) exhibited a similar activity pattern, but preferentially in impulsive individuals. A contrasting profile of delay-related and choice activation was observed in the anterior PFC (aPFC), but selectively in patient individuals. Functional connectivity analyses indicated that both vmPFC and aPFC exerted modulatory, but opposite, influences on VS activation. These results link behavioral impulsivity and self-control to dynamically evolving neural representations of future reward value, not just during choice, but also during postchoice delay periods

Impact of Size and Delay on Neural Activity in the Rat Limbic Corticostriatal System

A number of factors influence an animal’s economic decisions. Two most commonly studied are the magnitude of and delay to reward. To investigate how these factors are represented in the firing rates of single neurons, we devised a behavioral task that independently manipulated the expected delay to and size of reward. Rats perceived the differently delayed and sized rewards as having different values and were more motivated under short delay and big-reward conditions than under long delay and small reward conditions as measured by percent choice, accuracy, and reaction time. Since the creation of this task, we have recorded from several different brain areas including, orbitofrontal cortex, striatum, amygdala, substantia nigra pars reticulata, and midbrain dopamine neurons. Here, we review and compare those data with a substantial focus on those areas that have been shown to be critical for performance on classic time discounting procedures and provide a potential mechanism by which they might interact when animals are deciding between differently delayed rewards. We found that most brain areas in the cortico-limbic circuit encode both the magnitude and delay to reward delivery in one form or another, but only a few encode them together at the single neuron level

High temporal discounters overvalue immediate rewards rather than undervalue future rewards : an event-related brain potential study

Impulsivity is characterized in part by heightened sensitivity to immediate relative to future rewards. Although previous research has suggested that "high discounters" in intertemporal choice tasks tend to prefer immediate over future rewards because they devalue the latter, it remains possible that they instead overvalue immediate rewards. To investigate this question, we recorded the reward positivity, a component of the event-related brain potential (ERP) associated with reward processing, with participants engaged in a task in which they received both immediate and future rewards and nonrewards. The participants also completed a temporal discounting task without ERP recording. We found that immediate but not future rewards elicited the reward positivity. High discounters also produced larger reward positivities to immediate rewards than did low discounters, indicating that high discounters relatively overvalued immediate rewards. These findings suggest that high discounters may be more motivated than low discounters to work for monetary rewards, irrespective of the time of arrival of the incentives

I can't wait! Neural reward signals in impulsive individuals exaggerate the difference between immediate and future rewards

Waiting for rewards is difficult, and highly impulsive individuals with low self-control have an especially hard time with it. Here, we investigated whether neural responses to rewards in a delayed gratification task predict impulsivity and self-control. The EEG was recorded from participants engaged in a guessing game in which on each trial they could win either a large or small reward, paid either now or after 6 months. Ratings confirmed that participants preferred immediate, large rewards over small, delayed rewards. Electrophysiological reward signals reflecting the difference between immediate and future rewards predicted self-report measures of impulsivity and self-control. Further, these signals were highly reliable across two sessions over a 1-week interval, showing high temporal stability like stable personality traits. These results suggest that greater valuation of immediate rewards causes impulsive individuals to redirect control away from delayed rewards, indicating why it is so hard for them to wait

Behavioral modeling of human choices reveals dissociable effects of physical effort and temporal delay on reward devaluation

There has been considerable interest from the fields of biology, economics, psychology, and ecology about how decision costs decrease the value of rewarding outcomes. For example, formal descriptions of how reward value changes with increasing temporal delays allow for quantifying individual decision preferences, as in animal species populating different habitats, or normal and clinical human populations. Strikingly, it remains largely unclear how humans evaluate rewards when these are tied to energetic costs, despite the surge of interest in the neural basis of effort-guided decision-making and the prevalence of disorders showing a diminished willingness to exert effort (e.g., depression). One common assumption is that effort discounts reward in a similar way to delay. Here we challenge this assumption by formally comparing competing hypotheses about effort and delay discounting. We used a design specifically optimized to compare discounting behavior for both effort and delay over a wide range of decision costs (Experiment 1). We then additionally characterized the profile of effort discounting free of model assumptions (Experiment 2). Contrary to previous reports, in both experiments effort costs devalued reward in a manner opposite to delay, with small devaluations for lower efforts, and progressively larger devaluations for higher effort-levels (concave shape). Bayesian model comparison confirmed that delay-choices were best predicted by a hyperbolic model, with the largest reward devaluations occurring at shorter delays. In contrast, an altogether different relationship was observed for effort-choices, which were best described by a model of inverse sigmoidal shape that is initially concave. Our results provide a novel characterization of human effort discounting behavior and its first dissociation from delay discounting. This enables accurate modelling of cost-benefit decisions, a prerequisite for the investigation of the neural underpinnings of effort-guided choice and for understanding the deficits in clinical disorders characterized by behavioral inactivity

Individuals With Amnesia are not Stuck in Time: Evidence From Risky Decision-Making, Intertemporal Choice, and Scaffolded Narratives

This dissertation investigates the supposition that individuals with amnesia are cognitively stuck in time. In Experiment 1, I used a Galton-Crovitz cueing paradigm to test etiologically diverse amnesic cases on their ability to richly recollect autobiographical episodic memories and imagine future experiences. In Experiment 2, I use two behavioural economics tasks (a risky decision-making task and an intertemporal choice task) to examine whether amnesic cases judgment and decision-making reflects proneness to risky choices or steep disregard for the future. In Experiment 3, I examine the flexibility of amnesics intertemporal choice by testing whether cueing them with personal future events increases their value of future rewards as it does in healthy controls. In Experiment 4, I attempt to decrease the severity of amnesic cases episodic memory and prospection impairment by using structured and personally meaningful cues rather than the single cue words featured in the Galton-Crovitz paradigm. I replicated existing research showing that those with MTL damage have impaired ability to (re)construct rich and detailed narratives of past and future experiences, and I extended this finding for the first time to a lateral dorsal thalamic stroke case (Experiment 1). Despite this impairment in mental time travel, the same amnesic cases made financial decisions that a) systematically considered and valued the future and b) showed normal sensitivity to risk (Experiment 2). The normalcy of intertemporal choice in amnesia extends beyond basic rates of future reward discounting in intertemporal choice. In controls, cues to imagine future experiences can modulate decision-making by increasing the value one places on future rewards. Here, most amnesic cases also retain this modulatory effect, despite having impaired ability to generate detailed representations of future experiences (Experiment 3). Finally, I found that the severity of episodic prospection impairment in MTL amnesia is cue-dependent and likely overestimated in current research: specific, personally meaningful cues lead to an appreciable reduction of episodic prospection impairment over single cue words for those with mild-moderate amnesia (Experiment 4). Collectively, results challenge assumptions that amnesic populations are cognitively confined to the present and call for refinement to simple accounts of limited temporality in individuals with amnesia

Neurobiological bases of intertemporal choices: A comprehensive review

Intertemporal choices (ICs) are choices that involve trade-off between costs and benefits that take place at differ ent moments in time. The aim of this article is to present a comprehensive literature review on neurobiological bases of IC. We present the functional models of IC and data from neuroimaging studies, namely ALE analysis. With this paper we intended to show the presence of immediate value preference beyond that predicted by a single-parameter exponential discounting model and its mapping to the dual-systems model for brain function. Studies indicate that individuals tend to show inconsistent preferences depending on the time until the rewards are available and support a perspective that intertemporal evaluation reflects neural mechanisms that differ from other forms of choice, although associated value signals are later represented in the context of a common reward system. The IC induces activations in a “nuclear network” and auxiliary areas including inferior prefrontal cortex, medial prefrontal cortex, temporo-parietal cortex, and peri-splenial posterior cingulate. The network of areas sensitive to value is comprised of several regions that include ventral striatum, medial prefrontal cortex, orbitofrontal cortex, and anterior insula. Evidence from neuroimaging and EEG studies corroborates that choices are determined by a dual evaluation system.info:eu-repo/semantics/publishedVersio