Discounting Future Reward in an Uncertain World

Humans discount delayed relative to more immediate reward. A plausible explanation is that impatience arises partly from uncertainty, or risk, implicit in delayed reward. Existing theories of discounting-as-risk focus on a probability that delayed reward will not materialize. By contrast, we examine how uncertainty in the magnitude of delayed reward contributes to delay discounting. We propose a model wherein reward is discounted proportional to the rate of random change in its magnitude across time, termed volatility. We find evidence to support this model across three experiments (total N = 158). First, using a task where participants chose when to sell products, whose price dynamics they previously learned, we show discounting increases in line with price volatility. Second, we show that this effect pertains over naturalistic delays of up to 4 months. Using functional magnetic resonance imaging, we observe a volatility-dependent decrease in functional hippocampal–prefrontal coupling during intertemporal choice. Third, we replicate these effects in a larger online sample, finding that volatility discounting within each task correlates with baseline discounting outside of the task.We conclude that delay discounting partly reflects time-dependent uncertainty about reward magnitude, that is volatility. Our model captures how discounting adapts to volatility, thereby partly accounting for individual differences in impatience. Our imaging findings suggest a putative mechanism whereby uncertainty reduces prospective simulation of future outcomes

Dynamic computational models of risk and effort discounting in sequential decision making

Dissertation based on my publications in the field of risky behavior in dynamic, sequential decision making tasks.:1.- Introduction 2.- Context-dependent risk aversion: a model-based approach 3.- Modeling dynamic allocation of effort in a sequential task using discounting models 4.- General discussio

Dopaminergic Modulation of Intertemporal Choice

The overall goal of the studies presented in this dissertation is to improve our understanding of dopamine (DA)-associated changes in intertemporal preferences. Understanding these DA-mediated relationships is essential to our understanding of the continuous dimensions of human functioning and the promise of the RDoC framework (U.S. Department of Health and Human Services, National Institutes of 2016). Study 1 in this dissertation investigates DAergic modulation of intertemporal choice in healthy adult participants using the DA D2-receptor antagonist haloperidol and state-of-the-art computational approaches to further decompose the decision-process. Study 2 takes behavioral testing beyond the lab into real-life environments and assesses the effects of addiction related environments on intertemporal preferences and model-based reinforcement learning in regular slot machine gamblers. In Study 3 we examine whether patients with Tourette Syndrome show aberrations in intertemporal choice. This is of particular interest because Tourette Syndrome is associated with reward sensitivity and disturbances in DA neurotransmission. In Study 4 we investigate short- and long-term stability of intertemporal preferences as a function of acute and chronic deep brain stimulation in a cohort of Obsessive-Compulsive Disorder patients

Neural Dynamics Tracking Subjective Cognitive Effort

What patterns of brain activity reflect engagement with highly demanding cognitive tasks? How do these patterns relate to subjective, phenomenal effort? Answering these questions is critical to understanding what causes some people to experience cognitive tasks as more effortful than others. Subjective experience, in turn, is vital, with trait tendencies to exert effort having been linked to career and academic success. High subjective effort, as in schizophrenia and depression, can thus be extremely problematic. And yet, poor operational definitions have constrained research into basic questions about what neural dynamics track subjective effort. Here, a powerful, new behavioral economic operationalization is employed, in combination with fMRI, to investigate brain dynamics corresponding to subjectively costly cognitive effort. Brain regions varying in activity by working memory load and cognitive control demands are strong candidates for tracking subjective effort (Westbrook & Braver, 2015). To identify such regions, I examined BOLD data, collected while participants performed a well established working memory task (the N-back; Kirchner, 1958) that is both subjectively effortful, and for which subjective effort varies as a monotonic function of load (Westbrook et al., 2013). I focused my search within independently-defined networks of nodes that co-vary (within-network) across a wide range of brain states. Specifically, I examined a subset of a priori task-positive networks, as identified by Power et al. (2011), which typically show increasing, and a task-negative network which typically shows decreasing activity with greater load. Importantly, variation was examined over N-back loads for which data has never been published, thus the present study reveals novel insights about activity-load functions in independently-defined functional networks from very low (N = 1) to very high loads (N = 6). As expected, all task-positive networks showed robustly greater activity during the N-back. However, patterns of variation by load differed by network. While the task positive fronto-parietal (FP), dorsal attention (DorAtt), and salience (Sal) networks showed inverted-U functions, peaking mid-range (at the 2- or 3-back) and decreasing after, the cingulo-opercular network (CO) showed robust activity that did not further vary by load. Rather than encoding load per se, the CO simply encoded that a participant was performing the N-back. The task-negative default mode network (DMN) was robustly and increasingly de activated across all load levels examined. Given that both subjective effort (Westbrook et al., 2013) and DMN deactivation are approximately monotonic functions of load, the DMN is a strong candidate for tracking variation in subjective effort with load. By contrast, inverted-U functions in the FP, Sal, and DorAtt networks do not straightforwardly map to monotonically increasing effort. Performance measures instead suggest that inverted-U functions tracked individual differences in adaptive strategy shifting. Namely, when participants were divided by 3-back performance, better performers showed a pronounced inverted-U (over N = 1—3) while worse performers did not. Interestingly, a similar pattern was found when dividing participants according subjective effort, providing tentative support to a hypothesis that subjective effort acts as a cue to shift strategies adaptively under excessive demands. In any case, surprisingly, in none of the networks did load-specific changes in brain activity predict load-specific changes in subjective effort. Critically, although load-specific patterns of brain activity did not predict subjective effort, load-independent brain activity predicted individual differences in subjective effort. Namely, higher average brain activity in any of the task-positive networks predicted greater subjective effort. At the sub-network level, this was notably true for two key regions that have been implicated as core components of a cognitive control system, and also hypothesized to track effort costs: the dorsal anterior cingulate cortex (dACC) and the dorsolateral prefrontal cortex (dlPFC) (McGuire et al., 2010). Importantly, after controlling for performance, the dACC remained a reliable predictor of subjective effort, while the dlPFC did not, supporting that the dACC tracks cognitive effort apart from task difficulty (while the dlPFC may not). This is consistent with strong prior theory implicating the dACC in regulating the intensity of cognitive control in response to flagging performance and in proportion to the expected value of doing so (Shenhav et al., 2013). The present results begin to answer basic questions about how the brain tracks subjective effort. They also lay the foundation for future work addressing why subjective effort can be so much greater for some individuals, like those with schizophrenia or depression, and also future work developing interventions for promoting desirable effort expenditure

Virtual Reality as a Chance to Reconcile Ecological Validity and Experimental Control in Cue-Reactivity Research

High-performance virtual reality (VR) technology has opened new possibilities for the examination of the reactivity towards addiction-related cues (cue-reactivity) in addiction. We conducted two studies employing a virtual reality cue-reactivity design for participants suffering from gambling disorder (GD), that combined the assessment of subjective, physiological, and behavioral cue-reactivity. The first study aimed to examine the reliability of temporal discounting measures in VR and standard lab environments in a group of non-gambling control participants. Additionally, we aimed to explore the feasibility of applying sequential sampling models to temporal discounting data obtained in VR. The second study employed the VR design validated in the first study to investigate the subjective, behavioral, and physiological effects of VR gambling environment exposure in a group of regular gamblers (GD group) and matched non-gambling controls. Overall, the results obtained by both studies presented in this dissertation project revealed further evidence for the validity of temporal discounting and the two-step task as possible diagnostic markers of GD. We demonstrated high reliability of the temporal discounting task and reproduced established group differences in decision-making between participants suffering from GD and non-gambling controls in both behavioral tasks. Additionally, we showed that behavioral data obtained by both tasks in VR can be meaningfully interpreted with comprehensive computational modelling, especially with models including RTs such as the drift-diffusion model. In the context of cue-reactivity we found mixed results. While our design was effective in eliciting subjective craving in participants suffering from GD, we observed little evidence for behavioral or sympathetic physiological cue-reactivity

Time for a Change? Brain Activity and Behavioral Performance Reveal Different Dynamics at Short, Intermediate, and Long Delay Intervals During a Delay Discounting Task

In our day to day lives, the ability to make goal-oriented decisions plays a crucial role in both our work and social lives. Therefore, researchers have examined how factors such as a varying reward or delay may affect decision making. One’s performance when making intertemporal choices, decisions made between a smaller and sooner (SS) reward and a larger and later (LL) reward, are often examined to study these factors. Although time and reward magnitude are important dimensions when individuals make decisions during delay discounting, little is known about the relationship between time perception, reward magnitude, and underlying neural mechanisms. To address this gap in literature, participants completed a modified delay discounting task during fMRI with stimuli that included fluctuating reward and delay values. An exploratory factor analysis using behavioral data identified three categories of delays and reward values that were used to create brain contrasts. In these comparisons, the middle frontal gyrus and cingulate gyrus seemed to be more involved when choosing rewards of greater magnitude while the medial frontal gyrus and insula were found to be more active for longer delays. Our results suggest that delay and reward determination are handled by separate neural networks

Goal-Directed Decision Making with Spiking Neurons.

UNLABELLED: Behavioral and neuroscientific data on reward-based decision making point to a fundamental distinction between habitual and goal-directed action selection. The formation of habits, which requires simple updating of cached values, has been studied in great detail, and the reward prediction error theory of dopamine function has enjoyed prominent success in accounting for its neural bases. In contrast, the neural circuit mechanisms of goal-directed decision making, requiring extended iterative computations to estimate values online, are still unknown. Here we present a spiking neural network that provably solves the difficult online value estimation problem underlying goal-directed decision making in a near-optimal way and reproduces behavioral as well as neurophysiological experimental data on tasks ranging from simple binary choice to sequential decision making. Our model uses local plasticity rules to learn the synaptic weights of a simple neural network to achieve optimal performance and solves one-step decision-making tasks, commonly considered in neuroeconomics, as well as more challenging sequential decision-making tasks within 1 s. These decision times, and their parametric dependence on task parameters, as well as the final choice probabilities match behavioral data, whereas the evolution of neural activities in the network closely mimics neural responses recorded in frontal cortices during the execution of such tasks. Our theory provides a principled framework to understand the neural underpinning of goal-directed decision making and makes novel predictions for sequential decision-making tasks with multiple rewards. SIGNIFICANCE STATEMENT: Goal-directed actions requiring prospective planning pervade decision making, but their circuit-level mechanisms remain elusive. We show how a model circuit of biologically realistic spiking neurons can solve this computationally challenging problem in a novel way. The synaptic weights of our network can be learned using local plasticity rules such that its dynamics devise a near-optimal plan of action. By systematically comparing our model results to experimental data, we show that it reproduces behavioral decision times and choice probabilities as well as neural responses in a rich set of tasks. Our results thus offer the first biologically realistic account for complex goal-directed decision making at a computational, algorithmic, and implementational level.This research was supported by the Swiss National Science Foundation (J.F., Grant PBBEP3 146112) and the Wellcome Trust (J.F. and M.L.).This is the author accepted manuscript. It is currently under an indefinite embargo pending publication by the Society for Neuroscience