Simulating future value in intertemporal choice

The laboratory study of how humans and other animals trade-off value and time has a long and storied history, and is the subject of a vast literature. However, despite a long history of study, there is no agreed upon mechanistic explanation of how intertemporal choice preferences arise. Several theorists have recently proposed model-based reinforcement learning as a candidate framework. This framework describes a suite of algorithms by which a model of the environment, in the form of a state transition function and reward function, can be converted on-line into a decision. The state transition function allows the model-based system to make decisions based on projected future states, while the reward function assigns value to each state, together capturing the necessary components for successful intertemporal choice. Empirical work has also pointed to a possible relationship between increased prospection and reduced discounting. In the current paper, we look for direct evidence of a relationship between temporal discounting and model-based control in a large new data set (n = 168). However, testing the relationship under several different modeling formulations revealed no indication that the two quantities are related

BOLD and its connection to dopamine release in human striatum: a cross-cohort comparison

Activity in midbrain dopamine neurons modulates the release of dopamine in terminal structures including the striatum, and controls reward-dependent valuation and choice. This fluctuating release of dopamine is thought to encode reward prediction error (RPE) signals and other value-related information crucial to decision-making, and such models have been used to track prediction error signals in the striatum as encoded by BOLD signals. However, until recently there have been no comparisons of BOLD responses and dopamine responses except for one clear correlation of these two signals in rodents. No such comparisons have been made in humans. Here, we report on the connection between the RPE-related BOLD signal recorded in one group of subjects carrying out an investment task, and the corresponding dopamine signal recorded directly using fast-scan cyclic voltammetry in a separate group of Parkinson's disease patients undergoing DBS surgery while performing the same task. The data display some correspondence between the signal types; however, there is not a one-to-one relationship. Further work is necessary to quantify the relationship between dopamine release, the BOLD signal and the computational models that have guided our understanding of both at the level of the striatum.This article is part of the themed issue 'Interpreting BOLD: a dialogue between cognitive and cellular neuroscience'

Loss Aversion Correlates With the Propensity to Deploy Model-Based Control

Reward-based decision making is thought to be driven by at least two different types of decision systems: a simple stimulus–response cache-based system which embodies the common-sense notion of “habit,” for which model-free reinforcement learning serves as a computational substrate, and a more deliberate, prospective, model-based planning system. Previous work has shown that loss aversion, a well-studied measure of how much more on average individuals weigh losses relative to gains during decision making, is reduced when participants take all possible decisions and outcomes into account including future ones, relative to when they myopically focus on the current decision. Model-based control offers a putative mechanism for implementing such foresight. Using a well-powered data set (N = 117) in which participants completed two different tasks designed to measure each of the two quantities of interest, and four models of choice data for these tasks, we found consistent evidence of a relationship between loss aversion and model-based control but in the direction opposite to that expected based on previous work: loss aversion had a positive relationship with model-based control. We did not find evidence for a relationship between either decision system and risk aversion, a related aspect of subjective utility

Computational phenotyping of two-person interactions reveals differential neural response to depth-of-thought.

Reciprocating exchange with other humans requires individuals to infer the intentions of their partners. Despite the importance of this ability in healthy cognition and its impact in disease, the dimensions employed and computations involved in such inferences are not clear. We used a computational theory-of-mind model to classify styles of interaction in 195 pairs of subjects playing a multi-round economic exchange game. This classification produces an estimate of a subject's depth-of-thought in the game (low, medium, high), a parameter that governs the richness of the models they build of their partner. Subjects in each category showed distinct neural correlates of learning signals associated with different depths-of-thought. The model also detected differences in depth-of-thought between two groups of healthy subjects: one playing patients with psychiatric disease and the other playing healthy controls. The neural response categories identified by this computational characterization of theory-of-mind may yield objective biomarkers useful in the identification and characterization of pathologies that perturb the capacity to model and interact with other humans

Human substantia nigra and ventral tegmental area involvement in computing social error signals during the ultimatum game

Social norms play an essential role in our societies, and since the social environment is changing constantly, our internal models of it also need to change. In humans, there is mounting evidence that neural structures such as the insula and the ventral striatum are involved in detecting norm violation and updating internal models. However, because of methodological challenges, little is known about the possible involvement of midbrain structures in detecting norm violation and updating internal models of our norms. Here we used high-resolution cardiac-gated functional magnetic resonance imaging and a norm adaptation paradigm in healthy adults to investigate the role of the substantia nigra/ventral tegmental area (SN/VTA) complex in tracking signals related to norm violation that can be used to update internal norms. We show that the SN/VTA codes for the norm's variance prediction error (PE) and norm PE with spatially distinct regions coding for negative and positive norm PE. These results point to a common role played by the SN/VTA complex in supporting both simple reward-based and social decision making

An efficiency framework for valence processing systems inspired by soft cross-wiring

Recent experiments suggest that subsecond dopamine delivery to human striatum encodes a combination of reward prediction errors and counterfactual errors thus composing the actual with the possible into one neurochemical signal. Here, we present a model where the counterfactual part of these striatal dopamine fluctuations originates in another valuation system that shadows the dopamine system by acting as its near-antipode in terms of spike-rate encoding yet co-releases dopamine alongside its own native neurotransmitter. We show that such a hypothesis engenders important representational consequences where valence processing appears subject to the efficient encoding considerations common to the visual and auditory systems. This new perspective opens up important computational consequences for understanding how value-predicting information should integrate with sensory processing streams

Asymmetry in functional connectivity of the human habenula revealed by high-resolution cardiac-gated resting state imaging

The habenula is a hub for cognitive and emotional signals that are relayed to the aminergic centers in the midbrain and, thus, plays an important role in goal-oriented behaviors. Although it is well described in rodents and non-human primates, the habenula functional network remains relatively uncharacterized in humans, partly because of the methodological challenges associated with the functional magnetic resonance imaging of small structures in the brain. Using high-resolution cardiac-gated resting state imaging in healthy humans and precisely identifying each participants' habenula, we show that the habenula is functionally coupled with the insula, parahippocampus, thalamus, periaqueductal grey, pons, striatum and substantia nigra/ventral tegmental area complex. Furthermore, by separately examining and comparing the functional maps from the left and right habenula, we provide the first evidence of an asymmetry in the functional connectivity of the habenula in humans. Hum Brain Mapp 37:2602-2615, 2016. © 2016 The Authors Human Brain Mapping Published by Wiley Periodicals, Inc

Neural signatures of strategic types in a two-person bargaining game

The management and manipulation of our own social image in the minds of others requires difficult and poorly understood computations. One computation useful in social image management is strategic deception: our ability and willingness to manipulate other people's beliefs about ourselves for gain. We used an interpersonal bargaining game to probe the capacity of players to manage their partner's beliefs about them. This probe parsed the group of subjects into three behavioral types according to their revealed level of strategic deception; these types were also distinguished by neural data measured during the game. The most deceptive subjects emitted behavioral signals that mimicked a more benign behavioral type, and their brains showed differential activation in right dorsolateral prefrontal cortex and left Brodmann area 10 at the time of this deception. In addition, strategic types showed a significant correlation between activation in the right temporoparietal junction and expected payoff that was absent in the other groups. The neurobehavioral types identified by the game raise the possibility of identifying quantitative biomarkers for the capacity to manipulate and maintain a social image in another person's mind

Neural differences in self-perception during illness and after weight-recovery in anorexia nervosa

Anorexia nervosa (AN) is a severe mental illness characterized by problems with self-perception. Whole-brain neural activations in healthy women, women with AN and women in long-term weight recovery following AN were compared using two functional magnetic resonance imaging tasks probing different aspects of self-perception. The Social Identity-V2 task involved consideration about oneself and others using socially descriptive adjectives. Both the ill and weight-recovered women with AN engaged medial prefrontal cortex less than healthy women for self-relevant cognitions, a potential biological trait difference. Weight-recovered women also activated the inferior frontal gyri and dorsal anterior cingulate more for direct self-evaluations than for reflected self-evaluations, unlike both other groups, suggesting that recovery may include compensatory neural changes related to social perspectives. The Faces task compared viewing oneself to a stranger. Participants with AN showed elevated activity in the bilateral fusiform gyri for self-images, unlike the weight-recovered and healthy women, suggesting cognitive distortions about physical appearance are a state rather than trait problem in this disease. Because both ill and recovered women showed neural differences related to social self-perception, but only recovered women differed when considering social perspectives, these neurocognitive targets may be particularly important for treatment

Belief about Nicotine Modulates Subjective Craving and Insula Activity in Deprived Smokers

Little is known about the specific neural mechanisms through which cognitive factors influence craving and associated brain responses, despite the initial success of cognitive therapies in treating drug addiction. In this study, we investigated how cognitive factors such as beliefs influence subjective craving and neural activities in nicotine-addicted individuals using model-based functional magnetic resonance imaging (fMRI) and neuropharmacology. Deprived smokers (N = 24) participated in a two-by-two balanced placebo design, which crossed beliefs about nicotine (told "nicotine" vs. told "no nicotine") with the nicotine content in a cigarette (nicotine vs. placebo) which participants smoked immediately before performing a fMRI task involving reward learning. Subjects' reported craving was measured both before smoking and after the fMRI session. We found that first, in the presence of nicotine, smokers demonstrated significantly reduced craving after smoking when told "nicotine in cigarette" but showed no change in craving when told "no nicotine." Second, neural activity in the insular cortex related to craving was only significant when smokers were told "nicotine" but not when told "no nicotine." Both effects were absent in the placebo condition. Third, insula activation related to computational learning signals was modulated by belief about nicotine regardless of nicotine's presence. These results suggest that belief about nicotine has a strong impact on subjective craving and insula responses related to both craving and learning in deprived smokers, providing insights into the complex nature of belief-drug interactions