Separate encoding of model-based and model-free valuations in the human brain

Behavioral studies have long shown that humans solve problems in two ways, one intuitive and fast (System 1, model-free), and the other reflective and slow (System 2, model-based). The neurobiological basis of dual process problem solving remains unknown due to challenges of separating activation in concurrent systems. We present a novel neuroeconomic task that predicts distinct subjective valuation and updating signals corresponding to these two systems. We found two concurrent value signals in human prefrontal cortex: a System 1 model-free reinforcement signal and a System 2 model-based Bayesian signal. We also found a System 1 updating signal in striatal areas and a System 2 updating signal in lateral prefrontal cortex. Further, signals in prefrontal cortex preceded choices that are optimal according to either updating principle, while signals in anterior cingulate cortex and globus pallidus preceded deviations from optimal choice for reinforcement learning. These deviations tended to occur when uncertainty regarding optimal values was highest, suggesting that disagreement between dual systems is mediated by uncertainty rather than conflict, confirming recent theoretical proposals

Evidence for a Common Representation of Decision Values for Dissimilar Goods in Human Ventromedial Prefrontal Cortex

To make economic choices between goods, the brain needs to compute representations of their values. A great deal of research has been performed to determine the neural correlates of value representations in the human brain. However, it is still unknown whether there exists a region of the brain that commonly encodes decision values for different types of goods, or if, in contrast, the values of different types of goods are represented in distinct brain regions. We addressed this question by scanning subjects with functional magnetic resonance imaging while they made real purchasing decisions among different categories of goods (food, nonfood consumables, and monetary gambles). We found activity in a key brain region previously implicated in encoding goal-values: the ventromedial prefrontal cortex (vmPFC) was correlated with the subjects' value for each category of good. Moreover, we found a single area in vmPFC to be correlated with the subjects' valuations for all categories of goods. Our results provide evidence that the brain encodes a "common currency" that allows for a shared valuation for different categories of goods

Risk, Unexpected Uncertainty, and Estimation Uncertainty: Bayesian Learning in Unstable Settings

Recently, evidence has emerged that humans approach learning using Bayesian updating rather than (model-free) reinforcement algorithms in a six-arm restless bandit problem. Here, we investigate what this implies for human appreciation of uncertainty. In our task, a Bayesian learner distinguishes three equally salient levels of uncertainty. First, the Bayesian perceives irreducible uncertainty or risk: even knowing the payoff probabilities of a given arm, the outcome remains uncertain. Second, there is (parameter) estimation uncertainty or ambiguity: payoff probabilities are unknown and need to be estimated. Third, the outcome probabilities of the arms change: the sudden jumps are referred to as unexpected uncertainty. We document how the three levels of uncertainty evolved during the course of our experiment and how it affected the learning rate. We then zoom in on estimation uncertainty, which has been suggested to be a driving force in exploration, in spite of evidence of widespread aversion to ambiguity. Our data corroborate the latter. We discuss neural evidence that foreshadowed the ability of humans to distinguish between the three levels of uncertainty. Finally, we investigate the boundaries of human capacity to implement Bayesian learning. We repeat the experiment with different instructions, reflecting varying levels of structural uncertainty. Under this fourth notion of uncertainty, choices were no better explained by Bayesian updating than by (model-free) reinforcement learning. Exit questionnaires revealed that participants remained unaware of the presence of unexpected uncertainty and failed to acquire the right model with which to implement Bayesian updating

Neurobiological studies of risk assessment: A comparison of expected utility and mean-variance approaches

When modeling valuation under uncertainty, economists generally prefer expected utility because it has an axiomatic foundation, meaning that the resulting choices will satisfy a number of rationality requirements. In expected utility theory, values are computed by multiplying probabilities of each possible state of nature by the payoff in that state and summing the results. The drawback of this approach is that all state probabilities need to be dealt with separately, which becomes extremely cumbersome when it comes to learning. Finance academics and professionals, however, prefer to value risky prospects in terms of a trade-off between expected reward and risk, where the latter is usually measured in terms of reward variance. This mean-variance approach is fast and simple and greatly facilitates learning, but it impedes assigning values to new gambles on the basis of those of known ones. To date, it is unclear whether the human brain computes values in accordance with expected utility theory or with mean-variance analysis. In this article, we discuss the theoretical and empirical arguments that favor one or the other theory. We also propose a new experimental paradigm that could determine whether the human brain follows the expected utility or the mean-variance approach. Behavioral results of implementation of the paradigm are discussed

Decision in space

Human navigation is generally believed to rely on two types of strategy adoption, route- based and map-based strategies. Both types of navigation require making spatial decisions along the traversed way. Nevertheless, formal computational and neural links between navigational strategies and mechanisms of value based decision making have so far been underexplored in humans. Here, we employed functional magnetic resonance imaging (fMRI) while subjects located different target objects in a virtual environment. We then modelled their paths using reinforcement learning (RL) algorithms, which successfully explain decision behaviour and its neural correlates. Our results show that subjects used a mixture of route and map-based navigation, and their paths could be well explained by the model-free and model-based RL algorithms. Furthermore, the value signals of model-free choices during route-based navigation modulated the BOLD signals in the ventro-medial prefrontal cortex (vmPFC). On the contrary, the BOLD signals in parahippocampal and medial temporal lobe (MTL) regions pertained to model- based value signals during map-based navigation. Our findings suggest that the brain might share computational mechanisms and neural substrates for navigation and value- based decisions, such that model-free choice guides route-based navigation and model- based choice directs map-based navigation. These findings open new avenues for computational modelling of wayfinding by directing attention to value-based decision, differing from common direction and distances approaches. The ability to find one’s way in a complex environment is crucial to everyday functioning. This navigational ability relies on the integrity of several cognitive functions and different strategies, route and map-based navigation, that individuals may adopt while navigating in the environment. As the integrity of these cognitive functions often decline with age, navigational abilities show marked changes in both normal aging and dementia. Combining a wayfinding task in a virtual reality (VR) environment and modeling technique based on reinforcement learning (RL) algorithms, we investigated the effects of cognitive aging on the selection and adoption of navigation strategies in human. The older participants performed the wayfinding task while undergoing functional Magnetic Resonance Imaging (fMRI), and the younger participants performed the same task outside the MRI machine. Compared with younger participants, older participants traversed a longer distance. They also exhibited a higher tendency to repeat previously established routes to locate the target objects. Despite these differences, the traversed paths in both groups could be well explained by the model-free and model-based RL algorithms. Furthermore, neuroimaging results from the older participants show that BOLD signal in the ventromedial prefrontal cortex (vmPFC) pertained to model-free value signals. This result provide evidence on the utility of the RL algorithms to explain how the aging brain computationally prefer to rely more on the route-based navigation

A dynamic code for economic object valuation in prefrontal cortex neurons.

Neuronal reward valuations provide the physiological basis for economic behaviour. Yet, how such valuations are converted to economic decisions remains unclear. Here we show that the dorsolateral prefrontal cortex (DLPFC) implements a flexible value code based on object-specific valuations by single neurons. As monkeys perform a reward-based foraging task, individual DLPFC neurons signal the value of specific choice objects derived from recent experience. These neuronal object values satisfy principles of competitive choice mechanisms, track performance fluctuations and follow predictions of a classical behavioural model (Herrnstein's matching law). Individual neurons dynamically encode both, the updating of object values from recently experienced rewards, and their subsequent conversion to object choices during decision-making. Decoding from unselected populations enables a read-out of motivational and decision variables not emphasized by individual neurons. These findings suggest a dynamic single-neuron and population value code in DLPFC that advances from reward experiences to economic object values and future choices.Wellcome Trust, Behavioural and Clinical Neuroscience Institute (BCNI) Cambridg

Appetitive and Aversive Goal Values Are Encoded in the Medial Orbitofrontal Cortex at the Time of Decision Making

An essential feature of choice is the assignment of goal values (GVs) to the different options under consideration at the time of decision making. This computation is done when choosing among appetitive and aversive items. Several groups have studied the location of GV computations for appetitive stimuli, but the problem of valuation in aversive contexts at the time of decision making has been ignored. Thus, although dissociations between appetitive and aversive components of value signals have been shown in other domains such as anticipatory and outcome values, it is not known whether appetitive and aversive GVs are computed in similar brain regions or in separate ones. We investigated this question using two different functional magnetic resonance imaging studies while human subjects placed real bids in an economic auction for the right to eat/avoid eating liked/disliked foods. We found that activity in a common area of the medial orbitofrontal cortex and the dorsolateral prefrontal cortex correlated with both appetitive and aversive GVs. These findings suggest that these regions might form part of a common network

Cooperative Success Under Shared Cognitive States and Valuations

A mental model of the another person’s state of mind including their thoughts, feelings, and beliefs, otherwise known as Theory of Mind (ToM), can be created to better predict their behavior and optimize our own decisions. These representations can be explicitly modeled during both the development and presence of stable cooperation via communication outcomes, allowing us to understand the sophistication or depth of mental coordination, involved in an individual’s social perception and reasoning. Almost all current scientific studies of ToM take a spectatorial approach, relying on observation followed by evaluation (e.g., the Sally-Anne Task). However given evidence that social cognition fundamentally shifts during valuationally significant social encounters with others, this study adopts a second-person approach. Each participant’s actions under dynamic uncertainty influence the joint reward probabilities of both, favoring cooperation and coordination. Only Teachers have knowledge of the correct action-reward contingencies, while Learners must ascertain the Teacher’s directive and correctly adjust their actions to obtain the optimal reward. The complexity of cooperative behaviors cannot be captured with simple reinforcement learning models, however a similarity in valuation exists, probing further investigation

Contributions of the Medial Prefrontal Cortex to Social Influence in Economic Decision-Making

Economic decisions are guided by highly subjective reward valuations (SVs). Often these SVs are overridden when individuals conform to social norms. Yet, the neural mechanisms that underpin the distinct processing of such normative reward valuations (NV) are poorly understood. The dorsomedial and ventromedial portions of the prefrontal cortex (dmPFC/vmPFC) are putatively key regions for processing social and economic information respectively. However, the contribution of these regions to economic decisions guided by social norms is unclear. Using fMRI and computational modelling we examine the neural mechanisms underlying the processing of SVs and NVs. Subjects (n = 15) indicated either their own economic preferences or made similar choices based on a social norm - learnt during a training session. We found that that the vmPFC and dmPFC make dissociable contributions to the processing of SV and NV. Regions of the dmPFC processed only the value of rewards when making normative choices. In contrast, we identify a novel mechanism in the vmPFC for the coding of value. This region signalled both subjective and normative valuations, but activity was scaled positively for SV and negatively for NV. These results highlight some of the key mechanisms that underpin conformity and social influence in economic decision-making

The Methodologies of Neuroeconomics

We critically review the methodological practices of two research programs which are jointly called 'neuroeconomics'. We defend the first of these, termed 'neurocellular economics' (NE) by Ross (2008), from an attack on its relevance by Gul and Pesendorfer (2008) (GP). This attack arbitrarily singles out some but not all processing variables as unimportant to economics, is insensitive to the realities of empirical theory testing, and ignores the central importance to economics of 'ecological rationality' (Smith 2007). GP ironically share this last attitude with advocates of 'behavioral economics in the scanner' (BES), the other, and better known, branch of neuroeconomics. We consider grounds for skepticism about the accomplishments of this research program to date, based on its methodological individualism, its ad hoc econometrics, its tolerance for invalid reverse inference, and its inattention to the difficulties involved in extracting temporally lagged data if people's anticipation of reward causes pre-emptive blood flow.