Coding of Reward Probability and Risk by Single Neurons in Animals

Probability and risk are important factors for value-based decision making and optimal foraging. In order to survive in an unpredictable world, organisms must be able to assess the probability and risk attached to future events and use this information to generate adaptive behavior. Recent studies in non-human primates and rats have shown that both probability and risk are processed in a distributed fashion throughout the brain at the level of single neurons. Reward probability has mainly been shown to be coded by phasic increases and decreases in firing rates in neurons in the basal ganglia, midbrain, parietal, and frontal cortex. Reward variance is represented in orbitofrontal and posterior cingulate cortex and through a sustained response of dopaminergic midbrain neurons

Lateral orbitofrontal cortex promotes trial-by-trial learning of risky, but not spatial, biases

Individual choices are not made in isolation but are embedded in a series of past experiences, decisions, and outcomes. The effects of past experiences on choices, often called sequential biases, are ubiquitous in perceptual and value-based decision-making, but their neural substrates are unclear. We trained rats to choose between cued guaranteed and probabilistic rewards in a task in which outcomes on each trial were independent. Behavioral variability often reflected sequential effects, including increased willingness to take risks following risky wins, and spatial ‘win-stay/lose-shift’ biases. Recordings from lateral orbitofrontal cortex (lOFC) revealed encoding of reward history and receipt, and optogenetic inhibition of lOFC eliminated rats’ increased preference for risk following risky wins, but spared other sequential effects. Our data show that different sequential biases are neurally dissociable, and the lOFC’s role in adaptive behavior promotes learning of more abstract biases (here, biases for the risky option), but not spatial ones

The influence of contextual reward statistics on risk preference

Decision theories mandate that organisms should adjust their behaviour in the light of the contextual reward statistics. We tested this notion using a gambling choice task involving distinct contexts with different reward distributions. The best fitting model of subjects' behaviour indicated that the subjective values of options depended on several factors, including a baseline gambling propensity, a gambling preference dependent on reward amount, and a contextual reward adaptation factor. Combining this behavioural model with simultaneous functional magnetic resonance imaging we probed neural responses in three key regions linked to reward and value, namely ventral tegmental area/substantia nigra (VTA/SN), ventromedial prefrontal cortex (vmPFC) and ventral striatum (VST). We show that activity in the VTA/SN reflected contextual reward statistics to the extent that context affected behaviour, activity in the vmPFC represented a value difference between chosen and unchosen options while VST responses reflected a non-linear mapping between the actual objective rewards and their subjective value. The findings highlight a multifaceted basis for choice behaviour with distinct mappings between components of this behaviour and value sensitive brain regions

Contribution of the Orbitofrontal Cortex to Delayed Punishment Discounting

The ability to accumulate rewards while minimizing negative consequences is a valuable survival skill. Importantly, many psychiatric diseases such as substance use disorder (SUD; Bechara, 2005; Gowin et al., 2013), attention deficit hyperactivity (Magnus et al., 2021), anxiety (Hartley and Phelps, 2012), major depressive, bipolar, and schizophrenia disorders (Whitton et al., 2015) involve impaired decision-making that can lead to detrimental outcomes. One factor that causes maladaptive decision-making is insensitivity to negative consequences, especially those that occur later in time (Murphy et al., 2001; Bechara and Dolan, 2002; Field et al., 2019). These studies were among the first to investigate how the orbitofrontal cortex, a brain region implicated in cost/benefit decision-making (Floresco et al., 2008) and reward discounting (Zeeb et al., 2010), contributes to the discounting of delayed punishment. Information gathered from the current work provided the first evidence that inactivation of LOFC reduced choice of delayed punishment compared to saline baselines, and LOFC inhibition occurred prior to different types of safe reward choices compared to immediate punishment. Preliminary optogenetics data also found that pre-choice inhibition reduced delayed punishment choice. In summation, LOFC drives the undervaluation of delayed punishment, and future therapeutic treatments aiming to improve discounting of delayed punishments during decision-making would benefit from selectively suppressing LOFC activity

Contribution of the Orbitofrontal Cortex to Delayed Punishment Discounting

The ability to accumulate rewards while minimizing negative consequences is a valuable survival skill. Importantly, many psychiatric diseases such as substance use disorder (SUD; Bechara, 2005; Gowin et al., 2013), attention deficit hyperactivity (Magnus et al., 2021), anxiety (Hartley and Phelps, 2012), major depressive, bipolar, and schizophrenia disorders (Whitton et al., 2015) involve impaired decision-making that can lead to detrimental outcomes. One factor that causes maladaptive decision-making is insensitivity to negative consequences, especially those that occur later in time (Murphy et al., 2001; Bechara and Dolan, 2002; Field et al., 2019). These studies were among the first to investigate how the orbitofrontal cortex, a brain region implicated in cost/benefit decision-making (Floresco et al., 2008) and reward discounting (Zeeb et al., 2010), contributes to the discounting of delayed punishment. Information gathered from the current work provided the first evidence that inactivation of lateral orbitofrontal cortex (LOFC) reduced choice of delayed punishment compared to saline baselines, and LOFC inhibition occurred prior to different types of safe reward choices compared to immediate punishment. Preliminary optogenetics data also found that pre-choice inhibition reduced delayed punishment choice. In summation, LOFC drives the undervaluation of delayed punishment, and future therapeutic treatments aiming to improve discounting of delayed punishments during decision-making would benefit from selectively suppressing LOFC activity

Neurocomputational Accounts of Choice Variability and Affect during Decision-making

Humans exhibit surprising variability in behaviour, often making different choices under identical conditions. While the outcomes of these choices typically lead to explicit rewards that have been shown to influence subsequent affective states, less well understood is how the brain represents rewards that are intrinsically meaningful to an individual. The first part of this thesis examines the contributions of endogenous fluctuations in brain activity to behaviour. Resting-state studies suggest that ongoing endogenous fluctuations in brain activity can influence low-level perceptual and motor processes but it remains unknown whether such fluctuations also influence high-level cognitive processes including decision making. Using a novel application of real-time functional magnetic resonance imaging, I find that low pre-stimulus brain activity lead to increased occurrences of risky choice. Using computational modeling, I show that greater risk taking is explained by enhanced phasic responses to offers in a decision network. These findings demonstrate that endogenous brain activity provides a physiological basis for variability in complex behaviour. I then examine how the neuroanatomy of the brain in the form of tissue microstructure relates to risk preferences by leveraging on in vivo histology using magnetic resonance imaging. The second part of this thesis investigates how experienced events, such as rewards received following choice, are aggregated into affective states. Despite their relevance to ideas like goal-setting and well-being, little is known about the impact of intrinsic rewards on affective states and their representation in the brain. A reinforcement learning task incorporating a skilled performance component that did not influence payment was developed to examine this. Computational modeling revealed that momentary happiness depended on past extrinsic rewards and also intrinsic rewards related to the experience of successful skilled performance. Individuals for whom intrinsic rewards more strongly influence momentary happiness exhibit stronger ventromedial prefrontal cortex responses for successful skilled performance. These findings show that the ventromedial prefrontal cortex represents the subjective value of intrinsic rewards, and that computational models of mood dynamics provide a tool that can be used to measure implicit values of abstract goods and experiences

A Behavioral and Neural Evaluation of Prospective Decision-Making under Risk

Making the best choice when faced with a chain of decisions requires a person to judge both anticipated outcomes and future actions. Although economic decision-making models account for both risk and reward in single-choice contexts, there is a dearth of similar knowledge about sequential choice. Classical utility-based models assume that decision-makers select and follow an optimal predetermined strategy, regardless of the particular order in which options are presented. An alternative model involves continuously reevaluating decision utilities, without prescribing a specific future set of choices. Here, using behavioral and functional magnetic resonance imaging (fMRI) data, we studied human subjects in a sequential choice task and use these data to compare alternative decision models of valuation and strategy selection. We provide evidence that subjects adopt a model of reevaluating decision utilities, in which available strategies are continuously updated and combined in assessing action values. We validate this model by using simultaneously acquired fMRI data to show that sequential choice evokes a pattern of neural response consistent with a tracking of anticipated distribution of future reward, as expected in such a model. Thus, brain activity evoked at each decision point reflects the expected mean, variance, and skewness of possible payoffs, consistent with the idea that sequential choice evokes a prospective evaluation of both available strategies and possible outcomes

Framing and stakes: A survey study of decisions under uncertainty

Using a survey study of 261 decisions under uncertainty, we explore the factors that explain risk taking behavior and those that predict the importance of a decision. We also examine the relationship between framing and status quo, the similarity between monetary and non-monetary decisions, as well as the similarities and differences among our three subject groups (Undergraduates, MBAs and Executives). We find that framing, domain, and probability of success have a strong influence on the probability of taking risks. Other factors, such as group, importance of a decision, and whether the consequences are monetary or not, do not seem to influence risk attitudes. Our analysis of importance of a decision highlights the frequency with which a decision is taken as a key variable. Our results suggest that the cumulative effects of unimportant and frequent decisions are greater than the cumulative effects of very important and infrequent decisions.Decision making under uncertainty; Framing; Importance and frequency of decisions;

The rat frontal orienting field dynamically encodes value for economic decisions under risk

Frontal and parietal cortex are implicated in economic decision-making, but their causal roles are untested. Here we silenced the frontal orienting field (FOF) and posterior parietal cortex (PPC) while rats chose between a cued lottery and a small stable surebet. PPC inactivations produced minimal short-lived effects. FOF inactivations reliably reduced lottery choices. A mixed-agent model of choice indicated that silencing the FOF caused a change in the curvature of the rats' utility function (U = Vρ). Consistent with this finding, single-neuron and population analyses of neural activity confirmed that the FOF encodes the lottery value on each trial. A dynamical model, which accounts for electrophysiological and silencing results, suggests that the FOF represents the current lottery value to compare against the remembered surebet value. These results demonstrate that the FOF is a critical node in the neural circuit for the dynamic representation of action values for choice under risk

Computational mechanisms of curiosity and goal-directed exploration

Successful behaviour depends on the right balance between maximising reward and soliciting information about the world. Here, we show how different types of information-gain emerge when casting behaviour as surprise minimisation. We present two distinct mechanisms for goal-directed exploration that express separable profiles of active sampling to reduce uncertainty. 'Hidden state' exploration motivates agents to sample unambiguous observations to accurately infer the (hidden) state of the world. Conversely, 'model parameter' exploration, compels agents to sample outcomes associated with high uncertainty, if they are informative for their representation of the task structure. We illustrate the emergence of these types of information-gain, termed active inference and active learning, and show how these forms of exploration induce distinct patterns of 'Bayes-optimal' behaviour. Our findings provide a computational framework for understanding how distinct levels of uncertainty systematically affect the exploration-exploitation trade-off in decision-making