Genetic Dissection of Neural Circuits Underlying Value Based Decision Making

Decision making is the fundamental process that we utilize to accomplish objectives in everyday lives. To understand the neural substrates of this process, we developed a behavioral task for mice that required repetition of the processes of action initiation, action selection, and learning. The task is a two-option choice task with stochastic reward delivery and reversals. To map brain areas involved in this type of value-based learning, we inactivated neuronal activity in the prefrontal cortex (PFC) and the nucleus accumbens (NAc) while mice performed the task. Inactivation of the NAc resulted in altered action initiation and learning but had subtle effects on choice. To dissect underlying neural circuitry, specific cell types and inputs of the NAc were inactivated. Inactivation of two dominant cell types in the NAc, direct and indirect pathway medium spiny neurons (MSNs), showed partially overlapping but distinct behavioral effects. Inactivation of direct pathway MSNs showed the stronger effect on learning while inactivation of indirect pathway MSNs also showed the effect. In contrast, only inactivation of indirect pathway MSNs affected behavioral measures of action initiation. The contribution of specific inputs to the NAc, dopaminergic and glutamatergic inputs, were also studied. While both experiments affected behavioral measures of initiation of action, only the inactivation of dopaminergic inputs affected learning. The effect on learning was specific to trials after reward omissions, and the effect was more prominent in trials which animals spent less time to initiate. These results provided new insights into the function of the NAc in processing information about reward values. In contrast, inactivation of two subregions in the PFC, the anterior cingulate cortex (ACC) and the orbitofrontal cortex OFC, affected action initiation and action selection. The action initiation was affected by inactivation of both areas, but OFC inactivation affected more behavioral measures. In contrast, action selection was affected more prominently in ACC inactivation. These differential effects on action initiation and action selection suggested the functional distinction between these two areas. In this study, we have developed a behavioral assay that allowed us to dissect different aspects of cognitive functions for decisions in mice and revealed roles of distinct circuit elements in the NAc and the PFC. Utilizing temporally precise inactivation, we found that the same circuit element was used for different cognitive processes depending on the timing. Although this type of behavioral task has been used extensively in rats and primates to understand decision making, identification of cell types and circuits required for these behaviors has been difficult in these species due to the lack of the powerful genetic methodologies. The approach we have demonstrated here is important because it enables genetic dissection of complex behaviors in mice, allowing studies of circuit properties that are executed by specific cell types in the cerebral cortex and basal ganglia. Since the approach taken in this study can be expanded to other neural circuits and behavioral paradigms, this and future studies will reveal the neural basis of decision making and, perhaps, lead to new approaches to treatments for maladaptive behaviors

A Symbiotic Brain-Machine Interface through Value-Based Decision Making

BACKGROUND: In the development of Brain Machine Interfaces (BMIs), there is a great need to enable users to interact with changing environments during the activities of daily life. It is expected that the number and scope of the learning tasks encountered during interaction with the environment as well as the pattern of brain activity will vary over time. These conditions, in addition to neural reorganization, pose a challenge to decoding neural commands for BMIs. We have developed a new BMI framework in which a computational agent symbiotically decoded users' intended actions by utilizing both motor commands and goal information directly from the brain through a continuous Perception-Action-Reward Cycle (PARC). METHODOLOGY: The control architecture designed was based on Actor-Critic learning, which is a PARC-based reinforcement learning method. Our neurophysiology studies in rat models suggested that Nucleus Accumbens (NAcc) contained a rich representation of goal information in terms of predicting the probability of earning reward and it could be translated into an evaluative feedback for adaptation of the decoder with high precision. Simulated neural control experiments showed that the system was able to maintain high performance in decoding neural motor commands during novel tasks or in the presence of reorganization in the neural input. We then implanted a dual micro-wire array in the primary motor cortex (M1) and the NAcc of rat brain and implemented a full closed-loop system in which robot actions were decoded from the single unit activity in M1 based on an evaluative feedback that was estimated from NAcc. CONCLUSIONS: Our results suggest that adapting the BMI decoder with an evaluative feedback that is directly extracted from the brain is a possible solution to the problem of operating BMIs in changing environments with dynamic neural signals. During closed-loop control, the agent was able to solve a reaching task by capturing the action and reward interdependency in the brain

Recovery from addiction: Behavioral economics and value-based decision making.

Behavioral economics provides a general framework to explain the shift in behavioral allocation from substance use to substance-free activities that characterizes recovery from addiction, but it does not attempt to explain the internal processes that prompt those behavioral changes. In this article we outline a novel analysis of addiction recovery based on computational work on value-based decision making (VBDM), which can explain how people with addiction are able to overcome the reinforcement pathologies and decision-making vulnerabilities that characterize the disorder. The central tenet of this account is that shifts in molar reinforcer preferences over time from substance use to substance-free activities can be attributed to changes in evidence accumulation rates and response thresholds in the context of choices involving substance use and substance-free alternatives. We discuss how this account can be reconciled with the established mechanisms of action of psychosocial interventions for addiction and demonstrate how it has the potential to empirically address longstanding debates regarding the nature of impairments to self-control in addiction. We also highlight conceptual and methodological issues that require careful consideration in translating VBDM to addiction and recovery

Is attentional discounting in value-based decision making magnitude sensitive?

Choices in value-based decision making are affected by the magnitude of the alternatives (i.e. the summed values of the options). Magnitude sensitivity has been instrumental in discriminating between computational models of choice. Smith and Krajbich [(2019a). Gaze amplifies value in decision making. Psychological Science, 30(1), 116–128. https://doi.org/10.1177/0956797618810521] have shown that the attentional drift-diffusion model (aDDM) can account for magnitude sensitivity. This is because the discount parameter on the value of the nonfixated alternative ensures faster choices for high-magnitude alternatives, even in the case of high-magnitude equal alternatives compared to low-magnitude equal alternatives. Their result highlights the importance of visual fixations as a mechanism for magnitude sensitivity. This rationale relies on the untested assumption that the discount parameter is constant across magnitude levels. However, the discount parameter could vary as a function of the magnitude of the alternatives in unpredicted ways; this would suggest that the ability of the aDDM to account for magnitude sensitivity has been misinterpreted by previous research. Here, we reanalyse previous datasets and we directly test whether attentional discounting scales with the magnitude of the alternatives. Our analyses show that attentional discounting does not vary with magnitude. This result further strengthens the aDDM and the role that visual fixations could play as an explanation of magnitude sensitivity

No differences in value-based decision-making due to use of oral contraceptives

Fluctuating ovarian hormones have been shown to affect decision-making processes in women. While emerging evidence suggests effects of endogenous ovarian hormones such as estradiol and progesterone on value-based decision-making in women, the impact of exogenous synthetic hormones, as in most oral contraceptives, is not clear. In a between-subjects design, we assessed measures of value-based decision-making in three groups of women aged 18 to 29 years, during (1) active oral contraceptive intake (N = 22), (2) the early follicular phase of the natural menstrual cycle (N = 20), and (3) the periovulatory phase of the natural menstrual cycle (N = 20). Estradiol, progesterone, testosterone, and sex-hormone binding globulin levels were assessed in all groups via blood samples. We used a test battery which measured different facets of value-based decision-making: delay discounting, risk-aversion, risk-seeking, and loss aversion. While hormonal levels did show the expected patterns for the three groups, there were no differences in value-based decision-making parameters. Consequently, Bayes factors showed conclusive evidence in support of the null hypothesis. We conclude that women on oral contraceptives show no differences in value-based decision-making compared to the early follicular and periovulatory natural menstrual cycle phases. Copyright © 2022 Lewis, Kimmig, Kroemer, Pooseh, Smolka, Sacher and Derntl

Neural Evidence for Adaptive Strategy Selection in Value-Based Decision-Making

In everyday life, humans often encounter complex environments in which multiple sources of information can influence their decisions. We propose that in such situations, people select and apply different strategies representing different cognitive models of the decision problem. Learning advances by evaluating the success of using a strategy and eventually by switching between strategies. To test our strategy selection model, we investigated how humans solve a dynamic learning task with complex auditory and visual information, and assessed the underlying neural mechanisms with functional magnetic resonance imaging. Using the model, we were able to capture participants' choices and to successfully attribute expected values and reward prediction errors to activations in the dopaminoceptive system (e.g., ventral striatum [VS]) as well as decision conflict to signals in the anterior cingulate cortex. The model outperformed an alternative approach that did not update decision strategies, but the relevance of information itself. Activation of sensory areas depended on whether the selected strategy made use of the respective source of information. Selection of a strategy also determined how value-related information influenced effective connectivity between sensory systems and the VS. Our results suggest that humans can structure their search for and use of relevant information by adaptively selecting between decision strategie

A framework for studying the neurobiology of value-based decision making

Neuroeconomics is the study of the neurobiological and computational basis of value-based decision making. Its goal is to provide a biologically based account of human behaviour that can be applied in both the natural and the social sciences. This Review proposes a framework to investigate different aspects of the neurobiology of decision making. The framework allows us to bring together recent findings in the field, highlight some of the most important outstanding problems, define a common lexicon that bridges the different disciplines that inform neuroeconomics, and point the way to future applications

The Neural Representation of Unexpected Uncertainty during Value-Based Decision Making

Uncertainty is an inherent property of the environment and a central feature of models of decision-making and learning. Theoretical propositions suggest that one form, unexpected uncertainty, may be used to rapidly adapt to changes in the environment, while being influenced by two other forms: risk and estimation uncertainty. While previous studies have reported neural representations of estimation uncertainty and risk, relatively little is known about unexpected uncertainty. Here, participants performed a decision-making task while undergoing functional magnetic resonance imaging (fMRI), which, in combination with a Bayesian model-based analysis, enabled us to separately examine each form of uncertainty examined. We found representations of unexpected uncertainty in multiple cortical areas, as well as the noradrenergic brainstem nucleus locus coeruleus. Other unique cortical regions were found to encode risk, estimation uncertainty, and learning rate. Collectively, these findings support theoretical models in which several formally separable uncertainty computations determine the speed of learning

Individual Differences In Value-Based Decision-Making: Learning And Time Preference

Human decisions are strongly influenced by past experience or by the subjective values attributed to available choice options. Although decision processes show some common trends across individuals, they also vary considerably between individuals. The research presented in this dissertation focuses on two domains of decision-making, related to learning and time preference, and examines factors that explain decision-making differences between individuals. First, we focus on a form of reinforcement learning in a dynamic environment. Across three experiments, we investigated whether individual differences in learning were associated with differences in cognitive abilities, personality, and age. Participants made sequential predictions about an on-screen location in a video game. Consistent with previous work, participants showed high variability in their ability to implement normative strategies related to surprise and uncertainty. We found that higher cognitive ability, but not personality, was associated with stronger reliance on the normative factors that should govern learning. Furthermore, learning in older adults (age 60+) was less influenced by uncertainty, but also less influenced by reward, a non-normative factor that has substantial effects on learning across the lifespan. Second, we focus on delay discounting, the tendency to prefer smaller rewards delivered soon over larger rewards delivered after a delay. Delay discounting has been used as a behavioral measure of impulsivity and is associated with many undesirable real-life outcomes. Specifically, we examined how neuroanatomy is associated with individual differences in delay discounting in a large adolescent sample. Using a novel multivariate method, we identified networks where cortical thickness varied consistently across individuals and brain regions. Cortical thickness in several of these networks, including regions such as ventromedial prefrontal cortex, orbitofrontal cortex, and temporal pole, was negatively associated with delay discounting. Furthermore, this brain data predicted differences beyond those typically accounted for by other cognitive variables related to delay discounting. These results suggest that cortical thickness may be a useful brain phenotype of delay discounting and carry unique information about impulsivity. Collectively, this research furthers our understanding of how cognitive abilities, brain structure and healthy aging relate to individual differences in value-based decision-making