A Behavioral and Neural Evaluation of Prospective Decision-Making under Risk

Making the best choice when faced with a chain of decisions requires a person to judge both anticipated outcomes and future actions. Although economic decision-making models account for both risk and reward in single-choice contexts, there is a dearth of similar knowledge about sequential choice. Classical utility-based models assume that decision-makers select and follow an optimal predetermined strategy, regardless of the particular order in which options are presented. An alternative model involves continuously reevaluating decision utilities, without prescribing a specific future set of choices. Here, using behavioral and functional magnetic resonance imaging (fMRI) data, we studied human subjects in a sequential choice task and use these data to compare alternative decision models of valuation and strategy selection. We provide evidence that subjects adopt a model of reevaluating decision utilities, in which available strategies are continuously updated and combined in assessing action values. We validate this model by using simultaneously acquired fMRI data to show that sequential choice evokes a pattern of neural response consistent with a tracking of anticipated distribution of future reward, as expected in such a model. Thus, brain activity evoked at each decision point reflects the expected mean, variance, and skewness of possible payoffs, consistent with the idea that sequential choice evokes a prospective evaluation of both available strategies and possible outcomes

Die Rolle der Zielnähe und der investierten Anstrengung für den erwarteten Wert einer Handlung

In human neuroscientific research, there has been an increasing interest in how the brain computes the value of an anticipated outcome. However, evidence is still missing about which valuation related brain regions are modulated by the proximity to an expected goal and the previously invested effort to reach a goal. The aim of this dissertation is to investigate the effects of goal proximity and invested effort on valuation related regions in the human brain. We addressed this question in two fMRI studies by integrating a commonly used reward anticipation task in differential versions of a Multitrial Reward Schedule Paradigm. In both experiments, subjects had to perform consecutive reward anticipation tasks under two different reward contingencies: in the delayed condition, participants received a monetary reward only after successful completion of multiple consecutive trials. In the immediate condition, money was earned after every successful trial. In the first study, we could demonstrate that the rostral cingulate zone of the posterior medial frontal cortex signals action value contingent to goal proximity, thereby replicating neurophysiological findings about goal proximity signals in a homologous region in non-human primates. The findings of the second study imply that brain regions associated with general cognitive control processes are modulated by previous effort investment. Furthermore, we found the posterior lateral prefrontal cortex and the orbitofrontal cortex to be involved in coding for the effort-based context of a situation. In sum, these results extend the role of the human rostral cingulate zone in outcome evaluation to the continuous updating of action values over a course of action steps based on the proximity to the expected reward. Furthermore, we tentatively suggest that previous effort investment invokes processes under the control of the executive system, and that posterior lateral prefrontal cortex and the orbitofrontal cortex are involved in an effort-based context representation that can be used for outcome evaluation that is dependent on the characteristics of the current situation.Derzeit besteht im Bereich der Neurowissenschaften ein großes Interesse daran aufzuklären, auf welche Weise verschiedene Variablen die Wertigkeit eines erwarteten Handlungsziels beeinflussen bzw. welche Hirnregionen an der Repräsentation der Wertigkeit eines Handlungsziels beteiligt sind. Die meisten Untersuchungen beziehen sich dabei auf Einflussgrößen wie die erwartete Belohnungshöhe, die Wahrscheinlichkeit, mit der ein bestimmtes Ereignis eintritt, oder die Dauer bis zum Erhalt einer Belohnung. Bisher liegen jedoch kaum Untersuchungen vor bezüglich zweier anderer Variablen, die ebenfalls den erwarteten Wert eines Handlungsergebnisses beeinflussen. Das sind (a) die Nähe zu dem erwarteten Ziel und (b) die bisher investierte Anstrengung, um ein Ziel zu erreichen. Das Ziel der vorliegenden Dissertation ist zu untersuchen, wie die Nähe zum Ziel und die bisher investierte Anstrengung Gehirnregionen beeinflussen, die mit der Repräsentation von Wertigkeit im Zusammenhang stehen. Dazu führten wir zwei fMRT-Studien durch, in denen wir eine klassische Belohnungs-Antizipationsaufgabe in unterschiedliche Versionen eines „Multitrial Reward Schedule“ Paradigmas integriert haben. Das bedeutet, dass die Probanden Belohnungs-Antizipationsaufgaben unter zwei unterschiedlichen Belohnungskontingenzen bearbeiteten: In der verzögerten Bedingung erhielten die Probanden einen Geldbetrag nach der erfolgreichen Bearbeitung von mehreren aufeinanderfolgenden Aufgaben, in der direkten Bedingung dagegen nach jeder korrekt ausgeführten Aufgabe. In der ersten Studie konnte eine sukzessiv ansteigende Aktivität in Abhängigkeit zur Zielnähe in der rostralen cingulären Zone identifiziert werden. Das deutet darauf hin, dass dieses Areal den Wert einer Handlung in Abhängigkeit zur Nähe zum Ziel kodiert. Die Ergebnisse der zweiten Studie zeigten, dass die bisher investierte Anstrengung kortikale Regionen moduliert, die klassischerweise mit kognitiven Kontrollfunktionen in Zusammenhang gebracht werden. Außerdem repräsentierten der posteriore laterale präfrontale Cortex und der orbitofrontale Cortex den motivationalen Kontext eines Trials anhand des Risikos des Verlustes von bisher investierter Anstrengung. Insgesamt weisen diese Befunde darauf hin, dass die rostrale cinguläre Zone eine entscheidende Rolle spielt für die Kontrolle sequenzieller Handlungsstufen, die auf eine verzögerte Belohnung ausgerichtet sind. Diese Kontrollfunktion scheint auf der kontinuierlichen Aktualisierung des Wertes einer Handlungsstufe zu basieren, der von der aktuellen Zielnähe bestimmt wird. Die Befunde der zweiten Studie lassen darauf schließen, dass sich die bisher investierte Anstrengung zur Erreichung eines Handlungsziels auf die Bereitstellung von allgemeinen kognitiven Ressourcen auswirkt. Das Risiko des Verlustes von bisher investierter Anstrengung kann außerdem ein kontextuelles Merkmal der Situation darstellen, das als Bezugsrahmen für die Evaluation des erwarteten Wertes dienen kann

Nataly Aguirre - Stress, Risk, and Reward in Financial Decision-Making: The Roles of Probability and Magnitude

Considerable research suggests acute stress influences decision-making. There has, however, been a lack of research examining the possibility that separable components of the stress response may influence decision-making differently: the sympatho-adrenomedullary (SAM) and hypothalamic-pituitary-adrenal (HPA) axes. In the current pilot study, participants engaged in a gambling task where they made choices between decisions of varied probability and magnitude for potential gains of money after being exposed to acute stress (via a variant of the cold pressor task). Further, the timing of the stressor was varied to allow examination of SAM and HPA effects separately. Cortisol and skin conductance were measured. Given the task was in the gain frame only, in support of past research on framing results indicated that individuals made significantly more conservative or risk-averse choices in the gambling. Further, risk-taking scaled to the expected value of a decision. Males made more risk-seeking choices as compared to females. Divergent from the original hypothesis, however, stress of neither type had an effect on individuals’ risk-taking overall, nor as a function of probability or magnitude. This suggests that decisions framed as potential gains may not be influenced by stress as readily as decisions framed as potential losses, and that stress may not alter how people perceive the probability or magnitude associated with a decision. Methodological flaws highlighted by the pilot study which may have contributed to the lack of a stress effect will also be discussed.https://epublications.marquette.edu/mcnair_2013/1000/thumbnail.jp

Midbrain Dopamine Neurons Signal Belief in Choice Accuracy during a Perceptual Decision

Central to the organization of behavior is the ability to predict the values of outcomes to guide choices. The accuracy of such predictions is honed by a teaching signal that indicates how incorrect a prediction was (“reward prediction error,” RPE). In several reinforcement learning contexts, such as Pavlovian conditioning and decisions guided by reward history, this RPE signal is provided by midbrain dopamine neurons. In many situations, however, the stimuli predictive of outcomes are perceptually ambiguous. Perceptual uncertainty is known to influence choices, but it has been unclear whether or how dopamine neurons factor it into their teaching signal. To cope with uncertainty, we extended a reinforcement learning model with a belief state about the perceptually ambiguous stimulus; this model generates an estimate of the probability of choice correctness, termed decision confidence. We show that dopamine responses in monkeys performing a perceptually ambiguous decision task comply with the model’s predictions. Consequently, dopamine responses did not simply reflect a stimulus’ average expected reward value but were predictive of the trial-to-trial fluctuations in perceptual accuracy. These confidence-dependent dopamine responses emerged prior to monkeys’ choice initiation, raising the possibility that dopamine impacts impending decisions, in addition to encoding a post-decision teaching signal. Finally, by manipulating reward size, we found that dopamine neurons reflect both the upcoming reward size and the confidence in achieving it. Together, our results show that dopamine responses convey teaching signals that are also appropriate for perceptual decisions

The Effects of Discrete Emotions on Risky Decision Making

Contrary to the dominant view that generally equates feelings with poor thinking, converging evidence indicates that decisions – including those involving risk – are influenced by affective experiences. Research, however, is limited to studies on undifferentiated, global positive versus negative mood states; less is known about the influence of discrete emotions. The purpose of this research was to extend the affect-cognition literature by (a) examining the effects of discrete emotions varying along the dimensions of valence and arousal, and (b) identifying the systematic ways that discrete emotions underlie risky decision making. We used a set of emotion-laden IAPS images to elicit and compare the impact of incidental emotions on risky decision making. One hundred and twenty-two undergraduate students were randomly assigned to one of the four affective conditions: excitement, contentment, fear, and sadness. Following the emotion induction procedure, participants completed the Choice Dilemmas Questionnaire (CDQ) to assess their risk-taking propensity. Results indicated an interaction effect between valence and arousal for positive emotions, such that excited participants were significantly more risky in their decision making compared to contented participants. The discussion focuses on the theoretical and practical health implications of these findings. We recommend that future research capitalize on the insights gained from emotion research and use it favorably to improve decision making under risk

Direct and indirect effects of mood on risk decision making in safety-critical workers

The study aimed to examine the direct influence of specific moods (fatigue, anxiety, happiness) on risk in safety-critical decision making. It further aimed to explore indirect effects, specifically, the potential mediating effects of information processing assessed using a goodness-of-simulation task. Trait fatigue and anxiety were associated with an increase in risk taking on the Safety-Critical Personal Risk Inventory (S-CPRI), however the effect of fatigue was partialled out by anxiety. Trait happiness, in contrast was related to less risky decision making. Findings concerning the ability to simulate suggest that better simulators made less risky decisions. Anxious workers were generally less able to simulate. It is suggested that in this safety-critical environment happiness had a direct effect on risk decision making while the effect of trait anxiety was mediated by goodness-of-simulation

Outcome contingency selectively affects the neural coding of outcomes but not of tasks

Value-based decision-making is ubiquitous in every-day life, and critically depends on the contingency between choices and their outcomes. Only if outcomes are contingent on our choices can we make meaningful value-based decisions. Here, we investigate the effect of outcome contingency on the neural coding of rewards and tasks. Participants performed a reversal-learning paradigm in which reward outcomes were contingent on trial-by-trial choices, and performed a ‘free choice’ paradigm in which rewards were random and not contingent on choices. We hypothesized that contingent outcomes enhance the neural coding of rewards and tasks, which was tested using multivariate pattern analysis of fMRI data. Reward outcomes were encoded in a large network including the striatum, dmPFC and parietal cortex, and these representations were indeed amplified for contingent rewards. Tasks were encoded in the dmPFC at the time of decision-making, and in parietal cortex in a subsequent maintenance phase. We found no evidence for contingency-dependent modulations of task signals, demonstrating highly similar coding across contingency conditions. Our findings suggest selective effects of contingency on reward coding only, and further highlight the role of dmPFC and parietal cortex in value-based decision-making, as these were the only regions strongly involved in both reward and task coding

History-based action selection bias in posterior parietal cortex.

Making decisions based on choice-outcome history is a crucial, adaptive ability in life. However, the neural circuit mechanisms underlying history-dependent decision-making are poorly understood. In particular, history-related signals have been found in many brain areas during various decision-making tasks, but the causal involvement of these signals in guiding behavior is unclear. Here we addressed this issue utilizing behavioral modeling, two-photon calcium imaging, and optogenetic inactivation in mice. We report that a subset of neurons in the posterior parietal cortex (PPC) closely reflect the choice-outcome history and history-dependent decision biases, and PPC inactivation diminishes the history dependency of choice. Specifically, many PPC neurons show history- and bias-tuning during the inter-trial intervals (ITI), and history dependency of choice is affected by PPC inactivation during ITI and not during trial. These results indicate that PPC is a critical region mediating the subjective use of history in biasing action selection

Prospect relativity: how choice options influence decision under risk.

In many theories of decision under risk (e.g., expected utility theory, rank-dependent utility theory, and prospect theory), the utility of a prospect is independent of other options in the choice set. The experiments presented here show a large effect of the available options, suggesting instead that prospects are valued relative to one another. The judged certainty equivalent for a prospect is strongly influenced by the options available. Similarly, the selection of a preferred prospect is strongly influenced by the prospects available. Alternative theories of decision under risk (e.g., the stochastic difference model, multialternative decision field theory, and range frequency theory), where prospects are valued relative to one another, can provide an account of these context effects

The influence of the noradrenergic system on optimal control of neural plasticity

Decision making under uncertainty is challenging for any autonomous agent. The challenge increases when the environment’s stochastic properties change over time, i.e., when the environment is volatile. In order to efficiently adapt to volatile environments, agents must primarily rely on recent outcomes to quickly change their decision strategies; in other words, they need to increase their knowledge plasticity. On the contrary, in stable environments, knowledge stability must be preferred to preserve useful information against noise. Here we propose that in mammalian brain, the locus coeruleus (LC) is one of the nuclei involved in volatility estimation and in the subsequent control of neural plasticity. During a reinforcement learning task, LC activation, measured by means of pupil diameter, coded both for environmental volatility and learning rate. We hypothesize that LC could be responsible, through norepinephrinic modulation, for adaptations to optimize decision making in volatile environments. We also suggest a computational model on the interaction between the anterior cingulate cortex (ACC) and LC for volatility estimation