Die Rolle der Zielnähe und der investierten Anstrengung für den erwarteten Wert einer Handlung

In human neuroscientific research, there has been an increasing interest in how the brain computes the value of an anticipated outcome. However, evidence is still missing about which valuation related brain regions are modulated by the proximity to an expected goal and the previously invested effort to reach a goal. The aim of this dissertation is to investigate the effects of goal proximity and invested effort on valuation related regions in the human brain. We addressed this question in two fMRI studies by integrating a commonly used reward anticipation task in differential versions of a Multitrial Reward Schedule Paradigm. In both experiments, subjects had to perform consecutive reward anticipation tasks under two different reward contingencies: in the delayed condition, participants received a monetary reward only after successful completion of multiple consecutive trials. In the immediate condition, money was earned after every successful trial. In the first study, we could demonstrate that the rostral cingulate zone of the posterior medial frontal cortex signals action value contingent to goal proximity, thereby replicating neurophysiological findings about goal proximity signals in a homologous region in non-human primates. The findings of the second study imply that brain regions associated with general cognitive control processes are modulated by previous effort investment. Furthermore, we found the posterior lateral prefrontal cortex and the orbitofrontal cortex to be involved in coding for the effort-based context of a situation. In sum, these results extend the role of the human rostral cingulate zone in outcome evaluation to the continuous updating of action values over a course of action steps based on the proximity to the expected reward. Furthermore, we tentatively suggest that previous effort investment invokes processes under the control of the executive system, and that posterior lateral prefrontal cortex and the orbitofrontal cortex are involved in an effort-based context representation that can be used for outcome evaluation that is dependent on the characteristics of the current situation.Derzeit besteht im Bereich der Neurowissenschaften ein großes Interesse daran aufzuklären, auf welche Weise verschiedene Variablen die Wertigkeit eines erwarteten Handlungsziels beeinflussen bzw. welche Hirnregionen an der Repräsentation der Wertigkeit eines Handlungsziels beteiligt sind. Die meisten Untersuchungen beziehen sich dabei auf Einflussgrößen wie die erwartete Belohnungshöhe, die Wahrscheinlichkeit, mit der ein bestimmtes Ereignis eintritt, oder die Dauer bis zum Erhalt einer Belohnung. Bisher liegen jedoch kaum Untersuchungen vor bezüglich zweier anderer Variablen, die ebenfalls den erwarteten Wert eines Handlungsergebnisses beeinflussen. Das sind (a) die Nähe zu dem erwarteten Ziel und (b) die bisher investierte Anstrengung, um ein Ziel zu erreichen. Das Ziel der vorliegenden Dissertation ist zu untersuchen, wie die Nähe zum Ziel und die bisher investierte Anstrengung Gehirnregionen beeinflussen, die mit der Repräsentation von Wertigkeit im Zusammenhang stehen. Dazu führten wir zwei fMRT-Studien durch, in denen wir eine klassische Belohnungs-Antizipationsaufgabe in unterschiedliche Versionen eines „Multitrial Reward Schedule“ Paradigmas integriert haben. Das bedeutet, dass die Probanden Belohnungs-Antizipationsaufgaben unter zwei unterschiedlichen Belohnungskontingenzen bearbeiteten: In der verzögerten Bedingung erhielten die Probanden einen Geldbetrag nach der erfolgreichen Bearbeitung von mehreren aufeinanderfolgenden Aufgaben, in der direkten Bedingung dagegen nach jeder korrekt ausgeführten Aufgabe. In der ersten Studie konnte eine sukzessiv ansteigende Aktivität in Abhängigkeit zur Zielnähe in der rostralen cingulären Zone identifiziert werden. Das deutet darauf hin, dass dieses Areal den Wert einer Handlung in Abhängigkeit zur Nähe zum Ziel kodiert. Die Ergebnisse der zweiten Studie zeigten, dass die bisher investierte Anstrengung kortikale Regionen moduliert, die klassischerweise mit kognitiven Kontrollfunktionen in Zusammenhang gebracht werden. Außerdem repräsentierten der posteriore laterale präfrontale Cortex und der orbitofrontale Cortex den motivationalen Kontext eines Trials anhand des Risikos des Verlustes von bisher investierter Anstrengung. Insgesamt weisen diese Befunde darauf hin, dass die rostrale cinguläre Zone eine entscheidende Rolle spielt für die Kontrolle sequenzieller Handlungsstufen, die auf eine verzögerte Belohnung ausgerichtet sind. Diese Kontrollfunktion scheint auf der kontinuierlichen Aktualisierung des Wertes einer Handlungsstufe zu basieren, der von der aktuellen Zielnähe bestimmt wird. Die Befunde der zweiten Studie lassen darauf schließen, dass sich die bisher investierte Anstrengung zur Erreichung eines Handlungsziels auf die Bereitstellung von allgemeinen kognitiven Ressourcen auswirkt. Das Risiko des Verlustes von bisher investierter Anstrengung kann außerdem ein kontextuelles Merkmal der Situation darstellen, das als Bezugsrahmen für die Evaluation des erwarteten Wertes dienen kann

A framework for studying the neurobiology of value-based decision making

Neuroeconomics is the study of the neurobiological and computational basis of value-based decision making. Its goal is to provide a biologically based account of human behaviour that can be applied in both the natural and the social sciences. This Review proposes a framework to investigate different aspects of the neurobiology of decision making. The framework allows us to bring together recent findings in the field, highlight some of the most important outstanding problems, define a common lexicon that bridges the different disciplines that inform neuroeconomics, and point the way to future applications

Potential Vulnerabilities of Neuronal Reward, Risk, and Decision Mechanisms to Addictive Drugs

How do addictive drugs hijack the brain's reward system? This review speculates how normal, physiological reward processes may be affected by addictive drugs. Addictive drugs affect acute responses and plasticity in dopamine neurons and postsynaptic structures. These effects reduce reward discrimination, increase the effects of reward prediction error signals, and enhance neuronal responses to reward-predicting stimuli, which may contribute to compulsion. Addictive drugs steepen neuronal temporal reward discounting and create temporal myopia that impairs the control of drug taking. Tonically enhanced dopamine levels may disturb working memory mechanisms necessary for assessing background rewards and thus may generate inaccurate neuronal reward predictions. Drug-induced working memory deficits may impair neuronal risk signaling, promote risky behaviors, and facilitate preaddictive drug use. Malfunctioning adaptive reward coding may lead to overvaluation of drug rewards. Many of these malfunctions may result in inadequate neuronal decision mechanisms and lead to choices biased toward drug rewards

The role of the medial prefrontal cortex in delay discounting

Indiana University-Purdue University Indianapolis (IUPUI)Increased delay discounting (DD) has been associated with and is theorized to contribute to alcoholism and substance abuse. It is also been associated with numerous other mental disorders and is believed to be a trans-disease process (i.e., a process that occurs in and contributes to multiple different pathologies). Consequently insights gained from studying DD are likely to apply to many different diseases. Studies on the neurobiological underpinnings of DD have two main interpretations. The first interpretation is that two different neurobehavioral systems exist, one favoring delayed rewards (executive system) and one favoring immediate rewards (impulsive system), and the system with the greater relative activation determines choice made by an individual. Alternatively, a single valuation system may exist. This system integrates different information about outcomes and generates a value signal that then guides decision making. Preclinical investigations have steered clear of these two different interpretations and rather focused on the role of individual structures in DD. One such structure, the rat mPFC, may generate an outcome representation of delayed rewards that is critically involved in attributing value to delayed rewards. Moreover, there is evidence indicating the rat mPFC may correspond to the primate dlPFC, an executive system structure. The current body of work set about testing the hypotheses that the mPFC is necessary for attributing value to delayed rewards and that decreasing the activity in an executive system area, and thus the executive system, shifts inter-temporal preference towards immediate rewards. To this end the rat mPFC was inactivated using an hM4Di inhibitory designer receptor exclusively activated by designer drugs (DREADD; experiment 1) or microinjections of tetrodotoxin (TTX; experiment 2) while animals completed an adjusting amount DD task. Activation of the hM4Di inhibitory DREADD receptor caused a decrease in DD, opposite of what was predicted. Electrophysiological recordings revealed a subpopulation of neurons actually increased their firing in response to hM4Di receptor activation, potentially explaining the unpredicted results. Microinjections of TTX to completely silence neural activity in the mPFC failed to produce a change in DD. Together both results indicate that mPFC activity is capable of manipulating but is not necessary for DD and the attribution of value to the delayed reward. Consequently, a secondary role for the rat mPFC in DD is proposed in line with single valuation system accounts of DD. Further investigations determining the primary structures responsible for sustaining delayed reward valuation and how manipulating the mPFC may be a means to decrease DD are warranted, and continued investigation that delineates the neurobiological processes of delayed reward valuation may provide valuable insight to both addiction and psychopathology

Decision Making and Reward in Frontal Cortex: Complementary Evidence From Neurophysiological and Neuropsychological Studies

Patients with damage to the prefrontal cortex (PFC)—especially the ventral and medial parts of PFC—often show a marked inability to make choices that meet their needs and goals. These decision-making impairments often reflect both a deficit in learning concerning the consequences of a choice, as well as deficits in the ability to adapt future choices based on experienced value of the current choice. Thus, areas of PFC must support some value computations that are necessary for optimal choice. However, recent frameworks of decision making have highlighted that optimal and adaptive decision making does not simply rest on a single computation, but a number of different value computations may be necessary. Using this framework as a guide, we summarize evidence from both lesion studies and single-neuron physiology for the representation of different value computations across PFC areas

The Neural Representation of Value and individual Differences in Human Intertemporal Choice

Intertemporal choices, or decisions that involve tradeoffs between rewards and time, are ubiquitous in our daily lives. The tendency to devalue, or discount, future rewards has been linked to maladaptive long-term health and financial outcomes. Despite their broad clinical relevance, individual differences in discounting preferences are poorly understood. In this thesis, we make progress on the understanding of the neural basis of these decisions and factors that affect individual differences. The first two chapters focus on neurobiology. Chapter 2 investigates the decision-related variables that best explain the observed patterns of BOLD activity in ventromedial prefrontal cortex (VMPFC) and ventral striatum (VS) during intertemporal choice. We find that these regions carry different signals and likely contribute to different stages of the choice process. Across the brain, we find four kinds of value-responsive regions, each carrying different combinations of value-related signals. Next, we examine whether we can predict participants\u27 choices from any or all of these groups of regions, and find that we can predict choice from most value-responsive regions, with interesting exceptions. In Chapter 3, we identify a novel brain predictor of individual differences in discounting. When participants are making judgments about how far away some number of days feels, discount rates, measured a week later, can be predicted from how VMPFC and VS respond as a function of temporal distance. This difference in the basic response to delayed time intervals could be a target for interventions aiming to reduce discount rates. In the final chapter, we find several behavioral manipulations that are able to reduce discount rates persistently and to a significant degree. We find that there is a general lack of knowledge about the normative strategy in the monetary discounting task, and that providing information about this strategy - to accept all delayed offers that provide higher interest rates than one could obtain elsewhere - reduces discounting significantly, for at least one month. Information about peers\u27 strategies for making these decisions also reduces discounting. Taken together, this work advances our understanding of individual differences in discounting and further suggests interventions that could be used to reduce discounting

The Neural Correlated of Subjective Value During Intertemporal Choice

Neuroimaging studies of decision-making have generally related neural activity to objective measures (such as reward magnitude, probability or delay), despite choice preferences being subjective. However, economic theories posit that decision-makers behave as though different options have different subjective values. Here we use functional magnetic resonance imaging to show that neural activity in several brain regions—particularly the ventral striatum, medial prefrontal cortex and posterior cingulate cortex—tracks the revealed subjective value of delayed monetary rewards. This similarity provides unambiguous evidence that the subjective value of potential rewards is explicitly represented in the human brain

A ROLE FOR THE ANTERIOR CINGULATE CORTEX IN REINFORCEMENT-GUIDED LEARNING FOR COVERT ATTENTIONAL SELECTION

The anterior cingulate cortex (ACC) has been associated with a variety of functions including conflict monitoring, error detection and more recently reward based learning. In this study we recorded from the ACC, the ventromedial prefrontal cortex (vmPFC) and the dorsolateral prefrontal cortex (dlPFC) while macaque monkeys performed a variably rewarded spatial attention task. First, we found dynamic encoding of reward outcome and reward expectancy associated with attentional targets within the ACC, mPFC and dlPFC. These results expand the function of the ACC beyond merely action value associations and suggest this area serves a broader role in reinforcement guided learning and decision making. Secondly, analysis of outcome encoding relative to reward reversal revealed two distinct types of neurons: positive/negative prediction error neurons and positive/negative prediction certainty neurons. Prediction error neurons encoded outcome information only when reward associations had recently changed and thus new outcome information was most informative for establishing new reward expectations. Prediction certainty neurons on the other hand signaled the certainty of the reward prediction itself and encoded outcome information only later, when reward expectations had been built up. Prediction error neurons showed a correlation between reward selectivity during outcome periods and reward selectivity preceding subsequent reward predictive events. This finding could serve as a link between prediction error signals and behavioural adjustment. Finally, prediction error neurons predominated in the ventral ACC whereas prediction certainty neurons predominated in dlPFC area 9. Though not definitive this supports proposals that outcome predictions are developed and adjusted within the ACC and mPFC and these predictions are used by the dlPFC to determine the behavioural response

A neural network for uncertainty anticipation and information seeking

In a world flooded with ‘click bait’, ‘alternative facts’, and ‘fake news’ one’s ability to seek out, discern, and value information is of utmost importance. Although contemporary phenomena, these cultural ills take advantage of an evolutionarily-preserved drive for humans and nonhuman animals to monitor for and pursue opportunities to gain information. Indeed, in a natural environment where rewards are scarce and can be risky, animals often seek sensory cues as a source of information about future outcomes. Interestingly, humans and nonhuman animals will seek sensory information that provides advance information that predicts an outcome even when this information does not influence the event outcome or may even come at a cost to the eventual reward. This willingness to ‘pay’ for information, despite being unable to impact task outcome, indicates that the information itself has intrinsic value to subjects. But how and where in the brain are opportunities to learn new information about uncertain events signaled? How does the brain guide behavior towards pursuing this information? Elucidating these mechanisms would expand our understanding of how information seeking interacts with primary reward seeking in naturalistic environments and could further inform theories of attention, learning, and economic decision-making. Here, I demonstrate that connected regions of the anterior cingulate cortex (ACC), striatum, and pallidum contain neurons whose activity is selectively modulated by the presence and levels of outcome uncertainty. I describe the response of these neurons, many of which anticipate the resolution of uncertainty about an outcome— including when it is resolved through the animal seeking advance information. Finally, I demonstrate that the neural activity within areas of basal ganglia in this ‘uncertainty circuit’ causally contributes to information-seeking behaviors observed in nonhuman primates. This work demonstrates that connected regions of the brain previously associated with responses to primary rewards and motivation also contain a mechanism for anticipating uncertainty resolution and directing behaviors towards pursuing information that reduces uncertainty about upcoming events

Decision Making: The Neuroethological Turn

Neuroeconomics applies models from economics and psychology to inform neurobiological studies of choice. This approach has revealed neural signatures of concepts like value, risk, and ambiguity, which are known to influence decision making. Such observations have led theorists to hypothesize a single, unified decision process that mediates choice behavior via a common neural currency for outcomes like food, money, or social praise. In parallel, recent neuroethological studies of decision making have focused on natural behaviors like foraging, mate choice, and social interactions. These decisions strongly impact evolutionary fitness and thus are likely to have played a key role in shaping the neural circuits that mediate decision making. This approach has revealed a suite of computational motifs that appear to be shared across a wide variety of organisms. We argue that the existence of deep homologies in the neural circuits mediating choice may have profound implications for understanding human decision making in health and disease