Die Rolle der Zielnähe und der investierten Anstrengung für den erwarteten Wert einer Handlung

In human neuroscientific research, there has been an increasing interest in how the brain computes the value of an anticipated outcome. However, evidence is still missing about which valuation related brain regions are modulated by the proximity to an expected goal and the previously invested effort to reach a goal. The aim of this dissertation is to investigate the effects of goal proximity and invested effort on valuation related regions in the human brain. We addressed this question in two fMRI studies by integrating a commonly used reward anticipation task in differential versions of a Multitrial Reward Schedule Paradigm. In both experiments, subjects had to perform consecutive reward anticipation tasks under two different reward contingencies: in the delayed condition, participants received a monetary reward only after successful completion of multiple consecutive trials. In the immediate condition, money was earned after every successful trial. In the first study, we could demonstrate that the rostral cingulate zone of the posterior medial frontal cortex signals action value contingent to goal proximity, thereby replicating neurophysiological findings about goal proximity signals in a homologous region in non-human primates. The findings of the second study imply that brain regions associated with general cognitive control processes are modulated by previous effort investment. Furthermore, we found the posterior lateral prefrontal cortex and the orbitofrontal cortex to be involved in coding for the effort-based context of a situation. In sum, these results extend the role of the human rostral cingulate zone in outcome evaluation to the continuous updating of action values over a course of action steps based on the proximity to the expected reward. Furthermore, we tentatively suggest that previous effort investment invokes processes under the control of the executive system, and that posterior lateral prefrontal cortex and the orbitofrontal cortex are involved in an effort-based context representation that can be used for outcome evaluation that is dependent on the characteristics of the current situation.Derzeit besteht im Bereich der Neurowissenschaften ein großes Interesse daran aufzuklären, auf welche Weise verschiedene Variablen die Wertigkeit eines erwarteten Handlungsziels beeinflussen bzw. welche Hirnregionen an der Repräsentation der Wertigkeit eines Handlungsziels beteiligt sind. Die meisten Untersuchungen beziehen sich dabei auf Einflussgrößen wie die erwartete Belohnungshöhe, die Wahrscheinlichkeit, mit der ein bestimmtes Ereignis eintritt, oder die Dauer bis zum Erhalt einer Belohnung. Bisher liegen jedoch kaum Untersuchungen vor bezüglich zweier anderer Variablen, die ebenfalls den erwarteten Wert eines Handlungsergebnisses beeinflussen. Das sind (a) die Nähe zu dem erwarteten Ziel und (b) die bisher investierte Anstrengung, um ein Ziel zu erreichen. Das Ziel der vorliegenden Dissertation ist zu untersuchen, wie die Nähe zum Ziel und die bisher investierte Anstrengung Gehirnregionen beeinflussen, die mit der Repräsentation von Wertigkeit im Zusammenhang stehen. Dazu führten wir zwei fMRT-Studien durch, in denen wir eine klassische Belohnungs-Antizipationsaufgabe in unterschiedliche Versionen eines „Multitrial Reward Schedule“ Paradigmas integriert haben. Das bedeutet, dass die Probanden Belohnungs-Antizipationsaufgaben unter zwei unterschiedlichen Belohnungskontingenzen bearbeiteten: In der verzögerten Bedingung erhielten die Probanden einen Geldbetrag nach der erfolgreichen Bearbeitung von mehreren aufeinanderfolgenden Aufgaben, in der direkten Bedingung dagegen nach jeder korrekt ausgeführten Aufgabe. In der ersten Studie konnte eine sukzessiv ansteigende Aktivität in Abhängigkeit zur Zielnähe in der rostralen cingulären Zone identifiziert werden. Das deutet darauf hin, dass dieses Areal den Wert einer Handlung in Abhängigkeit zur Nähe zum Ziel kodiert. Die Ergebnisse der zweiten Studie zeigten, dass die bisher investierte Anstrengung kortikale Regionen moduliert, die klassischerweise mit kognitiven Kontrollfunktionen in Zusammenhang gebracht werden. Außerdem repräsentierten der posteriore laterale präfrontale Cortex und der orbitofrontale Cortex den motivationalen Kontext eines Trials anhand des Risikos des Verlustes von bisher investierter Anstrengung. Insgesamt weisen diese Befunde darauf hin, dass die rostrale cinguläre Zone eine entscheidende Rolle spielt für die Kontrolle sequenzieller Handlungsstufen, die auf eine verzögerte Belohnung ausgerichtet sind. Diese Kontrollfunktion scheint auf der kontinuierlichen Aktualisierung des Wertes einer Handlungsstufe zu basieren, der von der aktuellen Zielnähe bestimmt wird. Die Befunde der zweiten Studie lassen darauf schließen, dass sich die bisher investierte Anstrengung zur Erreichung eines Handlungsziels auf die Bereitstellung von allgemeinen kognitiven Ressourcen auswirkt. Das Risiko des Verlustes von bisher investierter Anstrengung kann außerdem ein kontextuelles Merkmal der Situation darstellen, das als Bezugsrahmen für die Evaluation des erwarteten Wertes dienen kann

Brain Cells in the Avian ‘Prefrontal Cortex’ Code for Features of Slot-Machine-Like Gambling

Slot machines are the most common and addictive form of gambling. In the current study, we recorded from single neurons in the ‘prefrontal cortex’ of pigeons while they played a slot-machine-like task. We identified four categories of neurons that coded for different aspects of our slot-machine-like task. Reward-Proximity neurons showed a linear increase in activity as the opportunity for a reward drew near. I-Won neurons fired only when the fourth stimulus of a winning (four-of-a-kind) combination was displayed. I-Lost neurons changed their firing rate at the presentation of the first nonidentical stimulus, that is, when it was apparent that no reward was forthcoming. Finally, Near-Miss neurons also changed their activity the moment it was recognized that a reward was no longer available, but more importantly, the activity level was related to whether the trial contained one, two, or three identical stimuli prior to the display of the nonidentical stimulus. These findings not only add to recent neurophysiological research employing simulated gambling paradigms, but also add to research addressing the functional correspondence between the avian NCL and primate PFC

Outcome modulation across tasks in the primate dorsolateral prefrontal cortex

Animals need to learn and to adapt to new and changing environments so that appropriate actions that lead to desirable outcomes are acquired within each context. The prefrontal cortex (PF) is known to underlie such function that directly implies that the outcome of each response must be represented in the brain for behavioral policies update. However, whether such PF signal is context dependent or it is a general representation beyond the specificity of a context is still unclear. Here, we analyzed the activity of neurons in the dorsolateral PF (PFdl) recorded while two monkeys performed two perceptual magnitude discrimination tasks. Both tasks were well known by the monkeys and unexpected changes did not occur but the difficulty of the task varied from trial to trial and thus the monkeys made mistakes in a proportion of trials. We show a context-independent coding of the response outcome with neurons maintaining similar selectivity in both task contexts. Using a classification method of the neural activity, we also show that the trial outcome could be well predicted from the activity of the same neurons in the two contexts. Altogether, our results provide evidence of high degree of outcome generality in PFdl

The Neurobiology of Decision: Consensus and Controversy

We review and synthesize recent neurophysiological studies of decision making in humans and nonhuman primates. From these studies, the basic outline of the neurobiological mechanism for primate choice is beginning to emerge. The identified mechanism is now known to include a multicomponent valuation stage, implemented in ventromedial prefrontal cortex and associated parts of striatum, and a choice stage, implemented in lateral prefrontal and parietal areas. Neurobiological studies of decision making are beginning to enhance our understanding of economic and social behavior as well as our understanding of significant health disorders where people\u27s behavior plays a key role

Investigating Habits: Strategies,Technologies and Models

Understanding habits at a biological level requires a combination of behavioral observations and measures of ongoing neural activity. Theoretical frameworks as well as definitions of habitual behaviors emerging from classic behavioral research have been enriched by new approaches taking account of the identification of brain regions and circuits related to habitual behavior. Together, this combination of experimental and theoretical work has provided key insights into how brain circuits underlying action-learning and action-selection are organized, and how a balance between behavioral flexibility and fixity is achieved. New methods to monitor and manipulate neural activity in real time are allowing us to have a first look under the hood of a habit as it is formed and expressed. Here we discuss ideas emerging from such approaches. We pay special attention to the unexpected findings that have arisen from our own experiments suggesting that habitual behaviors likely require the simultaneous activity of multiple distinct components, or operators, seen as responsible for the contrasting dynamics of neural activity in both cortico-limbic and sensorimotor circuits recorded concurrently during different stages of habit learning. The neural dynamics identified thus far do not fully meet expectations derived from traditional models of the structure of habits, and the behavioral measures of habits that we have made also are not fully aligned with these models. We explore these new clues as opportunities to refine an understanding of habits

Dorsolateral prefrontal lesions do not impair tests of scene learning and decision-making that require frontal–temporal interaction

Theories of dorsolateral prefrontal cortex (DLPFC) involvement in cognitive function variously emphasize its involvement in rule implementation, cognitive control, or working and/or spatial memory. These theories predict broad effects of DLPFC lesions on tests of visual learning and memory. We evaluated the effects of DLPFC lesions (including both banks of the principal sulcus) in rhesus monkeys on tests of scene learning and strategy implementation that are severely impaired following crossed unilateral lesions of frontal cortex and inferotemporal cortex. Dorsolateral lesions had no effect on learning of new scene problems postoperatively, or on the implementation of preoperatively acquired strategies. They were also without effect on the ability to adjust choice behaviour in response to a change in reinforcer value, a capacity that requires interaction between the amygdala and frontal lobe. These intact abilities following DLPFC damage support specialization of function within the prefrontal cortex, and suggest that many aspects of memory and strategic and goal-directed behaviour can survive ablation of this structure

A neural network for uncertainty anticipation and information seeking

In a world flooded with ‘click bait’, ‘alternative facts’, and ‘fake news’ one’s ability to seek out, discern, and value information is of utmost importance. Although contemporary phenomena, these cultural ills take advantage of an evolutionarily-preserved drive for humans and nonhuman animals to monitor for and pursue opportunities to gain information. Indeed, in a natural environment where rewards are scarce and can be risky, animals often seek sensory cues as a source of information about future outcomes. Interestingly, humans and nonhuman animals will seek sensory information that provides advance information that predicts an outcome even when this information does not influence the event outcome or may even come at a cost to the eventual reward. This willingness to ‘pay’ for information, despite being unable to impact task outcome, indicates that the information itself has intrinsic value to subjects. But how and where in the brain are opportunities to learn new information about uncertain events signaled? How does the brain guide behavior towards pursuing this information? Elucidating these mechanisms would expand our understanding of how information seeking interacts with primary reward seeking in naturalistic environments and could further inform theories of attention, learning, and economic decision-making. Here, I demonstrate that connected regions of the anterior cingulate cortex (ACC), striatum, and pallidum contain neurons whose activity is selectively modulated by the presence and levels of outcome uncertainty. I describe the response of these neurons, many of which anticipate the resolution of uncertainty about an outcome— including when it is resolved through the animal seeking advance information. Finally, I demonstrate that the neural activity within areas of basal ganglia in this ‘uncertainty circuit’ causally contributes to information-seeking behaviors observed in nonhuman primates. This work demonstrates that connected regions of the brain previously associated with responses to primary rewards and motivation also contain a mechanism for anticipating uncertainty resolution and directing behaviors towards pursuing information that reduces uncertainty about upcoming events

The Role Of The Prelimbic, Infralimbic, And Cerebellar Cortices In Operant Behavior

Operant (instrumental) conditioning is a laboratory method for investigating voluntary behavior and involves training a particular response, such as pressing a lever, to earn a reinforcer. Operant behavior is generally divided into two categories: actions and habits. Actions are goal-directed and controlled by response-outcome (R-O) associations. Habits are stimulus-driven and controlled by stimulus-response associations (S-R). Behavior is determined to be goal-directed or habitual by whether or not it is sensitive (action) or insensitive (habit) to reinforcer/outcome devaluation. Many brain regions have been linked to the learning and/or expression of actions and/or habits. This dissertation investigates a few different brain regions in goal-directed and habitual behavior, and determines more specific roles for the prelimbic cortex, infralimbic cortex, prelimbic cortex to dorsomedial striatum pathway, and Crus I/II of the cerebellum. Chapter two investigates the prelimbic and infralimbic cortices in goal-directed behavior. We trained rats on a two-response paradigm, where one response was extensively-trained, and a second response was minimally-trained in a separate context. This maintained both responses as goal-directed. In experiment 1, inactivation of the prelimbic cortex at time of test resulted in an attenuation of responding, but only for the minimally-trained response. This implicates the prelimbic cortex in the expression of goal-directed behavior, but only when that goal-directed behavior is minimally-trained. In experiment 2, we repeated the procedure with infralimbic cortex inactivation and found an attenuation of the extensively-trained response. This implicates the infralimbic cortex in the expression of extensively-trained behavior that is goal-directed. The third chapter examines the role of the prelimbic cortex-to-dorsomedial striatal pathway in minimally-trained operant behavior. Both regions have been implicated in operant behaviors and have strong anatomical connections, but few studies have directly linked them together in the mediation of operant behaviors. After minimal instrumental conditioning, we silenced projections from the prelimbic cortex to the dorsomedial striatum and found that instrumental behavior was reduced, implicating this PL-DMS pathway in the expression of minimally-trained operant responding. The final chapter examines the role of Crus I/II of the cerebellar cortex in the expression of goal-directed and habitual behavior. The cerebellum is well-characterized as a mediator of motor coordination via its connections with the motor cortex. There is also evidence of connections between Crus I/II and non-motor regions of the prefrontal cortex. Additionally, recent studies have pointed towards a role for Crus I/II in non-motor function. In experiment 1, rats learned one minimally-trained and one extensively-trained response, and both responses were goal-directed. Inactivation of Crus I/II attenuated responding only in rats that had undergone reinforcer devaluation. Residual responding in rats that have undergone reinforcer devaluation is attributed to habit, suggesting that Crus I/II may be involved in habit expression. In a follow-up experiment, we extensively-trained a single response and verified that it was expressed as a habit. This time, Crus I/II inactivation at time of test had no effect. Overall, this complex pattern of results suggests the possibility that Crus I/II of the cerebellar cortex is only engaged in habit expression when two responses are trained, but further experiments will be necessary to verify this

Representations of Relative Value Coding in the Orbitofrontal Cortex and Amygdala

In order to guide behavior, humans and animals must flexibly evaluate the motivational significance of stimuli in the environment. We sought to determine if, in different contexts, neurons in the amygdala and orbitofrontal cortex (OFC) indeed rescale their calculation of the motivational significance of stimuli that predict rewards. We used a "contrast revaluation" task in which the reward associated with one stimulus is held constant while other rewards within a particular context (or block of trials) change. This manipulation modulates the relative significance of the reward associated with one stimulus without changing its absolute amount. We recorded the activity of individual neurons in the amygdala and OFC of two monkeys while they performed the contrast revaluation task. On every trial, a monkey viewed one of two conditioned stimuli (CSs; distinct fractal patterns), each predictive of a different reward amount. CSs were novel for every experiment. Unconditioned stimulus (US, liquid reward) delivery followed CS presentation and a brief temporal gap (trace interval). The task consisted of three trial blocks, with switches between blocks occurring without warning. The presentation of CS2 predicted either a small (first and third blocks) or large US (second block). The presentation of CS1 predicted delivery of a medium US in all blocks. Thus CS1 corresponded to the "better" trial type in blocks 1 and 3, but not 2. Anticipatory licking behavior indicated that the monkey adapted its behavior depending upon the relative amount of expected reward. Although the reward amount associated with CS1 remained constant throughout the experiment, anticipatory licking decreased in block 2 and increased in block 3 - the blocks in which CS1 trials had become relatively less (block 2) and more (block 3) valuable. Strikingly, many individual amygdala and OFC neurons also modulated their responses to CS1 depending upon the block. Because this CS predicts the exact same reward in each block, these neurons cannot simply represent the sensory properties of a US associated with a CS. This finding demonstrates that amygdala and OFC neurons are often sensitive to the relative motivational significance of a CS, and not just to the sensory properties of its associated US or to the absolute value of the specific reward. Neurons in both the OFC and amygdala encode the relative value of CS1 but OFC neurons significantly encode relative value earlier than amygdala neurons. Cells in the amygdala and OFC code different properties during different time intervals during the trial and are consistent in valence when they code multiple properties. This implies that neurons are tracking state value: the overall motivational value of an organism's internal and external environment across time and sensory stimuli. Neurons that code relative value during the CS-trace interval and during reinforcement are also consistent in the valence that they code further supporting that these cells track state value. The neurons code with the same sign and strength whether the neuron is representing the relative value of the reward with no sensory input of the reward during CS or trace interval, or actually experiencing the reward during the US interval. Further, amygdala and OFC neural activity was correlated with the animal's behavioral performance, suggesting that these neurons could form the basis for animal's behavioral adaptation during contrast revaluation. These neural representations could also support behavior in other situations requiring flexible and adaptive evaluation of the motivational significance of stimuli