The role of the lateral prefrontal cortex and anterior cingulate in stimulus–response association reversals

Many complex tasks require us to flexibly switch between behavioral rules, associations, and strategies. The prefrontal cerebral cortex is thought to be critical to the performance of such behaviors, although the relative contribution of different components of this structure and associated subcortical regions are not fully understood. We used functional magnetic resonance imaging to measure brain activity during a simple task which required repeated reversals of a rule linking a colored cue and a left/right motor response. Each trial comprised three discrete events separated by variable delay periods. A colored cue instructed which response was to be executed, followed by a go signal which told the subject to execute the response and a feedback instruction which indicated whether to ‘‘hold’’ or ‘‘f lip’’ the rule linking the colored cue and response. The design allowed us to determine which brain regions were recruited by the specific demands of preparing a rule contingent motor response, executing such a response, evaluating the significance of the feedback, and reconfiguring stimulus–response (SR) associations. The results indicate that an increase in neural activity occurs within the anterior cingulate gyrus under conditions in which SR associations are labile. In contrast, lateral frontal regions are activated by unlikely/unexpected perceptual events regardless of their significance for behavior. A network of subcortical structures, including the mediodorsal nucleus of the thalamus and striatum were the only regions showing activity that was exclusively correlated with the neurocognitive demands of reversing SR associations. We conclude that lateral frontal regions act to evaluate the behavioral significance of perceptual events, whereas medial frontal–thalamic circuits are involved in monitoring and reconfiguring SR associations when necessary

Eligibility Traces and Plasticity on Behavioral Time Scales: Experimental Support of neoHebbian Three-Factor Learning Rules

Most elementary behaviors such as moving the arm to grasp an object or walking into the next room to explore a museum evolve on the time scale of seconds; in contrast, neuronal action potentials occur on the time scale of a few milliseconds. Learning rules of the brain must therefore bridge the gap between these two different time scales. Modern theories of synaptic plasticity have postulated that the co-activation of pre- and postsynaptic neurons sets a flag at the synapse, called an eligibility trace, that leads to a weight change only if an additional factor is present while the flag is set. This third factor, signaling reward, punishment, surprise, or novelty, could be implemented by the phasic activity of neuromodulators or specific neuronal inputs signaling special events. While the theoretical framework has been developed over the last decades, experimental evidence in support of eligibility traces on the time scale of seconds has been collected only during the last few years. Here we review, in the context of three-factor rules of synaptic plasticity, four key experiments that support the role of synaptic eligibility traces in combination with a third factor as a biological implementation of neoHebbian three-factor learning rules

A critic in action? A functional examination of the Striato-Pallido-Habenular circuit

The basal ganglia (BG) and the midbrain dopamine (DA) system are considered key loci of reinforcement learning (RL), or learning by "trial and error", in the brain. The BG are implicated in action selection, and thus the "trial" part of the learning process, and the dopamine (DA) system is known to encode "error" signals. This DA error signal—the reward prediction error—is thought to adjust the BG’s propensity to select a "tried" action again in the future. In RL terms, the action-selecting BG is called the "actor", and the action-critiquing DA system the "critic". Here, a candidate striato-pallido-habenular "critic pathway" upstream of the DA system is examined. The proposed critic pathway originates in the striosome compartment of the striatum, and projects via a non-canonical internal globus pallidus (GPi) population to the lateral habenula (LHb). LHb activity has been shown to encode the inverse of the DA reward prediction error signal, and to cause inhibition within the DA system. This posits the described striato-pallidohabenular pathway as key part of the critic circuit. To investigate the role of the striato-pallido-habenular pathway in action, we recorded and manipulated the neuronal activity of neurons of the striatal striosome (Article I) and the GPi (Article II & III) in mice performing tasks that engendered trial and error behavioral strategies. We found that the activity of striatal striosome neurons had much in common with that of neurons within the striatal "actor pathways". Moreover, all striatal neurons jointly represented the evolving behavioral context in a spatiotemporally continuous population code, undermining notions of discrete and well-de ned action selection and evaluation processes. The results of our experiments on the GPi challenged its proposed role in driving the LHb’s inverse reward prediction error signals, and implicated the GPi-adjacent lateral hypothalamus (LHA) in that role instead. In sum, the studies included here call into question whether the striato-pallido-habenular pathway serves as a critic in BG-mediated actio

Age Differences in Striatal Delay Sensitivity during Intertemporal Choice in Healthy Adults

Intertemporal choices are a ubiquitous class of decisions that involve selecting between outcomes available at different times in the future. We investigated the neural systems supporting intertemporal decisions in healthy younger and older adults. Using functional neuroimaging, we find that aging is associated with a shift in the brain areas that respond to delayed rewards. Although we replicate findings that brain regions associated with the mesolimbic dopamine system respond preferentially to immediate rewards, we find a separate region in the ventral striatum with very modest time dependence in older adults. Activation in this striatal region was relatively insensitive to delay in older but not younger adults. Since the dopamine system is believed to support associative learning about future rewards over time, our observed transfer of function may be due to greater experience with delayed rewards as people age. Identifying differences in the neural systems underlying these decisions may contribute to a more comprehensive model of age-related change in intertemporal choice

Cocaine Exposure Shifts the Balance of Associative Encoding from Ventral to Dorsolateral Striatum

Both dorsal and ventral striatum are implicated in the “habitization” of behavior that occurs in addiction. Here we examined the effect of cocaine exposure on associative encoding in these two regions. Neural activity was recorded during go/no-go discrimination learning and reversal. Activity in ventral striatum developed and reversed rapidly, tracking the valence of the predicted outcome, whereas activity in dorsolateral striatum developed and reversed more slowly, tracking discriminative responding. This difference is consistent with the putative roles of these two areas in promoting habit-like behavior. Dorsolateral striatum has been directly implicated in habit or stimulus–response learning, whereas ventral striatum appears to be involved indirectly by allowing cues associated with reward to exert a general motivational influence on responding. Interestingly cocaine exposure did not uniformly enhance processing across both regions. Instead cocaine reduced the degree and flexibility of cue-evoked firing in ventral striatum while marginally enhanced cue-selective firing in dorsolateral striatum. Thus cocaine exposure causes regionally specific effects on neural processing in striatum; these effects may promote the habitization of behavior by shifting control from ventral to dorsolateral regions

The basal ganglia: A vertebrate solution to the selection problem?

A selection problem arises whenever two or more competing systems seek simultaneous access to a restricted resource. Consideration of several selection architectures suggests there are significant advantages for systems which incorporate a central switching mechanism. We propose that the vertebrate basal ganglia have evolved as a centralized selection device, specialized to resolve conflicts over access to limited motor and cognitive resources. Analysis of basal ganglia functional architecture and its position within a wider anatomical framework suggests it can satisfy many of the requirements expected of an efficient selection mechanism

The Nucleus Accumbens Core Dopamine D1 and Glutamate AMPA/NMDA Receptors Play a Transient Role in the Performance of Pavlovian Approach Behavior

The role of the nucleus accumbens core (NAc core) continues to be redefined with newly acquired data on neurochemical mechanisms mediating the learning and performance of behavior. Previous empirical data showed that dopamine transmission at the D1 receptor (D1R) plays a transient role in the expression of learned Pavlovian approach behavior. Here we show that, prior to overtraining, dopamine activity at D1Rs specifically within the NAc core is critical for the performance of approach behavior elicited by the recently-acquired reward-paired cue. Blockade of D1Rs in the NAc core, but not the dorsomedial striatum or NAc shell, disrupted approach responses during early training; however, the dependence of Pavlovian approach on D1R transmission declined throughout training. Upon blockade of NAc core D1Rs during extended training, the expression of Pavlovian approach responses remained intact. Given these findings we next explored whether a) neuronal activity within the core of accumbens still mediates cued approach during the late training stages in the absence of D1R transmission by relying on glutamatergic transmission or b) whether mediation of the cued approach becomes independent of the NAc core itself, i.e., shifts to another substrate. We blocked AMPA/NMDA receptors in the NAc core during early versus extended training and showed that loss of neuronal activation in the NAc core only disrupted expression of conditioned stimulus-elicited responses during early training. Our results indicate that NAc core activity is not necessary for the expression of well-acquired approach

Striatal neuropeptides enhance selection and rejection of sequential actions

The striatum is the primary input nucleus for the basal ganglia, and receives glutamatergic afferents from the cortex. Under the hypothesis that basal ganglia perform action selection, these cortical afferents encode potential “action requests.” Previous studies have suggested the striatum may utilize a mutually inhibitory network of medium spiny neurons (MSNs) to filter these requests so that only those of high salience are selected. However, the mechanisms enabling the striatum to perform clean, rapid switching between distinct actions that form part of a learned action sequence are still poorly understood. Substance P (SP) and enkephalin are neuropeptides co-released with GABA in MSNs preferentially expressing D1 or D2 dopamine receptors respectively. SP has a facilitatory effect on subsequent glutamatergic inputs to target MSNs, while enkephalin has an inhibitory effect. Blocking the action of SP in the striatum is also known to affect behavioral transitions. We constructed phenomenological models of the effects of SP and enkephalin, and integrated these into a hybrid model of basal ganglia comprising a spiking striatal microcircuit and rate–coded populations representing other major structures. We demonstrated that diffuse neuropeptide connectivity enhanced the selection of unordered action requests, and that for true action sequences, where action semantics define a fixed structure, a patterning of the SP connectivity reflecting this ordering enhanced selection of actions presented in the correct sequential order and suppressed incorrect ordering. We also showed that selective pruning of SP connections allowed context–sensitive inhibition of specific undesirable requests that otherwise interfered with selection of an action group. Our model suggests that the interaction of SP and enkephalin enhances the contrast between selection and rejection of action requests, and that patterned SP connectivity in the striatum allows the “chunking” of actions and improves selection of sequences. Efficient execution of action sequences may therefore result from a combination of ordered cortical inputs and patterned neuropeptide connectivity within striatum

Organization of brain circuits that control motivated behaviors

Eudemonia (Greek: εὐδαιμονία) is an anthropocentric term describing the absolute well-being in the Aristotelian ethics, along with the terms “arête” (virtue) and “phronesis” (wisdom from an ethical and practical point of view). Arête and virtue will guide ones decisions to promote future eudemonia and ultimately survival. Some of the classic parameters that shape decisions across species include the ability to experience reward, the good prognosis of reward and the avoidance of events that would harm ones physiology and psychology. Such behavioral complexity is orchestrated by a plethora of brain circuits. A well-established candidate mediating reward-related behaviors is the neurotransmitter dopamine and the dopaminergic pathways. The activity of the dopaminergic pathways is influenced and modulated by cortical and subcortical areas involved in pain, mood regulation, arousal, stress and substance abuse (Hikosaka et al., 2008). These neuronal networks, affecting directly or indirectly the dopamine activity, are shaping motivated behaviors and decisionmaking. The conceptual aim of my thesis is to understand molecular, anatomical and functional features of brain reward circuits. In this thesis I will be summarizing the background literature of my projects with special references to the brain areas including the basal ganglia structures, the habenula complex and the parabrachial nucleus, in the context of motivation, anhedonia and pain. In Chapter 1, the brain-reward system will be discussed, with references to the old and new methodologies used to study its neurobiology. The diverse modulatory effects of dopamine and its implications in the prediction of reward are also described here. Chapter 2 reviews the basal ganglia literature. The neurochemistry and neuroanatomy of the main elements of basal ganglia are described here, with the main focus being on the area of striatum. The physiology and pathophysiology of striatal circuits are central in the control of motor, cognitive and limbic functions. Relevant to this chapter are the Papers I and II. The work in Paper I, describes the cell-type specific corticostriatal projections as well as the molecular discrimination of the cortical layers. It will be also examined here, the acute-cocaineexposure effect in motor and reward function on a molecular and behavioral level. During this study, an open-access pipeline was developed in order to easily identify and map neuronal features (cell bodies, axons, dendrites) on a standardized brain atlas. Paper II deals with the complexity of striatal anatomy and neurochemistry. This work describes how cell-type-dependent and cell-type-independent spaces uniquely constitute the striatum. There are special references to a newly identified striatal medium spiny projection neuron. In Chapter 3, the epithalamic structure of lateral habenula is reviewed on an anatomical and functional level. The general focus of the chapter is on how the lateral habenula is mediating anti-reward signals through its connectivity patterns and its direct effects on neuromodulatory systems. Paper III is related to this chapter. The Paper III, reveals how the lateral habenula-mediated aversive and fear behaviors are reflected on a network connectivity level. Central in this project are the adjacent populations of the basal ganglia-hypothalamus borders that are of distinct neurochemistry and connectivity and also convey differential anti-reward signals to the lateral habenula. Chapter 4, reviews the parabrachial nucleus structure and function. It is discussed here, the importance of the parabrachial activity in conveying internal and external signals (pain, fear, visceral malaise) to forebrain in order to promote decision-making. The last Paper IV (manuscript) relates to chapter 4. Paper IV, deals with the unique cytoarchitecture of the lateral parabrachial nucleus and its afferent and efferent connectivity. The main focal point in this work is the previously undescribed cholinergic subpopulation of the nucleus, which displays both classic and distinct neuroanatomical features when compared with the well-established markers of the area. In summary, this thesis captures the diversity of neuronal substrates scattered across the brain and yet function in the direction of maximizing well-being. Common substrate of these structures is their direct or indirect connections with the dopaminergic centers and the extended brain reward circuit. Paradoxically, activation of most of the areas examined here is primarily signaling aversion, fear and pain rather that reward! The combinatorial view when studying advance behavioral aspects leads to a finer comprehension of the physiological function