From the ventral to the dorsal striatum: Devolving views of their roles in drug addiction

AbstractWe revisit our hypothesis that drug addiction can be viewed as the endpoint of a series of transitions from initial voluntarily drug use to habitual, and ultimately compulsive drug use. We especially focus on the transitions in striatal control over drug seeking behaviour that underlie these transitions since functional heterogeneity of the striatum was a key area of Ann Kelley's research interests and one in which she made enormous contributions. We also discuss the hypothesis in light of recent data that the emergence of a compulsive drug seeking habit both reflects a shift to dorsal striatal control over behaviour and impaired prefontal cortical inhibitory control mechanisms. We further discuss aspects of the vulnerability to compulsive drug use and in particular the impact of impulsivity. In writing this review we acknowledge the untimely death of an outstanding scientist and a dear personal friend

Disentangling the Roles of Approach, Activation and Valence in Instrumental and Pavlovian Responding

Hard-wired, Pavlovian, responses elicited by predictions of rewards and punishments exert significant benevolent and malevolent influences over instrumentally-appropriate actions. These influences come in two main groups, defined along anatomical, pharmacological, behavioural and functional lines. Investigations of the influences have so far concentrated on the groups as a whole; here we take the critical step of looking inside each group, using a detailed reinforcement learning model to distinguish effects to do with value, specific actions, and general activation or inhibition. We show a high degree of sophistication in Pavlovian influences, with appetitive Pavlovian stimuli specifically promoting approach and inhibiting withdrawal, and aversive Pavlovian stimuli promoting withdrawal and inhibiting approach. These influences account for differences in the instrumental performance of approach and withdrawal behaviours. Finally, although losses are as informative as gains, we find that subjects neglect losses in their instrumental learning. Our findings argue for a view of the Pavlovian system as a constraint or prior, facilitating learning by alleviating computational costs that come with increased flexibility

Human Reinforcement Learning: Insights from intracranial recordings and stimulation

Reinforcement learning is the process by which individuals alter their decisions to maximize positive outcomes, and minimize negative outcomes. It is a cognitive process that is widely used in our daily lives and is often disrupted during psychiatric disease. Thus, a major goal of neuroscience is to characterize the neural underpinnings of reinforcement learning. Whereas animal studies have utilized invasive physiological methods to characterize several neural mechanisms that underlie reinforcement learning, human studies have largely relied on non-invasive techniques that have reduced physiological precision. Although ethical limitations preclude the use of invasive physiological methods in healthy human populations, patient populations undergoing certain neurosurgical interventions offer a rare opportunity to directly assay neural activity from the brain during human reinforcement learning. This dissertation presents early findings from this research effort

Critical Roles for Anterior Insula and Dorsal Striatum in Punishment-Based Avoidance Learning

SummaryThe division of human learning systems into reward and punishment opponent modules is still a debated issue. While the implication of ventral prefrontostriatal circuits in reward-based learning is well established, the neural underpinnings of punishment-based learning remain unclear. To elucidate the causal implication of brain regions that were related to punishment learning in a previous functional neuroimaging study, we tested the effects of brain damage on behavioral performance, using the same task contrasting monetary gains and losses. Cortical and subcortical candidate regions, the anterior insula and dorsal striatum, were assessed in patients presenting brain tumor and Huntington disease, respectively. Both groups exhibited selective impairment of punishment-based learning. Computational modeling suggested complementary roles for these structures: the anterior insula might be involved in learning the negative value of loss-predicting cues, whereas the dorsal striatum might be involved in choosing between those cues so as to avoid the worst

Dopamine, decision-making, and aging : neural and behavioural correlates

On any given day, we make tons of decisions. These can be as simple as deciding how to dress or what to eat, or more complex, such as whether to spend or invest money. Good decision-making involves being able to select the best alternative from a range of options, and adjust one’s preferences based on what is happening in the environment. As humans get older, their ability to do this changes. Age-related changes in decision-making ability result from changes in brain structure and function, such as the deterioration of the brain’s dopaminergic system in old age. In this thesis, we used a sample of 30 older and 30 younger participants to investigate age-related differences in neural and behavioural correlates of value-based decision-making, which involves making decisions that can result in rewards and punishments. Such decisions are known to rely on dopaminergic functioning. In the brain, we have looked at neural activity reflecting value and reward prediction errors (RPEs), the availability of dopamine D1 receptors, and integrity of white matter microstructure. For the behavioural data, we have used computational modelling to disentangle motivational biases and other parameters reflecting parts of the learning process that underlies successful decision-making. In study 1, we investigated whether performance on a value-based decision-making task differed between the two age groups. We also looked at whether performance differences could be explained by differential neural processing of RPEs and expected value in the striatum and prefrontal cortex (PFC). We used a novel computational model to estimate expected value, decision uncertainty and confidence. We found that older adults earned fewer rewards on the task. The number of rewards earned could be predicted by the strength of the neural signal reflecting expected value in the ventromedial PFC (vmPFC), which was attenuated in older adults. Beyond age, the strength of this neural signal could be predicted by dopamine D1 receptor (D1-R) availability in the nucleus accumbens (NAcc). In study 2, we showed that integrity of white matter microstructure in the pathway between the NAcc and vmPFC also predicted the neural value signal in the vmPFC, independently of age and D1-R availability in the NAcc. In study 3 and 4, we focused on dissociating the effects of action and valence on neural and behavioural correlates of decision-making. In study 3, we used com-putational modelling to characterize faster learning to act in response to rewards, and abstaining from acting in response to punishments, as being the result of biased instrumental learning. Study 3 also showed that variability in dopamine D1-R availability could be divided into cortical, dorsal striatal and ventral striatal components. Regardless of age, dopamine D1-R availability in the dorsal striatal component was related to biased learning from rewarded actions. In study 4 we investigated anticipatory value signals after learning had reached an asymptote. We observed no differences between age groups in anticipatory neural responses to action and valence, and no relationship between anticipatory neural signals and dopamine D1-R availability. Older adults did show an attenuated punishment prediction error signal in the insula, compared with younger adults. The strength of differentiation between reward- and punishment prediction error signals in the insula was related to dopamine D1-R availability in the cortex. These studies have demonstrated that the existing theoretical framework sur-rounding the role of dopamine system in decision-making and aging fits with dopamine D1-R availability data and behavioural data in older and younger adults, and partly explain why older adults show behavioural differences in value-based decision-making tasks. Collectively, the studies in this thesis provide important multimodal evidence that increases our understanding of the neural correlates that underlie value-based decision-making and how they are affected in healthy aging

Adolescent Development of Multiple Learning Systems

Adolescence is a time filled with opportunities for making choices that have not been encountered before. How do adolescents learn to make these decisions? Maturation of learning processes coupled with dynamic changes in brain systems for learning must be studied in order to determine the mechanisms that underlie adolescent decision making. Research in adults has found contributions from multiple learning systems for decision making. One such system learns incrementally from feedback and reinforcement, and depends in part on the striatum. Another system, in the hippocampus, encodes episodes and allows for flexible use of learned information when required by novel contexts. Recent research in adults explores how these systems can cooperate and compete to facilitate decision making. Ongoing research into learning and decision making processes over the course of adolescence has also implicated the striatum in learning and decision making, but how the hippocampus and striatum interact for decision making remains unknown. In this dissertation I investigate contributions of multiple learning systems for learning and decision making in adolescence. I leverage what is known about underlying brain systems for learning and decision making in adults, and consider how changes in these same systems over adolescence might contribute to behavioral shifts in adolescence. Specifically, in the studies included here, I show how developmental trajectories for learning can enhance performance in adolescents for some types of learning and not others. In the first study I ask how do the striatal and hippocampal systems contribute to feedback based learning in adolescence? I show that in adolescents, both the hippocampus and the striatum contribute to probabilistic feedback learning, and that this type of learning is better in adolescents than in adults. This response to feedback in the hippocampus was found to relate to memory accuracy for features of feedback events only in adolescents. Pushing the finding of hippocampal activation in adolescents, in the second study I ask how does learned value influence flexible decision making in adolescence? Adolescents did not show reliable transfer of value, but there were individual differences in this tendency. Thus, in the third study, I ask which brain regions account for individual differences in learning and value transfer? I show that variability in connectivity at rest between the hippocampus and the vmPFC related to the tendency to transfer value in adults. Taken together, these results contribute to a growing body of research in adolescent decision making, and extend upon our understanding of the mechanisms for learning and decision making systems, and how they change over development

Separate neural representations of prediction error valence and surprise: Evidence from an fMRI meta-analysis.

Learning occurs when an outcome differs from expectations, generating a reward prediction error signal (RPE). The RPE signal has been hypothesized to simultaneously embody the valence of an outcome (better or worse than expected) and its surprise (how far from expectations). Nonetheless, growing evidence suggests that separate representations of the two RPE components exist in the human brain. Meta-analyses provide an opportunity to test this hypothesis and directly probe the extent to which the valence and surprise of the error signal are encoded in separate or overlapping networks. We carried out several meta-analyses on a large set of fMRI studies investigating the neural basis of RPE, locked at decision outcome. We identified two valence learning systems by pooling studies searching for differential neural activity in response to categorical positive-versus-negative outcomes. The first valence network (negative > positive) involved areas regulating alertness and switching behaviours such as the midcingulate cortex, the thalamus and the dorsolateral prefrontal cortex whereas the second valence network (positive > negative) encompassed regions of the human reward circuitry such as the ventral striatum and the ventromedial prefrontal cortex. We also found evidence of a largely distinct surprise-encoding network including the anterior cingulate cortex, anterior insula and dorsal striatum. Together with recent animal and electrophysiological evidence this meta-analysis points to a sequential and distributed encoding of different components of the RPE signal, with potentially distinct functional roles

Computational Psychiatry: towards a mathematically informed understanding of mental illness

Computational Psychiatry aims to describe the relationship between the brain's neurobiology, its environment and mental symptoms in computational terms. In so doing, it may improve psychiatric classification and the diagnosis and treatment of mental illness. It can unite many levels of description in a mechanistic and rigorous fashion, while avoiding biological reductionism and artificial categorisation. We describe how computational models of cognition can infer the current state of the environment and weigh up future actions, and how these models provide new perspectives on two example disorders, depression and schizophrenia. Reinforcement learning describes how the brain can choose and value courses of actions according to their long-term future value. Some depressive symptoms may result from aberrant valuations, which could arise from prior beliefs about the loss of agency ('helplessness'), or from an inability to inhibit the mental exploration of aversive events. Predictive coding explains how the brain might perform Bayesian inference about the state of its environment by combining sensory data with prior beliefs, each weighted according to their certainty (or precision). Several cortical abnormalities in schizophrenia might reduce precision at higher levels of the inferential hierarchy, biasing inference towards sensory data and away from prior beliefs. We discuss whether striatal hyperdopaminergia might have an adaptive function in this context, and also how reinforcement learning and incentive salience models may shed light on the disorder. Finally, we review some of Computational Psychiatry's applications to neurological disorders, such as Parkinson's disease, and some pitfalls to avoid when applying its methods

Reinforcement magnitudes modulate subthalamic beta band activity in patients with Parkinson's disease

We set out to investigate whether beta oscillations in the human basal ganglia are modulated during reinforcement learning. Based on previous research, we assumed that beta activity might either reflect the magnitudes of individuals' received reinforcements (reinforcement hypothesis), their reinforcement prediction errors (dopamine hypothesis) or their tendencies to repeat versus adapt responses based upon reinforcements (status-quo hypothesis). We tested these hypotheses by recording local field potentials (LFPs) from the subthalamic nuclei of 19 Parkinson's disease patients engaged in a reinforcement-learning paradigm. We then correlated patients' reinforcement magnitudes, reinforcement prediction errors and response repetition tendencies with task-related power changes in their LFP oscillations. During feedback presentation, activity in the frequency range of 14 to 27 Hz (beta spectrum) correlated positively with reinforcement magnitudes. During responding, alpha and low beta activity (6 to 18 Hz) was negatively correlated with previous reinforcement magnitudes. Reinforcement prediction errors and response repetition tendencies did not correlate significantly with LFP oscillations. These results suggest that alpha and beta oscillations during reinforcement learning reflect patients' observed reinforcement magnitudes, rather than their reinforcement prediction errors or their tendencies to repeat versus adapt their responses, arguing both against an involvement of phasic dopamine and against applicability of the status-quo theory