Mesolimbic confidence signals guide perceptual learning in the absence of external feedback

It is well established that learning can occur without external feedback, yet normative reinforcement learning theories have difficulties explaining such instances of learning. Here, we propose that human observers are capable of generating their own feedback signals by monitoring internal decision variables. We investigated this hypothesis in a visual perceptual learning task using fMRI and confidence reports as a measure for this monitoring process. Employing a novel computational model in which learning is guided by confidence-based reinforcement signals, we found that mesolimbic brain areas encoded both anticipation and prediction error of confidence—in remarkable similarity to previous findings for external reward-based feedback. We demonstrate that the model accounts for choice and confidence reports and show that the mesolimbic confidence prediction error modulation derived through the model predicts individual learning success. These results provide a mechanistic neurobiological explanation for learning without external feedback by augmenting reinforcement models with confidence-based feedback

Independent circuits in basal ganglia and cortex for the processing of reward and precision feedback

In order to understand human decision making it is necessary to understand how the brain uses feedback to guide goal-directed behavior. The ventral striatum (VS) appears to be a key structure in this function, responding strongly to explicit reward feedback. However, recent results have also shown striatal activity following correct task performance even in the absence of feedback. This raises the possibility that, in addition to processing external feedback, the dopamine-centered reward circuit might regulate endogenous reinforcement signals, like those triggered by satisfaction in accurate task performance. Here we use functional magnetic resonance imaging (fMRI) to test this idea. Participants completed a simple task that garnered both reward feedback and feedback about the precision of performance. Importantly, the design was such that we could manipulate information about the precision of performance within different levels of reward magnitude. Using parametric modulation and functional connectivity analysis we identified brain regions sensitive to each of these signals. Our results show a double dissociation: frontal and posterior cingulate regions responded to explicit reward but were insensitive to task precision, whereas the dorsal striatum - and putamen in particular - was insensitive to reward but responded strongly to precision feedback in reward-present trials. Both types of feedback activated the VS, and sensitivity in this structure to precision feedback was predicted by personality traits related to approach behavior and reward responsiveness. Our findings shed new light on the role of specific brain regions in integrating different sources of feedback to guide goal-directed behavior

Role Of The Dorsal Striatum In Learning and Decision Making

The striatum, the input region of the basal ganglia, has been shown to mediate many cognitive functions. The striatum itself can be functionally segregated into dorsal (DS) and ventral striatum (VS). For more than 60 years, DS has been reported to mediate stimulus-response learning, though evidence has been accruing pointing to a role in decision making. These literatures have been growing independently and an aim of this thesis was to bridge these two bodies of knowledge. We directly investigated the role of DS in stimulus-response learning versus decision making using functional magnetic resonance imaging (fMRI) in patients with Parkinson’s disease (Chapter 2) and obsessive compulsive disorder (Chapter 3). In Chapter 4, the role of DS in stimulus-response habit learning was tested in healthy individuals using fMRI. In three separate experiments (Chapters 2-4), all of the results strongly support the notion that DS mediates decision making and not learning. DS is implicated in many disorders ranging from Parkinson’s disease, obsessive compulsive disorder and addiction, and clarifying the role of DS in cognitive function is paramount for understanding substrates of disease and developing treatments

Functional organisation of behavioural inhibitory control mechanisms in cortico-basal ganglia circuitry: implications for stimulant use disorder.

The neural and psychological mechanisms of inhibitory control processes were investigated, focusing on the cortico-basal ganglia circuits in rats and humans. These included behavioural flexibility, ‘waiting’ and ‘stopping’ impulsivity and involved serial spatial reversal learning task in rodents, and in humans, premature responses in the Monetary Incentive Delay (MID) task and the stop-signal reaction time task. Chapter 2 and Chapter 3 focus on individual differences in behavioural flexibility in rats while Chapter 4, Chapter 5 and Chapter 6 consider how inhibitory control mechanisms are affected by the psychostimulant drug cocaine in both rats and humans. As reported in Chapter 2, systemic modulation of monoaminergic transmission by monoamine oxidase A (MAO-A) inhibitors enhanced reversal learning performance, selectively by decreasing the lose-shift probability, thereby implicating a role for dopamine, serotonin and noradrenaline in facilitating learning from negative feedback. Resting state functional magnetic resonance imaging (fMRI) revealed enhanced functional connectivity of the orbitofrontal and motor cortices as a correlate of flexible reversal learning performance, consistent with elevated levels of monoamines in these region (Chapter 3). Having clarified the mechanisms underlying behavioural flexibility in rats, Chapter 4 reports that escalation of intravenous cocaine self-administration induces behavioural inflexibility in rats even after a relatively short period of cocaine intake. Computational models, including a reinforced and Bayesian learner, revealed a lack of exploitation of the learned response-outcome relationships in cocaine-exposed rats. Chapter 5 focused on impulse control in human volunteers, identifying the striatal and cingulo-opercular networks as substrates of impulsive, premature responding in healthy 4 volunteers, stimulant-dependent individuals and their unaffected siblings. Loss of impulse control was elicited by different incentives for drug-free participants as opposed to drug users. Drug cues elicited striatal activation and increased premature responses in the stimulant-dependent group compared with the control group. In contrast, the ventral striatum was linked to incentive specific activation to reward anticipation. Task-based fMRI demonstrated that interactions between dorsal striatum and cingulo-opercular “cold cognition” networks underlie failures of impulse control in the control, at-risk and stimulant-dependent groups. However, whereas the cingulo-opercular networks were associated with premature responding in all groups, the reward system was activated specifically by the drug incentive cues in the stimulant group, and by monetary incentive cues in the drug-free groups. Chapter 6 presents evidence that corticostriatal functional and effective connectivity in an overlapping network that includes the anterior cingulate and inferior frontal cortices as well as motor cortex, the subthalamic nucleus and dorsal striatum, is critical to stopping impulse control in both control and cocaine individuals. No stopping efficiency impairments were observed in the cocaine-dependent group. Nevertheless, lower structural corticostriatal connectivity measured using diffusion MRI was associated with response execution impairments in cocaine participants performing a stop-signal reaction time task. Further, response execution was rescued by the selective noradrenaline reuptake inhibitor atomoxetine, which also increased corticostriatal effective connectivity. Finally, increased impulsivity and behavioural inflexibility seen in stimulant use disorder in Chapter 5 and Chapter 4, respectively, were not observed in the endophenotype at risk for developing stimulant abuse but were rather a consequence of stimulant abuse. These results further clarify the monoaminergic substrates of behavioural flexibility and specify the neural and computational impairments in inhibitory control induced by stimulant dependence.Pinsent Darwin Studentship from the Dept of Physiology, Development and Neuroscienc

Extending the concept of emotion regulation with model-based fMRI

Effective emotion regulation is essential for our social and emotional well-being. Yet, the concept of emotion regulation, as it is conventionally regarded in the field, does not take important aspects of emotions and emotion regulation into account. The overarching aim of the current thesis was to include such missing aspects and thereby expand the concept of emotion regulation. The expansion occurred in two directions: firstly, the definition of emotion within the field of emotion regulation was widened to include the motivational aspect of emotions in terms of value-based prediction errors and their neural implementation; and secondly, an underestimated type of emotion regulation – the social emotion regulation – and its neural underpinnings were investigated. Projects 1 and 2 of the current thesis expand the emotion part of emotion regulation. Project 1 investigated whether emotion regulation affects not only emotional response-related brain activity but also influences aversive prediction error-related activity, i.e., the motivation-related brain signal. We found that self- initiated reappraisal, a type of cognitive emotion regulation, indeed affected prediction error-related activity, such that this activity was enhanced in the ventral tegmental area, ventral striatum, insula and hippocampus, possibly via a prefrontal-tegmental pathway. Project 2 further examined the way emotion regulation affects emotions and prediction errors, by testing whether self- initiated reappraisal directly targets the brain network for motivated behaviour previously outlined by animal studies. We found that superior (in contrast to inferior) regulators affected the balance of competing influences of ventral striatal afferents on striatal aversive prediction error signals; they reduced the impact of subcortical striatal afferents (i.e., hippocampus, amygdala and ventral tegmental area), while keeping the influence of the prefrontal cortex on ventral striatal prediction errors constant. Inferior regulators, on the other hand, failed to supress subcortical inputs into the ventral striatum and instead counterproductively reduced the prefrontal influence on ventral striatal prediction error signals. Projects 3 and 4 of the thesis extend the regulation part of emotion regulation. Project 3 explored the neural correlates of social cognitive emotion regulation, specifically reappraisal, and directly compared them with those of self-initiated reappraisal. We found that regions of the anterior, the medial parietal, and the lateral temporo-parietal default mode network were specifically involved in social emotion regulation, and that social regulation success and the default mode network involvement during regulation were related to participants’ attachment security scores. Project 4 investigated social emotion modulation and its impact on two distinct types of emotional brain activity – emotional response- and aversive prediction error-related activity. We found – for the simple contrast of being with somebody versus being alone – a three-fold dissociation between signal types and insula subregions, including left and right anterior and posterior insula parts. Social emotion modulation reduced aversive stimulus-related activity in the posterior insula, while simultaneously increasing aversive prediction error-related activity in the anterior insula. Furthermore, the social effect on prediction error-related activity was positively associated with aversive learning in the right, but negatively in the left anterior insula. Altogether, by expanding the concept of emotion regulation, projects of the current thesis provide new insights into both the effects and the neural underpinnings of three distinct emotion regulation types. Considering that problems in both intrapersonal emotion regulation and social interaction are linked to affective disorders, our findings might contribute to a better understanding of these disorders and the disorder-specific emotional and social impairments

Analysis of individual differences in neurofeedback training illuminates successful self-regulation of the dopaminergic midbrain

The dopaminergic midbrain is associated with reinforcement learning, motivation and decision-making – functions often disturbed in neuropsychiatric disorders. Previous research has shown that dopaminergic midbrain activity can be endogenously modulated via neurofeedback. However, the robustness of endogenous modulation, a requirement for clinical translation, is unclear. Here, we examine whether the activation of particular brain regions associates with successful regulation transfer when feedback is no longer available. Moreover, to elucidate mechanisms underlying effective self-regulation, we study the relation of successful transfer with learning (temporal difference coding) outside the midbrain during neurofeedback training and with individual reward sensitivity in a monetary incentive delay (MID) task. Fifty-nine participants underwent neurofeedback training either in standard (Study 1 N = 15, Study 2 N = 28) or control feedback group (Study 1, N = 16). We find that successful self-regulation is associated with prefrontal reward sensitivity in the MID task (N = 25), with a decreasing relation between prefrontal activity and midbrain learning signals during neurofeedback training and with increased activity within cognitive control areas during transfer. The association between midbrain self-regulation and prefrontal temporal difference and reward sensitivity suggests that reinforcement learning contributes to successful self-regulation. Our findings provide insights in the control of midbrain activity and may facilitate individually tailoring neurofeedback training

Action control in uncertain environments

A long-standing dichotomy in neuroscience pits automatic or reflexive drivers of behaviour against deliberate or reflective processes. In this thesis I explore how this concept applies to two stages of action control: decision-making and response inhibition. The first part of this thesis examines the decision-making process itself during which actions need to be selected that maximise rewards. Decisions arise through influences from model-free stimulus-response associations as well as model-based, goal-directed thought. Using a task that quantifies their respective contributions, I describe three studies that manipulate the balance of control between these two systems. I find that a pharmacological manipulation with levodopa increases model-based control without affecting model-free function; disruption of dorsolateral prefrontal cortex via magnetic stimulation disrupts model-based control; and direct current stimulation to the same prefrontal region has no effect on decision-making. I then examine how the intricate anatomy of frontostriatal circuits subserves reinforcement learning using functional, structural and diffusion magnetic resonance imaging (MRI). A second stage of action control discussed in this thesis is post-decision monitoring and adjustment of action. Specifically, I develop a response inhibition task that dissociates reactive, bottom-up inhibitory control from proactive, top-down forms of inhibition. Using functional MRI I show that, unlike the strong neural segregation in decision-making systems, neural mechanisms of reactive and proactive response inhibition overlap to a great extent in their frontostriatal circuitry. This leads to the hypothesis that neural decline, for 4 example in the context of ageing, might affect reactive and proactive control similarly. I test this in a large population study administered through a smartphone app. This shows that, against my prediction, reactive control reliably declines with age but proactive control shows no such decline. Furthermore, in line with data on gender differences in age-related neural degradation, reactive control in men declines faster with age than that of women

Neurocomputational models of corticostriatal interactions in action selection

Schema theory is a framework based on the idea that behaviour in many areas depends on abstractions over instances called schemas, which work in a cooperative or sequential fashion, but also compete with each other for activation. Cooper & Shallice (2000) provide an implementation of schema-theory with their model that simulates how routine actions works in healthy and neurologically-impaired populations. While schema theory is helpful in representing functional interactions in the action-perception cycle, it has no commitment to a specific neural implementation. Redgrave et al.’s (2001) model of the basal ganglia is, in principle, compatible with a device that regulates the competition among schemas, carrying out action selection. This thesis is mainly concerned with improving the neurobiological plausibility of the schema theoretic account of action selection without sacrificing its theoretical underpinning. We therefore start by combining an implementation of schema-theory with a reparametrised version of the original basal ganglia model, building the model from the ground up. The model simulates two widely used neuropsychological tasks, the Wisconsin Card Sorting Test (WCST), and the Brixton Task (BRX). In order to validate the model, we then present a study with 25 younger and 25 over-60 individuals performing the WCST and BRX, and we simulate their performance using the schema-theoretic basal ganglia model. Experimental results indicate a dissociation between loss of representation (present in older adults) and perseveration of response (absent in older adults) in the WCST, and the model fits adequately simulate these findings while grounding the interpretation of parameters to the neurobiology of aging. We subsequently present a further study with 50 participants, 14 of whom have an ADHD diagnosis, performing the WCST under an untimed and a timed condition, and we then use our model to fit response time. Results indicate that impulsivity traits, but not inattention ones, predict a slower tail of responses in the untimed task and an increased number of missed responses and variability across subtasks. Using the model, we show that these results can be produced by variation of a combination of two parameters representing basal ganglia activity and top-down excitation. We conclude with recommendations on how to improve and extend the model