The Role of the Noradrenergic System in the Exploration–Exploitation Trade-Off: A Psychopharmacological Study

Animal research and computational modeling have indicated an important role for the neuromodulatory locus coeruleus–norepinephrine (LC–NE) system in the control of behavior. According to the adaptive gain theory, the LC–NE system is critical for optimizing behavioral performance by regulating the balance between exploitative and exploratory control states. However, crucial direct empirical tests of this theory in human subjects have been lacking. We used a pharmacological manipulation of the LC–NE system to test predictions of this theory in humans. In a double-blind parallel-groups design (N = 52), participants received 4 mg reboxetine (a selective norepinephrine reuptake inhibitor), 30 mg citalopram (a selective serotonin reuptake inhibitor), or placebo. The adaptive gain theory predicted that the increased tonic NE levels induced by reboxetine would promote task disengagement and exploratory behavior. We assessed the effects of reboxetine on performance in two cognitive tasks designed to examine task (dis)engagement and exploitative versus exploratory behavior: a diminishing-utility task and a gambling task with a non-stationary pay-off structure. In contrast to predictions of the adaptive gain theory, we did not find differences in task (dis)engagement or exploratory behavior between the three experimental groups, despite demonstrable effects of the two drugs on non-specific central and autonomic nervous system parameters. Our findings suggest that the LC–NE system may not be involved in the regulation of the exploration–exploitation trade-off in humans, at least not within the context of a single task. It remains to be examined whether the LC–NE system is involved in random exploration exceeding the current task context

What is in the feedback? Effect of induced happiness vs. sadness on probabilistic learning with vs. without exploration

According to dominant neuropsychological theories of affect, emotions signal salience of events and in turn facilitate a wide spectrum of response options or action tendencies. Valence of an emotional experience is pivotal here, as it alters reward and punishment processing, as well as the balance between safety and risk taking, which can be translated into changes in the exploration exploitation trade-off during reinforcement learning (RL). To test this idea, we compared the behavioral performance of three groups of participants that all completed a variant of a standard probabilistic learning task, but who differed regarding which mood state was actually induced and maintained (happy, sad or neutral). To foster a change from an exploration to an exploitation based mode, we removed feedback information once learning was reliably established. Although changes in mood were successful, learning performance was balanced between the three groups. Critically, when focusing on exploitation-driven learning only, they did not differ either. Moreover, mood valence did not alter the learning rate or exploration per se, when titrated using complementing computational modeling. By comparing systematically these results to our previous study (Bakic et al., 2014), we found that arousal levels did differ between studies, which might account for limited modulatory effects of (positive) mood on RL in the present case. These results challenge the assumption that mood valence alone is enough to create strong shifts in the way exploitation or exploration is eventually carried out during (probabilistic) learning. In this context, we discuss the possibility that both valence and arousal are actually necessary components of the emotional mood state to yield changes in the use and exploration of incentives cues during RL

Different brain networks mediate the effects of social and conditioned expectations on pain.

Information about others' experiences can strongly influence our own feelings and decisions. But how does such social information affect the neural generation of affective experience, and are the brain mechanisms involved distinct from those that mediate other types of expectation effects? Here, we used fMRI to dissociate the brain mediators of social influence and associative learning effects on pain. Participants viewed symbolic depictions of other participants' pain ratings (social information) and classically conditioned pain-predictive cues before experiencing painful heat. Social information and conditioned stimuli each had significant effects on pain ratings, and both effects were mediated by self-reported expectations. Yet, these effects were mediated by largely separable brain activity patterns, involving different large-scale functional networks. These results show that learned versus socially instructed expectations modulate pain via partially different mechanisms-a distinction that should be accounted for by theories of predictive coding and related top-down influences

Group-regularized individual prediction: theory and application to pain

Multivariate pattern analysis (MVPA) has become an important tool for identifying brain representations of psychological processes and clinical outcomes using fMRI and related methods. Such methods can be used to predict or ‘decode’ psychological states in individual subjects. Single-subject MVPA approaches, however, are limited by the amount and quality of individual-subject data. In spite of higher spatial resolution, predictive accuracy from single-subject data often does not exceed what can be accomplished using coarser, group-level maps, because single-subject patterns are trained on limited amounts of often-noisy data. Here, we present a method that combines population-level priors, in the form of biomarker patterns developed on prior samples, with single-subject MVPA maps to improve single-subject prediction. Theoretical results and simulations motivate a weighting based on the relative variances of biomarker-based prediction—based on population-level predictive maps from prior groups—and individual-subject, cross-validated prediction. Empirical results predicting pain using brain activity on a trial-by-trial basis (single-trial prediction) across 6 studies (N = 180 participants) confirm the theoretical predictions. Regularization based on a population-level biomarker—in this case, the Neurologic Pain Signature (NPS)—improved single-subject prediction accuracy compared with idiographic maps based on the individuals' data alone. The regularization scheme that we propose, which we term group-regularized individual prediction (GRIP), can be applied broadly to within-person MVPA-based prediction. We also show how GRIP can be used to evaluate data quality and provide benchmarks for the appropriateness of population-level maps like the NPS for a given individual or study

Colours sometimes count: Awareness and bidirectionality in grapheme-colour synaesthesia

Three experiments were conducted with 10 grapheme-colour synaesthetes and 10 matched controls to investigate (a) whether awareness of the inducer grapheme is necessary for synaesthetic colour induction and (b) whether grapheme-colour synaesthesia may be bidirectional in the sense that not only do graphemes induce colours, but that colours influence the processing of graphemes. Using attentional blink and Stroop paradigms with digit targets, we found that some synaesthetes did report "seeing" synaesthetic colours even when they were not able to report the inducing digit. Moreover, congruency effects (effects of matching the colour of digit presentation with the synaesthetic colour associated with that digit) suggested that grapheme-colour synaesthesia can be bidirectional, at least for some synaesthetes

Why does the marshmallow test predict later life outcomes? A commentary on Michaelson and Munakata (2020)

Overall, Munakata and Michaelson (2020) provide novel and provocative findings on the predictive validity of the preschool marshmallow test on the one hand, and the importance of social vs. cognitive (self-control) predictors of later life success on the other hand. Regarding the question of why the marshmallow test predicts later life success, we argue that—in contrast to the authors’ conclusion—their results suggest that self-control provides a better explanation than social support

Conceptual Conditioning

Classical conditioning can profoundly modify subsequent pain responses, but the mechanisms driving this effect are unresolved. Pain-conditioning studies typically condition cues to primary aversive reinforcers; hence, subsequent pain modulation could reflect learned pre-cognitive associations and/or expectancies that are conceptual in nature. We isolated conceptual contributions using a thermal pain-conditioning procedure in which different cues (CS(high) and CS(low)) were repeatedly paired with symbolic representations of high and low noxious heat, respectively. In a subsequent test phase, identical noxious stimuli evoked larger skin-conductance responses (SCRs) and pain ratings when preceded by CS(high) than CS(low) cues. These effects were mediated by participants’ self-reported expectancies. CS(high) cues also evoked larger anticipatory SCRs, but larger anticipatory SCRs predicted smaller subsequent heat-evoked SCRs. These results provide novel evidence that conditioned modulation of pain physiology can be acquired through purely conceptual processes, and that self-reported expectancies and physiological threat responses have opposing effects on pain