Risk, Unexpected Uncertainty, and Estimation Uncertainty: Bayesian Learning in Unstable Settings

Recently, evidence has emerged that humans approach learning using Bayesian updating rather than (model-free) reinforcement algorithms in a six-arm restless bandit problem. Here, we investigate what this implies for human appreciation of uncertainty. In our task, a Bayesian learner distinguishes three equally salient levels of uncertainty. First, the Bayesian perceives irreducible uncertainty or risk: even knowing the payoff probabilities of a given arm, the outcome remains uncertain. Second, there is (parameter) estimation uncertainty or ambiguity: payoff probabilities are unknown and need to be estimated. Third, the outcome probabilities of the arms change: the sudden jumps are referred to as unexpected uncertainty. We document how the three levels of uncertainty evolved during the course of our experiment and how it affected the learning rate. We then zoom in on estimation uncertainty, which has been suggested to be a driving force in exploration, in spite of evidence of widespread aversion to ambiguity. Our data corroborate the latter. We discuss neural evidence that foreshadowed the ability of humans to distinguish between the three levels of uncertainty. Finally, we investigate the boundaries of human capacity to implement Bayesian learning. We repeat the experiment with different instructions, reflecting varying levels of structural uncertainty. Under this fourth notion of uncertainty, choices were no better explained by Bayesian updating than by (model-free) reinforcement learning. Exit questionnaires revealed that participants remained unaware of the presence of unexpected uncertainty and failed to acquire the right model with which to implement Bayesian updating

Pharmacogenetic Modulation of Orexin Neurons Alters Sleep/Wakefulness States in Mice

Hypothalamic neurons expressing neuropeptide orexins are critically involved in the control of sleep and wakefulness. Although the activity of orexin neurons is thought to be influenced by various neuronal input as well as humoral factors, the direct consequences of changes in the activity of these neurons in an intact animal are largely unknown. We therefore examined the effects of orexin neuron-specific pharmacogenetic modulation in vivo by a new method called the Designer Receptors Exclusively Activated by Designer Drugs approach (DREADD). Using this system, we successfully activated and suppressed orexin neurons as measured by Fos staining. EEG and EMG recordings suggested that excitation of orexin neurons significantly increased the amount of time spent in wakefulness and decreased both non-rapid eye movement (NREM) and rapid eye movement (REM) sleep times. Inhibition of orexin neurons decreased wakefulness time and increased NREM sleep time. These findings clearly show that changes in the activity of orexin neurons can alter the behavioral state of animals and also validate this novel approach for manipulating neuronal activity in awake, freely-moving animals

Brainstem and Spinal Cord Circuitry Regulating REM Sleep and Muscle Atonia

Previous work has suggested, but not demonstrated directly, a critical role for both glutamatergic and GABAergic neurons of the pontine tegmentum in the regulation of rapid eye movement (REM) sleep.To determine the in vivo roles of these fast-acting neurotransmitters in putative REM pontine circuits, we injected an adeno-associated viral vector expressing Cre recombinase (AAV-Cre) into mice harboring lox-P modified alleles of either the vesicular glutamate transporter 2 (VGLUT2) or vesicular GABA-glycine transporter (VGAT) genes. Our results show that glutamatergic neurons of the sublaterodorsal nucleus (SLD) and glycinergic/GABAergic interneurons of the spinal ventral horn contribute to REM atonia, whereas a separate population of glutamatergic neurons in the caudal laterodorsal tegmental nucleus (cLDT) and SLD are important for REM sleep generation. Our results further suggest that presynaptic GABA release in the cLDT-SLD, ventrolateral periaqueductal gray matter (vlPAG) and lateral pontine tegmentum (LPT) are not critically involved in REM sleep control.These findings reveal the critical and divergent in vivo role of pontine glutamate and spinal cord GABA/glycine in the regulation of REM sleep and atonia and suggest a possible etiological basis for REM sleep behavior disorder (RBD)

The waking brain: an update

Wakefulness and consciousness depend on perturbation of the cortical soliloquy. Ascending activation of the cerebral cortex is characteristic for both waking and paradoxical (REM) sleep. These evolutionary conserved activating systems build a network in the brainstem, midbrain, and diencephalon that contains the neurotransmitters and neuromodulators glutamate, histamine, acetylcholine, the catecholamines, serotonin, and some neuropeptides orchestrating the different behavioral states. Inhibition of these waking systems by GABAergic neurons allows sleep. Over the past decades, a prominent role became evident for the histaminergic and the orexinergic neurons as a hypothalamic waking center

Learning and generalization under ambiguity: an fMRI study.

Adaptive behavior often exploits generalizations from past experience by applying them judiciously in new situations. This requires a means of quantifying the relative importance of prior experience and current information, so they can be balanced optimally. In this study, we ask whether the brain generalizes in an optimal way. Specifically, we used Bayesian learning theory and fMRI to test whether neuronal responses reflect context-sensitive changes in ambiguity or uncertainty about experience-dependent beliefs. We found that the hippocampus expresses clear ambiguity-dependent responses that are associated with an augmented rate of learning. These findings suggest candidate neuronal systems that may be involved in aberrations of generalization, such as over-confidence

Waking selective neurons in the posterior hypothalamus and their response to histamine H3-receptor ligands: an electrophysiological study in freely moving cats.

International audienceNeurons which discharge selectively during waking (waking selective) have been found in the tuberomamillary nucleus (TM) and adjacent areas of the posterior hypothalamus. Although they share some electrophysiological properties with aminergic neurons, there is no direct evidence that they are histaminergic. We have recorded from posterior hypothalamic neurons during the sleep-wake cycle in freely moving cats, and investigated the effects on waking selective neurons of specific ligands of histaminergic H3-receptors, which autoregulate the activity of histaminergic neurons. Two types of neurons were seen. Waking selective neurons, termed "waking-on (W-on)," were located exclusively within the TM and adjacent areas, and discharged at a low regular rate during waking (1.71-2.97 Hz), decreased firing during light slow wave sleep (SWS), became silent during deep SWS and paradoxical sleep (PS) and resumed their activity on, or a few seconds before, awakening. "Waking-related" neurons, located in an area dorsal to the TM, displayed a similar, although less regular, low rate of firing (1.74-5.41 Hz) and a similar discharge profile during the sleep-wake cycle; however, unlike "W-on" neurons, they did not completely stop firing during deep SWS and PS. Intramuscular (i.m.) injection of ciproxifan (an H3-receptor antagonist, 1mg/kg), significantly increased the discharge rate of W-on neurons and induced c-fos expression in histamine-immunoreactive neurons, whereas i.m. injection of imetit (an H3-receptor agonist, 1mg/kg) or microinjection of alpha-methylhistamine (another H3-receptor agonist, 0.025-0.1 microg/0.2 microl) in the vicinity of these cells significantly decreased their discharge rate. Moreover, the effect of the antagonist was reversed by the agonists and vice versa. In contrast, "waking-related" neurons were unaffected by these H3-receptor ligands. These data provide evidence for the histaminergic nature of "W-on" neurons and their role in cortical desynchronization during waking, and highlight the heterogeneity of posterior hypothalamic neuronal populations, which might serve different functions during the wakefulness

The Hippocampus Codes the Uncertainty of Cue-Outcome Associations: An Intracranial Electrophysiological Study in Humans

International audienceLearning to predict upcoming outcomes based on environmental cues is essential for adaptative behavior. In monkeys, midbrain dopaminergic neurons code two statistical properties of reward: a prediction error at the outcome and uncertainty during the delay period between cues and outcomes. Although the hippocampus is sensitive to reward processing, and hippocampal–midbrain functional interactions are well documented, it is unknown whether it also codes the statistical properties of reward information. To address this question, we recorded local field potentials from intracranial electrodes in human hippocampus while subjects learned to associate cues of slot machines with various monetary reward probabilities (P). We found that the amplitudes of negative event-related potentials covaried with uncertainty at the outcome, being maximal for P = 0.5 and minimal for P = 0 and P = 1, regardless of winning or not. These results show that the hippocampus computes an uncertainty signal that may constitute a fundamental mechanism underlying the role of this brain region in a number of functions, including attention-based learning, associative learning, probabilistic classification, and binding of stimulus elements

The neural dynamics of reward value and risk coding in the human orbitofrontal cortex

International audienceThe orbitofrontal cortex is known to carry information regarding expected reward, risk and experienced outcome. Yet, due to inherent limitations in lesion and neuroimaging methods, the neural dynamics of these computations has remained elusive in humans. Here, taking advantage of the high temporal definition of intracranial recordings, we characterize the neurophysiological signatures of the intact orbitofrontal cortex in processing information relevant for risky decisions. Local field potentials were recorded from the intact orbitofrontal cortex of patients suffering from drug-refractory partial epilepsy with implanted depth electrodes as they performed a probabilistic reward learning task that required them to associate visual cues with distinct reward probabilities. We observed three successive signals: (i) around 400 ms after cue presentation, the amplitudes of the local field potentials increased with reward probability; (ii) a risk signal emerged during the late phase of reward anticipation and during the outcome phase; and (iii) an experienced value signal appeared at the time of reward delivery. Both the medial and lateral orbitofrontal cortex encoded risk and reward probability while the lateral orbitofrontal cortex played a dominant role in coding experienced value. The present study provides the first evidence from intracranial recordings that the human orbitofrontal cortex codes reward risk both during late reward anticipation and during the outcome phase at a time scale of milliseconds. Our findings offer insights into the rapid mechanisms underlying the ability to learn structural relationships from the environment