Risk, Unexpected Uncertainty, and Estimation Uncertainty: Bayesian Learning in Unstable Settings

Recently, evidence has emerged that humans approach learning using Bayesian updating rather than (model-free) reinforcement algorithms in a six-arm restless bandit problem. Here, we investigate what this implies for human appreciation of uncertainty. In our task, a Bayesian learner distinguishes three equally salient levels of uncertainty. First, the Bayesian perceives irreducible uncertainty or risk: even knowing the payoff probabilities of a given arm, the outcome remains uncertain. Second, there is (parameter) estimation uncertainty or ambiguity: payoff probabilities are unknown and need to be estimated. Third, the outcome probabilities of the arms change: the sudden jumps are referred to as unexpected uncertainty. We document how the three levels of uncertainty evolved during the course of our experiment and how it affected the learning rate. We then zoom in on estimation uncertainty, which has been suggested to be a driving force in exploration, in spite of evidence of widespread aversion to ambiguity. Our data corroborate the latter. We discuss neural evidence that foreshadowed the ability of humans to distinguish between the three levels of uncertainty. Finally, we investigate the boundaries of human capacity to implement Bayesian learning. We repeat the experiment with different instructions, reflecting varying levels of structural uncertainty. Under this fourth notion of uncertainty, choices were no better explained by Bayesian updating than by (model-free) reinforcement learning. Exit questionnaires revealed that participants remained unaware of the presence of unexpected uncertainty and failed to acquire the right model with which to implement Bayesian updating

Pharmacogenetic Modulation of Orexin Neurons Alters Sleep/Wakefulness States in Mice

Hypothalamic neurons expressing neuropeptide orexins are critically involved in the control of sleep and wakefulness. Although the activity of orexin neurons is thought to be influenced by various neuronal input as well as humoral factors, the direct consequences of changes in the activity of these neurons in an intact animal are largely unknown. We therefore examined the effects of orexin neuron-specific pharmacogenetic modulation in vivo by a new method called the Designer Receptors Exclusively Activated by Designer Drugs approach (DREADD). Using this system, we successfully activated and suppressed orexin neurons as measured by Fos staining. EEG and EMG recordings suggested that excitation of orexin neurons significantly increased the amount of time spent in wakefulness and decreased both non-rapid eye movement (NREM) and rapid eye movement (REM) sleep times. Inhibition of orexin neurons decreased wakefulness time and increased NREM sleep time. These findings clearly show that changes in the activity of orexin neurons can alter the behavioral state of animals and also validate this novel approach for manipulating neuronal activity in awake, freely-moving animals

Brainstem and Spinal Cord Circuitry Regulating REM Sleep and Muscle Atonia

Previous work has suggested, but not demonstrated directly, a critical role for both glutamatergic and GABAergic neurons of the pontine tegmentum in the regulation of rapid eye movement (REM) sleep.To determine the in vivo roles of these fast-acting neurotransmitters in putative REM pontine circuits, we injected an adeno-associated viral vector expressing Cre recombinase (AAV-Cre) into mice harboring lox-P modified alleles of either the vesicular glutamate transporter 2 (VGLUT2) or vesicular GABA-glycine transporter (VGAT) genes. Our results show that glutamatergic neurons of the sublaterodorsal nucleus (SLD) and glycinergic/GABAergic interneurons of the spinal ventral horn contribute to REM atonia, whereas a separate population of glutamatergic neurons in the caudal laterodorsal tegmental nucleus (cLDT) and SLD are important for REM sleep generation. Our results further suggest that presynaptic GABA release in the cLDT-SLD, ventrolateral periaqueductal gray matter (vlPAG) and lateral pontine tegmentum (LPT) are not critically involved in REM sleep control.These findings reveal the critical and divergent in vivo role of pontine glutamate and spinal cord GABA/glycine in the regulation of REM sleep and atonia and suggest a possible etiological basis for REM sleep behavior disorder (RBD)

The waking brain: an update

Wakefulness and consciousness depend on perturbation of the cortical soliloquy. Ascending activation of the cerebral cortex is characteristic for both waking and paradoxical (REM) sleep. These evolutionary conserved activating systems build a network in the brainstem, midbrain, and diencephalon that contains the neurotransmitters and neuromodulators glutamate, histamine, acetylcholine, the catecholamines, serotonin, and some neuropeptides orchestrating the different behavioral states. Inhibition of these waking systems by GABAergic neurons allows sleep. Over the past decades, a prominent role became evident for the histaminergic and the orexinergic neurons as a hypothalamic waking center

Learning and generalization under ambiguity: an fMRI study.

Adaptive behavior often exploits generalizations from past experience by applying them judiciously in new situations. This requires a means of quantifying the relative importance of prior experience and current information, so they can be balanced optimally. In this study, we ask whether the brain generalizes in an optimal way. Specifically, we used Bayesian learning theory and fMRI to test whether neuronal responses reflect context-sensitive changes in ambiguity or uncertainty about experience-dependent beliefs. We found that the hippocampus expresses clear ambiguity-dependent responses that are associated with an augmented rate of learning. These findings suggest candidate neuronal systems that may be involved in aberrations of generalization, such as over-confidence

Sleep-waking discharge patterns of neurons recorded in the rat perifornical lateral hypothalamic area

The perifornical lateral hypothalamic area (PF-LHA) has been implicated in the control of several waking behaviours, including feeding, motor activity and arousal. Several cell types are located in the PF-LHA, including projection neurons that contain the hypocretin peptides (also known as orexins). Recent findings suggest that hypocretin neurons are involved in sleep-wake regulation. Loss of hypocretin neurons in the human disorder narcolepsy is associated with excessive somnolence, cataplexy and increased propensity for rapid eye movement (REM) sleep. However, the relationship of PF-LHA neuronal activity to different arousal states is unknown. We recorded neuronal activity in the PF-LHA of rats during natural sleep and waking. Neuronal discharge rates were calculated during active waking (waking accompanied by movement), quiet waking, non-REM sleep and REM sleep. Fifty-six of 106 neurons (53 %) were classified as wake/REM-related. These neurons exhibited peak discharge rates during waking and REM sleep and reduced discharge rates during non-REM sleep. Wake-related neurons (38 %) exhibited reduced discharge rates during both non-REM and REM sleep when compared to that during waking. Wake-related neurons exhibited significantly higher discharge rates during active waking than during quiet waking. The discharge of wake-related neurons was positively correlated with muscle activity across all sleep-waking states. Recording sites were located within the hypocretin-immunoreactive neuronal field of the PF-LHA. Although the neurotransmitter phenotype of recorded cells was not determined, the prevalence of neurons with wake-related discharge patterns is consistent with the hypothesis that the PF-LHA participates in the regulation of arousal, muscle activity and sleep-waking states