Reinstated episodic context guides sampling-based decisions for reward.

How does experience inform decisions? In episodic sampling, decisions are guided by a few episodic memories of past choices. This process can yield choice patterns similar to model-free reinforcement learning; however, samples can vary from trial to trial, causing decisions to vary. Here we show that context retrieved during episodic sampling can cause choice behavior to deviate sharply from the predictions of reinforcement learning. Specifically, we show that, when a given memory is sampled, choices (in the present) are influenced by the properties of other decisions made in the same context as the sampled event. This effect is mediated by fMRI measures of context retrieval on each trial, suggesting a mechanism whereby cues trigger retrieval of context, which then triggers retrieval of other decisions from that context. This result establishes a new avenue by which experience can guide choice and, as such, has broad implications for the study of decisions

Compositional inductive biases in function learning.

How do people recognize and learn about complex functional structure? Taking inspiration from other areas of cognitive science, we propose that this is achieved by harnessing compositionality: complex structure is decomposed into simpler building blocks. We formalize this idea within the framework of Bayesian regression using a grammar over Gaussian process kernels, and compare this approach with other structure learning approaches. Participants consistently chose compositional (over non-compositional) extrapolations and interpolations of functions. Experiments designed to elicit priors over functional patterns revealed an inductive bias for compositional structure. Compositional functions were perceived as subjectively more predictable than non-compositional functions, and exhibited other signatures of predictability, such as enhanced memorability and reduced numerosity. Taken together, these results support the view that the human intuitive theory of functions is inherently compositional

Seeing Tree Structure from Vibration

Humans recognize object structure from both their appearance and motion; often, motion helps to resolve ambiguities in object structure that arise when we observe object appearance only. There are particular scenarios, however, where neither appearance nor spatial-temporal motion signals are informative: occluding twigs may look connected and have almost identical movements, though they belong to different, possibly disconnected branches. We propose to tackle this problem through spectrum analysis of motion signals, because vibrations of disconnected branches, though visually similar, often have distinctive natural frequencies. We propose a novel formulation of tree structure based on a physics-based link model, and validate its effectiveness by theoretical analysis, numerical simulation, and empirical experiments. With this formulation, we use nonparametric Bayesian inference to reconstruct tree structure from both spectral vibration signals and appearance cues. Our model performs well in recognizing hierarchical tree structure from real-world videos of trees and vessels.Comment: ECCV 2018. The first two authors contributed equally to this work. Project page: http://tree.csail.mit.edu

Retrospective model-based inference guides model-free credit assignment

An extensive reinforcement learning literature shows that organisms assign credit efficiently, even under conditions of state uncertainty. However, little is known about credit-assignment when state uncertainty is subsequently resolved. Here, we address this problem within the framework of an interaction between model-free (MF) and model-based (MB) control systems. We present and support experimentally a theory of MB retrospective-inference. Within this framework, a MB system resolves uncertainty that prevailed when actions were taken thus guiding an MF credit-assignment. Using a task in which there was initial uncertainty about the lotteries that were chosen, we found that when participants’ momentary uncertainty about which lottery had generated an outcome was resolved by provision of subsequent information, participants preferentially assigned credit within a MF system to the lottery they retrospectively inferred was responsible for this outcome. These findings extend our knowledge about the range of MB functions and the scope of system interactions

The Neural Representation of Prospective Choice during Spatial Planning and Decisions

We are remarkably adept at inferring the consequences of our actions, yet the neuronal mechanisms that allow us to plan a sequence of novel choices remain unclear. We used functional magnetic resonance imaging (fMRI) to investigate how the human brain plans the shortest path to a goal in novel mazes with one (shallow maze) or two (deep maze) choice points. We observed two distinct anterior prefrontal responses to demanding choices at the second choice point: one in rostrodorsal medial prefrontal cortex (rd-mPFC)/superior frontal gyrus (SFG) that was also sensitive to (deactivated by) demanding initial choices and another in lateral frontopolar cortex (lFPC), which was only engaged by demanding choices at the second choice point. Furthermore, we identified hippocampal responses during planning that correlated with subsequent choice accuracy and response time, particularly in mazes affording sequential choices. Psychophysiological interaction (PPI) analyses showed that coupling between the hippocampus and rd-mPFC increases during sequential (deep versus shallow) planning and is higher before correct versus incorrect choices. In short, using a naturalistic spatial planning paradigm, we reveal how the human brain represents sequential choices during planning without extensive training. Our data highlight a network centred on the cortical midline and hippocampus that allows us to make prospective choices while maintaining initial choices during planning in novel environments

Structure Learning in Human Sequential Decision-Making

Studies of sequential decision-making in humans frequently find suboptimal performance relative to an ideal actor that has perfect knowledge of the model of how rewards and events are generated in the environment. Rather than being suboptimal, we argue that the learning problem humans face is more complex, in that it also involves learning the structure of reward generation in the environment. We formulate the problem of structure learning in sequential decision tasks using Bayesian reinforcement learning, and show that learning the generative model for rewards qualitatively changes the behavior of an optimal learning agent. To test whether people exhibit structure learning, we performed experiments involving a mixture of one-armed and two-armed bandit reward models, where structure learning produces many of the qualitative behaviors deemed suboptimal in previous studies. Our results demonstrate humans can perform structure learning in a near-optimal manner

Lifespan Extension by Preserving Proliferative Homeostasis in Drosophila

Regenerative processes are critical to maintain tissue homeostasis in high-turnover tissues. At the same time, proliferation of stem and progenitor cells has to be carefully controlled to prevent hyper-proliferative diseases. Mechanisms that ensure this balance, thus promoting proliferative homeostasis, are expected to be critical for longevity in metazoans. The intestinal epithelium of Drosophila provides an accessible model in which to test this prediction. In aging flies, the intestinal epithelium degenerates due to over-proliferation of intestinal stem cells (ISCs) and mis-differentiation of ISC daughter cells, resulting in intestinal dysplasia. Here we show that conditions that impair tissue renewal lead to lifespan shortening, whereas genetic manipulations that improve proliferative homeostasis extend lifespan. These include reduced Insulin/IGF or Jun-N-terminal Kinase (JNK) signaling activities, as well as over-expression of stress-protective genes in somatic stem cell lineages. Interestingly, proliferative activity in aging intestinal epithelia correlates with longevity over a range of genotypes, with maximal lifespan when intestinal proliferation is reduced but not completely inhibited. Our results highlight the importance of the balance between regenerative processes and strategies to prevent hyperproliferative disorders and demonstrate that promoting proliferative homeostasis in aging metazoans is a viable strategy to extend lifespan

Control of Metabolic Homeostasis by Stress Signaling Is Mediated by the Lipocalin NLaz

Metabolic homeostasis in metazoans is regulated by endocrine control of insulin/IGF signaling (IIS) activity. Stress and inflammatory signaling pathways—such as Jun-N-terminal Kinase (JNK) signaling—repress IIS, curtailing anabolic processes to promote stress tolerance and extend lifespan. While this interaction constitutes an adaptive response that allows managing energy resources under stress conditions, excessive JNK activity in adipose tissue of vertebrates has been found to cause insulin resistance, promoting type II diabetes. Thus, the interaction between JNK and IIS has to be tightly regulated to ensure proper metabolic adaptation to environmental challenges. Here, we identify a new regulatory mechanism by which JNK influences metabolism systemically. We show that JNK signaling is required for metabolic homeostasis in flies and that this function is mediated by the Drosophila Lipocalin family member Neural Lazarillo (NLaz), a homologue of vertebrate Apolipoprotein D (ApoD) and Retinol Binding Protein 4 (RBP4). Lipocalins are emerging as central regulators of peripheral insulin sensitivity and have been implicated in metabolic diseases. NLaz is transcriptionally regulated by JNK signaling and is required for JNK-mediated stress and starvation tolerance. Loss of NLaz function reduces stress resistance and lifespan, while its over-expression represses growth, promotes stress tolerance and extends lifespan—phenotypes that are consistent with reduced IIS activity. Accordingly, we find that NLaz represses IIS activity in larvae and adult flies. Our results show that JNK-NLaz signaling antagonizes IIS and is critical for metabolic adaptation of the organism to environmental challenges. The JNK pathway and Lipocalins are structurally and functionally conserved, suggesting that similar interactions represent an evolutionarily conserved system for the control of metabolic homeostasis

A computational psychiatry approach identifies how alpha-2A noradrenergic agonist Guanfacine affects feature-based reinforcement learning in the macaque

[EN] Noradrenaline is believed to support cognitive flexibility through the alpha 2A noradrenergic receptor (a2A-NAR) acting in prefrontal cortex. Enhanced flexibility has been inferred from improved working memory with the a2A-NA agonist Guanfacine. But it has been unclear whether Guanfacine improves specific attention and learning mechanisms beyond working memory, and whether the drug effects can be formalized computationally to allow single subject predictions. We tested and confirmed these suggestions in a case study with a healthy nonhuman primate performing a feature-based reversal learning task evaluating performance using Bayesian and Reinforcement learning models. In an initial dose-testing phase we found a Guanfacine dose that increased performance accuracy, decreased distractibility and improved learning. In a second experimental phase using only that dose we examined the faster feature-based reversal learning with Guanfacine with single-subject computational modeling. Parameter estimation suggested that improved learning is not accounted for by varying a single reinforcement learning mechanism, but by changing the set of parameter values to higher learning rates and stronger suppression of non-chosen over chosen feature information. These findings provide an important starting point for developing nonhuman primate models to discern the synaptic mechanisms of attention and learning functions within the context of a computational neuropsychiatry framework.This research was supported by grants from the Canadian Institutes of Health Research (CIHR), the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Ontario Ministry of Economic Development and Innovation (MEDI). We thank Dr. Hongying Wang for invaluable help with drug administration and animal careHassani, SA.; Oemisch, M.; Balcarras, M.; Westendorff, S.; Ardid-Ramírez, JS.; Van Der Meer, MA.; Tiesinga, P.... (2017). A computational psychiatry approach identifies how alpha-2A noradrenergic agonist Guanfacine affects feature-based reinforcement learning in the macaque. Scientific Reports. 7:1-19. https://doi.org/10.1038/srep40606S1197Arnsten, A. F., Wang, M. J. & Paspalas, C. D. Neuromodulation of thought: flexibilities and vulnerabilities in prefrontal cortical network synapses. Neuron 76, 223–239 (2012).Arnsten, A. F. & Dudley, A. G. Methylphenidate improves prefrontal cortical cognitive function through alpha2 adrenoceptor and dopamine D1 receptor actions: Relevance to therapeutic effects in Attention Deficit Hyperactivity Disorder. Behav Brain Funct 1, 2 (2005).Clark, K. L. & Noudoost, B. The role of prefrontal catecholamines in attention and working memory. Front Neural Circuits 8, 33 (2014).Wang, M. et al. Neuronal basis of age-related working memory decline. Nature 476, 210–213 (2011).Wang, M. et al. Alpha2A-adrenoceptors strengthen working memory networks by inhibiting cAMP-HCN channel signaling in prefrontal cortex. Cell 129, 397–410 (2007).Aston-Jones, G. & Cohen, J. D. An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. Annu Rev Neurosci 28, 403–450 (2005).Yu, A. J. & Dayan, P. Uncertainty, neuromodulation, and attention. Neuron 46, 681–692 (2005).Mather, M., Clewett, D., Sakaki, M. & Harley, C. W. Norepinephrine ignites local hot spots of neuronal excitation: How arousal amplifies selectivity in perception and memory. Behav Brain Sci, 1–100, doi: 10.1017/S0140525X15000667 (2015).Amemiya, S. & Redish, A. D. Manipulating Decisiveness in Decision Making: Effects of Clonidine on Hippocampal Search Strategies. J Neurosci 36, 814–827 (2016).Doya, K. Metalearning and neuromodulation. Neural Netw 15, 495–506 (2002).Uhlen, S., Muceniece, R., Rangel, N., Tiger, G. & Wikberg, J. E. Comparison of the binding activities of some drugs on alpha 2A, alpha 2B and alpha 2C-adrenoceptors and non-adrenergic imidazoline sites in the guinea pig. Pharmacology & toxicology 76, 353–364 (1995).Mao, Z. M., Arnsten, A. F. & Li, B. M. Local infusion of an alpha-1 adrenergic agonist into the prefrontal cortex impairs spatial working memory performance in monkeys. Biological psychiatry 46, 1259–1265 (1999).Arnsten, A. F. & Goldman-Rakic, P. S. Analysis of alpha-2 adrenergic agonist effects on the delayed nonmatch-to-sample performance of aged rhesus monkeys. Neurobiol Aging 11, 583–590 (1990).Franowicz, J. S. & Arnsten, A. F. The alpha-2a noradrenergic agonist, guanfacine, improves delayed response performance in young adult rhesus monkeys. Psychopharmacology 136, 8–14 (1998).Caetano, M. S. et al. Noradrenergic control of error perseveration in medial prefrontal cortex. Frontiers in Integrative Neuroscience 6, 125 (2012).Kim, S., Bobeica, I., Gamo, N. J., Arnsten, A. F. & Lee, D. Effects of alpha-2A adrenergic receptor agonist on time and risk preference in primates. Psychopharmacology 219, 363–375 (2012).Seu, E., Lang, A., Rivera, R. J. & Jentsch, J. D. Inhibition of the norepinephrine transporter improves behavioral flexibility in rats and monkeys. Psychopharmacology 202, 505–519 (2009).Kawaura, K., Karasawa, J., Chaki, S. & Hikichi, H. Stimulation of postsynapse adrenergic alpha2A receptor improves attention/cognition performance in an animal model of attention deficit hyperactivity disorder. Behav Brain Res 270, 349–356 (2014).Aoki, C., Go, C. G., Venkatesan, C. & Kurose, H. Perikaryal and synaptic localization of alpha 2A-adrenergic receptor-like immunoreactivity. Brain Res 650, 181–204 (1994).Barth, A. M., Vizi, E. S., Zelles, T. & Lendvai, B. Alpha2-adrenergic receptors modify dendritic spike generation via HCN channels in the prefrontal cortex. J Neurophysiol 99, 394–401 (2008).Ji, X. H., Ji, J. Z., Zhang, H. & Li, B. M. Stimulation of alpha2-adrenoceptors suppresses excitatory synaptic transmission in the medial prefrontal cortex of rat. Neuropsychopharmacology 33, 2263–2271 (2008).Yi, F., Liu, S. S., Luo, F., Zhang, X. H. & Li, B. M. Signaling mechanism underlying alpha2A -adrenergic suppression of excitatory synaptic transmission in the medial prefrontal cortex of rats. Eur J Neurosci 38, 2364–2373 (2013).Engberg, G. & Eriksson, E. Effects of alpha 2-adrenoceptor agonists on locus coeruleus firing rate and brain noradrenaline turnover in N-ethoxycarbonyl-2-ethoxy-1,2-dihydroquinoline (EEDQ)-treated rats. Naunyn-Schmiedeberg’s archives of pharmacology 343, 472–477 (1991).Jakala, P. et al. Guanfacine, but not clonidine, improves planning and working memory performance in humans. Neuropsychopharmacology 20, 460–470 (1999).Jakala, P. et al. Guanfacine and clonidine, alpha 2-agonists, improve paired associates learning, but not delayed matching to sample, in humans. Neuropsychopharmacology 20, 119–130 (1999).Muller, U. et al. Lack of effects of guanfacine on executive and memory functions in healthy male volunteers. Psychopharmacology 182, 205–213 (2005).Scahill, L. et al. A placebo-controlled study of guanfacine in the treatment of children with tic disorders and attention deficit hyperactivity disorder. The American journal of psychiatry 158, 1067–1074 (2001).Huys, Q. J. M., Maia, T. V. & Frank, M. J. Computational psychiatry as a bridge from neuroscience to clinical applications. Nat Neurosci 19, 404–413 (2016).Stephan, K. E. et al. Computational neuroimaging strategies for single patient predictions. NeuroImage in press (2015).Arnsten, A. F., Cai, J. X. & Goldman-Rakic, P. S. The alpha-2 adrenergic agonist guanfacine improves memory in aged monkeys without sedative or hypotensive side effects: evidence for alpha-2 receptor subtypes. J Neurosci 8, 4287–4298 (1988).Callado, L. F. & Stamford, J. A. Alpha2A- but not alpha2B/C-adrenoceptors modulate noradrenaline release in rat locus coeruleus: voltammetric data. Eur J Pharmacol 366, 35–39 (1999).Millan, M. J. et al. Cognitive dysfunction in psychiatric disorders: characteristics, causes and the quest for improved therapy. Nature reviews. Drug discovery 11, 141–168 (2012).Niv, Y. et al. Reinforcement learning in multidimensional environments relies on attention mechanisms. J Neurosci 35, 8145–8157 (2015).Balcarras, M., Ardid, S., Kaping, D., Everling, S. & Womelsdorf, T. Attentional Selection Can Be Predicted by Reinforcement Learning of Task-relevant Stimulus Features Weighted by Value-independent Stickiness. J Cogn Neurosci 28, 333–349 (2016).Redish, A. D., Jensen, S., Johnson, A. & Kurth-Nelson, Z. Reconciling reinforcement learning models with behavioral extinction and renewal: implications for addiction, relapse, and problem gambling. Psychol Rev 114, 784–805 (2007).Nassar, M. R. et al. Rational regulation of learning dynamics by pupil-linked arousal systems. Nat Neurosci 15, 1040–1046 (2012).O’Reilly, J. X. et al. Dissociable effects of surprise and model update in parietal and anterior cingulate cortex. Proc Natl Acad Sci USA 110, 3660–3669 (2013).Shenhav, A., Botvinick, M. M. & Cohen, J. D. The expected value of control: an integrative theory of anterior cingulate cortex function. Neuron 79, 217–240 (2013).Womelsdorf, T. & Everling, S. Long-Range Attention Networks: Circuit Motifs Underlying Endogenously Controlled Stimulus Selection. Trends Neurosci 38, 682–700 (2015).Yang, Y. et al. Nicotinic alpha7 receptors enhance NMDA cognitive circuits in dorsolateral prefrontal cortex. Proc Natl Acad Sci USA 110, 12078–12083 (2013).Aston-Jones, G., Rajkowski, J. & Cohen, J. Role of locus coeruleus in attention and behavioral flexibility. Biological psychiatry 46, 1309–1320 (1999).Cole, B. J. & Robbins, T. W. Forebrain norepinephrine: role in controlled information processing in the rat. Neuropsychopharmacology 7, 129–142 (1992).Dalley, J. W., Cardinal, R. N. & Robbins, T. W. Prefrontal executive and cognitive functions in rodents: neural and neurochemical substrates. Neuroscience and biobehavioral reviews 28, 771–784 (2004).Devauges, V. & Sara, S. J. Activation of the noradrenergic system facilitates an attentional shift in the rat. Behav Brain Res 39, 19–28 (1990).Connor, D. F., Arnsten, A. F., Pearson, G. S. & Greco, G. F. Guanfacine extended release for the treatment of attention-deficit/hyperactivity disorder in children and adolescents. Expert opinion on pharmacotherapy 15, 1601–1610 (2014).Sallee, F. R. et al. Guanfacine extended release in children and adolescents with attention-deficit/hyperactivity disorder: a placebo-controlled trial. J Am Acad Child Adolesc Psychiatry 48, 155–165 (2009).Steere, J. C. & Arnsten, A. F. The alpha-2A noradrenergic receptor agonist guanfacine improves visual object discrimination reversal performance in aged rhesus monkeys. Behav Neurosci 111, 883–891 (1997).Doya, K. Modulators of decision making. Nat Neurosci 11, 410–416 (2008).Wang, X. J. & Krystal, J. H. Computational psychiatry. Neuron 84, 638–654 (2014).Wiecki, T. V. et al. A Computational Cognitive Biomarker for Early-Stage Huntington’s Disease. PLoS One 11, e0148409, doi: 10.1371/journal.pone.0148409 (2016).Huys, Q. J., Pizzagalli, D. A., Bogdan, R. & Dayan, P. Mapping anhedonia onto reinforcement learning: a behavioural meta-analysis. Biol Mood Anxiety Disord 3, 12 (2013).Gershman, S. J. & Niv, Y. Learning latent structure: carving nature at its joints. Curr Opin Neurobiol 20, 251–256 (2010).Voon, V. et al. Disorders of compulsivity: a common bias towards learning habits. Mol Psychiatry 20, 345–352 (2015).Maia, T. V. & Frank, M. J. From reinforcement learning models to psychiatric and neurological disorders. Nature Neuroscience 14, 154–162 (2011).Adams, R. A., Huys, Q. J. M. & Roiser, J. P. Computational Psychiatry: towards a mathematically informed understanding of mental illness. Journal of Neurology, Neurosurgery, and Psychiatry 87, 53–63 (2015).Schlagenhauf, F. et al. Striatal dysfunction during reversal learning in unmedicated schizophrenia patients. NeuroImage 89, 171–180 (2014).Harlé, K. M. et al. Bayesian neural adjustment of inhibitory control predicts emergence of problem stimulant use. Brain 138, 3413–3426 (2015).Zhang, J. et al. Different decision deficits impair response inhibition in progressive supranuclear palsy and Parkinson’s disease. Brain 139, 161–173 (2016).Frank, M. J. et al. fMRI and EEG Predictors of Dynamic Decision Parameters during Human Reinforcement Learning. Journal of Neuroscience 35, 485–494 (2015).Smith, A. C. & Brown, E. N. Estimating a state-space model from point process observations. Neural Comput 15, 965–991 (2003).Wilson, R. C. & Niv, Y. Inferring relevance in a changing world. Frontiers in human neuroscience 5, 189 (2011).Rämä, P. et al. Medetomidine, atipamezole, and guanfacine in delayed response performance of aged monkeys. Pharmacology Biochemistry and Behavior 55, 415–422 (1996).Arnsten, A. F. T. & Contant, T. A. Alpha-2 adrenergic agonists decrease distractibility in aged monkeys performing the delayed response task. Psychopharmacology 108, 159–169 (1992).O’Neill, J. et al. Effects of guanfacine on three forms of distraction in the aging macaque. Life Sciences 67, 877–885 (2000).Wang, M., Ji, J.-Z. & Li, B.-M. The α2A-Adrenergic Agonist Guanfacine Improves Visuomotor Associative Learning in Monkeys. Neuropsychopharmacology 29, 86–92 (2004).Witte, E. a. & Marrocco, R. T. Alteration of brain noradrenergic activity in rhesus monkeys affects the alerting component of covert orienting. Psychopharmacology 132, 315–323 (1997).Decamp, E., Clark, K. & Schneider, J. S. Effects of the alpha-2 adrenoceptor agonist guanfacine on attention and working memory in aged non-human primates. European Journal of Neuroscience 34, 1018–1022 (2011)