Eligibility Traces and Plasticity on Behavioral Time Scales: Experimental Support of neoHebbian Three-Factor Learning Rules

Most elementary behaviors such as moving the arm to grasp an object or walking into the next room to explore a museum evolve on the time scale of seconds; in contrast, neuronal action potentials occur on the time scale of a few milliseconds. Learning rules of the brain must therefore bridge the gap between these two different time scales. Modern theories of synaptic plasticity have postulated that the co-activation of pre- and postsynaptic neurons sets a flag at the synapse, called an eligibility trace, that leads to a weight change only if an additional factor is present while the flag is set. This third factor, signaling reward, punishment, surprise, or novelty, could be implemented by the phasic activity of neuromodulators or specific neuronal inputs signaling special events. While the theoretical framework has been developed over the last decades, experimental evidence in support of eligibility traces on the time scale of seconds has been collected only during the last few years. Here we review, in the context of three-factor rules of synaptic plasticity, four key experiments that support the role of synaptic eligibility traces in combination with a third factor as a biological implementation of neoHebbian three-factor learning rules

Spiking Neural Networks for Inference and Learning: A Memristor-based Design Perspective

On metrics of density and power efficiency, neuromorphic technologies have the potential to surpass mainstream computing technologies in tasks where real-time functionality, adaptability, and autonomy are essential. While algorithmic advances in neuromorphic computing are proceeding successfully, the potential of memristors to improve neuromorphic computing have not yet born fruit, primarily because they are often used as a drop-in replacement to conventional memory. However, interdisciplinary approaches anchored in machine learning theory suggest that multifactor plasticity rules matching neural and synaptic dynamics to the device capabilities can take better advantage of memristor dynamics and its stochasticity. Furthermore, such plasticity rules generally show much higher performance than that of classical Spike Time Dependent Plasticity (STDP) rules. This chapter reviews the recent development in learning with spiking neural network models and their possible implementation with memristor-based hardware

The role of excitation and inhibition in learning and memory formation

The neurons in the mammalian brain can be classified into two broad categories: excitatory and inhibitory neurons. The former has been historically associated to information processing whereas the latter has been linked to network homeostasis. More recently, inhibitory neurons have been related to several computational roles such as the gating of signal propagation, mediation of network competition, or learning. However, the ways by which excitation and inhibition can regulate learning have not been exhaustively explored. Here we explore several model systems to investigate the role of excitation and inhibition in learning and memory formation. Additionally, we investigate the effect that third factors such as neuromodulators and network state exert over this process. Firstly, we explore the effect of neuromodulators onto excitatory neurons and excitatory plasticity. Next, we investigate the plasticity rules governing excitatory connections while the neural network oscillates in a sleep-like cycle, shifting between Up and Down states. We observe that this plasticity rule depends on the state of the network. To study the role of inhibitory neurons in learning, we then investigate the mechanisms underlying place field emergence and consolidation. Our simulations suggest that dendrite-targeting interneurons play an important role in both promoting the emergence of new place fields and in ensuring place field stabilization. Soma-targeting interneurons, on the other hand, are suggested to be related to quick, context-specific changes in the assignment of place and silent cells. We next investigate the mechanisms underlying the plasticity of synaptic connections from specific types of interneurons. Our experiments suggest that different types of interneurons undergo different synaptic plasticity rules. Using a computational model, we implement these plasticity rules in a simplified network. Our simulations indicate that the interaction between the different forms of plasticity account for the development of stable place fields across multiple environments. Moreover, these plasticity rules seems to be gated by the postsynaptic membrane voltage. Inspired by these findings, we propose a voltage-based inhibitory synaptic plasticity rule. As a consequence of this rule, the network activity is kept controlled by the imposition of a maximum pyramidal cell firing rate. Remarkably, this rule does not constrain the postsynaptic firing rate to a narrow range. Overall, through multiple stages of interactions between experiments and computational simulations, we investigate the effect of excitation and inhibition in learning. We propose mechanistic explanations for experimental data, and suggest possible functional implications of experimental findings. Finally, we proposed a voltage-based inhibitory synaptic plasticity model as a mechanism for flexible network homeostasis.Open Acces

A History of Spike-Timing-Dependent Plasticity

How learning and memory is achieved in the brain is a central question in neuroscience. Key to today’s research into information storage in the brain is the concept of synaptic plasticity, a notion that has been heavily influenced by Hebb's (1949) postulate. Hebb conjectured that repeatedly and persistently co-active cells should increase connective strength among populations of interconnected neurons as a means of storing a memory trace, also known as an engram. Hebb certainly was not the first to make such a conjecture, as we show in this history. Nevertheless, literally thousands of studies into the classical frequency-dependent paradigm of cellular learning rules were directly inspired by the Hebbian postulate. But in more recent years, a novel concept in cellular learning has emerged, where temporal order instead of frequency is emphasized. This new learning paradigm – known as spike-timing-dependent plasticity (STDP) – has rapidly gained tremendous interest, perhaps because of its combination of elegant simplicity, biological plausibility, and computational power. But what are the roots of today’s STDP concept? Here, we discuss several centuries of diverse thinking, beginning with philosophers such as Aristotle, Locke, and Ribot, traversing, e.g., Lugaro’s plasticità and Rosenblatt’s perceptron, and culminating with the discovery of STDP. We highlight interactions between theoretical and experimental fields, showing how discoveries sometimes occurred in parallel, seemingly without much knowledge of the other field, and sometimes via concrete back-and-forth communication. We point out where the future directions may lie, which includes interneuron STDP, the functional impact of STDP, its mechanisms and its neuromodulatory regulation, and the linking of STDP to the developmental formation and continuous plasticity of neuronal networks

The influence of dopamine on prediction, action and learning

In this thesis I explore functions of the neuromodulator dopamine in the context of autonomous learning and behaviour. I first investigate dopaminergic influence within a simulated agent-based model, demonstrating how modulation of synaptic plasticity can enable reward-mediated learning that is both adaptive and self-limiting. I describe how this mechanism is driven by the dynamics of agentenvironment interaction and consequently suggest roles for both complex spontaneous neuronal activity and specific neuroanatomy in the expression of early, exploratory behaviour. I then show how the observed response of dopamine neurons in the mammalian basal ganglia may also be modelled by similar processes involving dopaminergic neuromodulation and cortical spike-pattern representation within an architecture of counteracting excitatory and inhibitory neural pathways, reflecting gross mammalian neuroanatomy. Significantly, I demonstrate how combined modulation of synaptic plasticity and neuronal excitability enables specific (timely) spike-patterns to be recognised and selectively responded to by efferent neural populations, therefore providing a novel spike-timing based implementation of the hypothetical ‘serial-compound’ representation suggested by temporal difference learning. I subsequently discuss more recent work, focused upon modelling those complex spike-patterns observed in cortex. Here, I describe neural features likely to contribute to the expression of such activity and subsequently present novel simulation software allowing for interactive exploration of these factors, in a more comprehensive neural model that implements both dynamical synapses and dopaminergic neuromodulation. I conclude by describing how the work presented ultimately suggests an integrated theory of autonomous learning, in which direct coupling of agent and environment supports a predictive coding mechanism, bootstrapped in early development by a more fundamental process of trial-and-error learning

Neuropharmacological Investigation Of Stress And Nicotine Self-Administration Among Current Cigarette Smokers

ABSTRACT NEUROPHARMACOLOGICAL INVESTIGATION OF STRESS AND NICOTINE SELF-ADMINISTRATION AMONG CURRENT CIGARETTE SMOKERS by ERIC ANDREW WOODCOCK August 2017 Advisor: Dr. Mark K. Greenwald Major: Neuroscience (Translational) Degree: Doctor of Philosophy Nicotine use, especially cigarette smoking, is a significant public health problem. Existing pharmacotherapies attenuate nicotine craving and withdrawal symptoms. However, the majority of patients relapse within the first year of treatment. Treatment studies indicate a commonly cited precipitant to smoking relapse is stress. Pharmacotherapies do not attenuate, and may exacerbate, the effects of acute stress. Experimental studies (preclinical and clinical) indicate that acute stress potentiates drug-seeking behavior across drugs of abuse. Despite a robust literature linking acute stress and substance use, neurobiological mechanisms remain poorly understood. A more complete understanding of the neurobiological effects of acute stress on brain function may facilitate development of novel interventions. Adjunctive stress-blunting medications may improve the effectiveness of existing pharmacotherapies. The present study investigated the effects of pharmacological stress-induction among cigarette smokers. Non-treatment-seeking cigarette smokers were recruited locally and screened for psychiatric, medical, and neuroimaging contraindications. Using a double-blind, placebo-controlled within-subject random cross-over design, participants (N = 21) completed two oral-dosing experimental sessions: active (yohimbine [YOH] 54mg + hydrocortisone [HYD] 10mg) and placebo (YOH 0mg + HYD 0mg) stress. Prior research indicated that YOH+HYD is a robust pharmacological stress-induction technique that stimulates the Autonomic Nervous System (ANS) and Hypothalamic-Pituitary-Adrenal (HPA) axis systems, increases circulating levels of noradrenaline and cortisol (two primary stress hormones), and potentiates drug-seeking behavior. Throughout each experimental session, subjective and physiological effects were measured. In addition, participants completed a 60min magnetic resonance imaging (MRI) scan which consisted of three task paradigms: 1) letter 2-back, 2) smoking cued letter N-back, and 3) breath-hold challenge. Participants completed a working memory paradigm (letter 2-back) during proton functional magnetic resonance spectroscopy (1H fMRS). Left dorsolateral prefrontal cortex (dlPFC) neurochemistry was evaluated during letter 2-back task performance. Next, participants completed a cued N-back paradigm that consisted of images (cigarette smoking or neutral) centered behind capitalized letters across three levels of N-back task difficulty: 0-, 1-, and 2-back. Finally, participants were instructed (visually) to control their breathing across three phases: ‘normal’ breathing, paced breathing (3s in/3s out), and breath-hold challenge (11s). After the MRI scan, participants completed a choice progressive ratio task. Across 11 independent choice trials, participants could earn one cigarette puff (preferred brand) or money ($0.25) via behavioral responding. Each successive unit earned (puffs or money, independently) was associated with a higher response requirement (progressive ratio schedule). At the end of the 30min task, participants smoked the exact number of cigarette puffs earned and/or were provided the amount of money earned. Number of puffs earned and smoked was a direct measure of nicotine-seeking and self-administration behavior (nicotine motivation). Participants were compensated for their time. Results indicated that oral pretreatment with YOH+HYD increased biomarkers of a physiological stress response: systolic and diastolic blood pressure, heart rate, saliva cortisol and α-amylase (indirect biomarker of noradrenaline levels), relative to placebo. YOH+HYD potentiated nicotine-seeking and self-administration behavior (controlling for nicotine dependence level), relative to placebo. Appetitive and relief-motivated cigarette craving, nicotine withdrawal symptoms, negative affect, and anxiety levels increased throughout each session, but did not differ by experimental session (active vs. placebo stress). Similarly, positive affect decreased throughout each session, but did not as a function of stress. 1H fMRS indicated that letter 2-back performance increased left dlPFC glutamate (GLU) levels relative to interleaved fixation cross rest (indicative of task engagement) during the placebo, but not active stress, session. Further, YOH+HYD impaired letter 2-back response accuracy, relative to placebo. Across N-back levels (0-, 1-, and 2-back), fMRI indicated more robust neural activation across ‘reward’-associated brain regions in response to smoking images (\u3e neutral images) during placebo, relative to active stress. Results demonstrated YOH+HYD induced a sustained physiological stress response (ANS and HPA axis) and potentiated nicotine-seeking and self-administration. YOH+HYD attenuated dlPFC task engagement and impaired response accuracy during a well-established working memory task. These findings provide experimental support for a plausible neurobiological mechanism through which acute stress may potentiate nicotine self-administration. Acute stress-impaired dlPFC function may potentiate nicotine self-administration and, among abstinence-motivated individuals, precipitate smoking relapse. Prior research demonstrated dlPFC function is associated with a host of cognitive processes (e.g. delayed gratification, self-control, decision making, etc.) associated with prolonged smoking abstinence. Future studies are needed to confirm this hypothesis, investigate dose-response relationships, and evaluate the efficacy of stress-blunting medications in combination with existing pharmacotherapies for smoking cessation

Modulation of Spike-Timing Dependent Plasticity: Towards the Inclusion of a Third Factor in Computational Models

In spike-timing dependent plasticity (STDP) change in synaptic strength depends on the timing of pre- vs. postsynaptic spiking activity. Since STDP is in compliance with Hebb’s postulate, it is considered one of the major mechanisms of memory storage and recall. STDP comprises a system of two coincidence detectors with N-methyl-D-aspartate receptor (NMDAR) activation often posited as one of the main components. Numerous studies have unveiled a third component of this coincidence detection system, namely neuromodulation and glia activity shaping STDP. Even though dopaminergic control of STDP has most often been reported, acetylcholine, noradrenaline, nitric oxide (NO), brain-derived neurotrophic factor (BDNF) or gamma-aminobutyric acid (GABA) also has been shown to effectively modulate STDP. Furthermore, it has been demonstrated that astrocytes, via the release or uptake of glutamate, gate STDP expression. At the most fundamental level, the timing properties of STDP are expected to depend on the spatiotemporal dynamics of the underlying signaling pathways. However in most cases, due to technical limitations experiments grant only indirect access to these pathways. Computational models carefully constrained by experiments, allow for a better qualitative understanding of the molecular basis of STDP and its regulation by neuromodulators. Recently, computational models of calcium dynamics and signaling pathway molecules have started to explore STDP emergence in ex and in vivo-like conditions. These models are expected to reproduce better at least part of the complex modulation of STDP as an emergent property of the underlying molecular pathways. Elucidation of the mechanisms underlying STDP modulation and its consequences on network dynamics is of critical importance and will allow better understanding of the major mechanisms of memory storage and recall both in health and disease