Eligibility Traces and Plasticity on Behavioral Time Scales: Experimental Support of neoHebbian Three-Factor Learning Rules

Most elementary behaviors such as moving the arm to grasp an object or walking into the next room to explore a museum evolve on the time scale of seconds; in contrast, neuronal action potentials occur on the time scale of a few milliseconds. Learning rules of the brain must therefore bridge the gap between these two different time scales. Modern theories of synaptic plasticity have postulated that the co-activation of pre- and postsynaptic neurons sets a flag at the synapse, called an eligibility trace, that leads to a weight change only if an additional factor is present while the flag is set. This third factor, signaling reward, punishment, surprise, or novelty, could be implemented by the phasic activity of neuromodulators or specific neuronal inputs signaling special events. While the theoretical framework has been developed over the last decades, experimental evidence in support of eligibility traces on the time scale of seconds has been collected only during the last few years. Here we review, in the context of three-factor rules of synaptic plasticity, four key experiments that support the role of synaptic eligibility traces in combination with a third factor as a biological implementation of neoHebbian three-factor learning rules

Spiking Neural Networks for Inference and Learning: A Memristor-based Design Perspective

On metrics of density and power efficiency, neuromorphic technologies have the potential to surpass mainstream computing technologies in tasks where real-time functionality, adaptability, and autonomy are essential. While algorithmic advances in neuromorphic computing are proceeding successfully, the potential of memristors to improve neuromorphic computing have not yet born fruit, primarily because they are often used as a drop-in replacement to conventional memory. However, interdisciplinary approaches anchored in machine learning theory suggest that multifactor plasticity rules matching neural and synaptic dynamics to the device capabilities can take better advantage of memristor dynamics and its stochasticity. Furthermore, such plasticity rules generally show much higher performance than that of classical Spike Time Dependent Plasticity (STDP) rules. This chapter reviews the recent development in learning with spiking neural network models and their possible implementation with memristor-based hardware

The role of excitation and inhibition in learning and memory formation

The neurons in the mammalian brain can be classified into two broad categories: excitatory and inhibitory neurons. The former has been historically associated to information processing whereas the latter has been linked to network homeostasis. More recently, inhibitory neurons have been related to several computational roles such as the gating of signal propagation, mediation of network competition, or learning. However, the ways by which excitation and inhibition can regulate learning have not been exhaustively explored. Here we explore several model systems to investigate the role of excitation and inhibition in learning and memory formation. Additionally, we investigate the effect that third factors such as neuromodulators and network state exert over this process. Firstly, we explore the effect of neuromodulators onto excitatory neurons and excitatory plasticity. Next, we investigate the plasticity rules governing excitatory connections while the neural network oscillates in a sleep-like cycle, shifting between Up and Down states. We observe that this plasticity rule depends on the state of the network. To study the role of inhibitory neurons in learning, we then investigate the mechanisms underlying place field emergence and consolidation. Our simulations suggest that dendrite-targeting interneurons play an important role in both promoting the emergence of new place fields and in ensuring place field stabilization. Soma-targeting interneurons, on the other hand, are suggested to be related to quick, context-specific changes in the assignment of place and silent cells. We next investigate the mechanisms underlying the plasticity of synaptic connections from specific types of interneurons. Our experiments suggest that different types of interneurons undergo different synaptic plasticity rules. Using a computational model, we implement these plasticity rules in a simplified network. Our simulations indicate that the interaction between the different forms of plasticity account for the development of stable place fields across multiple environments. Moreover, these plasticity rules seems to be gated by the postsynaptic membrane voltage. Inspired by these findings, we propose a voltage-based inhibitory synaptic plasticity rule. As a consequence of this rule, the network activity is kept controlled by the imposition of a maximum pyramidal cell firing rate. Remarkably, this rule does not constrain the postsynaptic firing rate to a narrow range. Overall, through multiple stages of interactions between experiments and computational simulations, we investigate the effect of excitation and inhibition in learning. We propose mechanistic explanations for experimental data, and suggest possible functional implications of experimental findings. Finally, we proposed a voltage-based inhibitory synaptic plasticity model as a mechanism for flexible network homeostasis.Open Acces

The influence of dopamine on prediction, action and learning

In this thesis I explore functions of the neuromodulator dopamine in the context of autonomous learning and behaviour. I first investigate dopaminergic influence within a simulated agent-based model, demonstrating how modulation of synaptic plasticity can enable reward-mediated learning that is both adaptive and self-limiting. I describe how this mechanism is driven by the dynamics of agentenvironment interaction and consequently suggest roles for both complex spontaneous neuronal activity and specific neuroanatomy in the expression of early, exploratory behaviour. I then show how the observed response of dopamine neurons in the mammalian basal ganglia may also be modelled by similar processes involving dopaminergic neuromodulation and cortical spike-pattern representation within an architecture of counteracting excitatory and inhibitory neural pathways, reflecting gross mammalian neuroanatomy. Significantly, I demonstrate how combined modulation of synaptic plasticity and neuronal excitability enables specific (timely) spike-patterns to be recognised and selectively responded to by efferent neural populations, therefore providing a novel spike-timing based implementation of the hypothetical ‘serial-compound’ representation suggested by temporal difference learning. I subsequently discuss more recent work, focused upon modelling those complex spike-patterns observed in cortex. Here, I describe neural features likely to contribute to the expression of such activity and subsequently present novel simulation software allowing for interactive exploration of these factors, in a more comprehensive neural model that implements both dynamical synapses and dopaminergic neuromodulation. I conclude by describing how the work presented ultimately suggests an integrated theory of autonomous learning, in which direct coupling of agent and environment supports a predictive coding mechanism, bootstrapped in early development by a more fundamental process of trial-and-error learning

A History of Spike-Timing-Dependent Plasticity

How learning and memory is achieved in the brain is a central question in neuroscience. Key to today’s research into information storage in the brain is the concept of synaptic plasticity, a notion that has been heavily influenced by Hebb's (1949) postulate. Hebb conjectured that repeatedly and persistently co-active cells should increase connective strength among populations of interconnected neurons as a means of storing a memory trace, also known as an engram. Hebb certainly was not the first to make such a conjecture, as we show in this history. Nevertheless, literally thousands of studies into the classical frequency-dependent paradigm of cellular learning rules were directly inspired by the Hebbian postulate. But in more recent years, a novel concept in cellular learning has emerged, where temporal order instead of frequency is emphasized. This new learning paradigm – known as spike-timing-dependent plasticity (STDP) – has rapidly gained tremendous interest, perhaps because of its combination of elegant simplicity, biological plausibility, and computational power. But what are the roots of today’s STDP concept? Here, we discuss several centuries of diverse thinking, beginning with philosophers such as Aristotle, Locke, and Ribot, traversing, e.g., Lugaro’s plasticità and Rosenblatt’s perceptron, and culminating with the discovery of STDP. We highlight interactions between theoretical and experimental fields, showing how discoveries sometimes occurred in parallel, seemingly without much knowledge of the other field, and sometimes via concrete back-and-forth communication. We point out where the future directions may lie, which includes interneuron STDP, the functional impact of STDP, its mechanisms and its neuromodulatory regulation, and the linking of STDP to the developmental formation and continuous plasticity of neuronal networks

Modulation of Spike-Timing Dependent Plasticity: Towards the Inclusion of a Third Factor in Computational Models

In spike-timing dependent plasticity (STDP) change in synaptic strength depends on the timing of pre- vs. postsynaptic spiking activity. Since STDP is in compliance with Hebb’s postulate, it is considered one of the major mechanisms of memory storage and recall. STDP comprises a system of two coincidence detectors with N-methyl-D-aspartate receptor (NMDAR) activation often posited as one of the main components. Numerous studies have unveiled a third component of this coincidence detection system, namely neuromodulation and glia activity shaping STDP. Even though dopaminergic control of STDP has most often been reported, acetylcholine, noradrenaline, nitric oxide (NO), brain-derived neurotrophic factor (BDNF) or gamma-aminobutyric acid (GABA) also has been shown to effectively modulate STDP. Furthermore, it has been demonstrated that astrocytes, via the release or uptake of glutamate, gate STDP expression. At the most fundamental level, the timing properties of STDP are expected to depend on the spatiotemporal dynamics of the underlying signaling pathways. However in most cases, due to technical limitations experiments grant only indirect access to these pathways. Computational models carefully constrained by experiments, allow for a better qualitative understanding of the molecular basis of STDP and its regulation by neuromodulators. Recently, computational models of calcium dynamics and signaling pathway molecules have started to explore STDP emergence in ex and in vivo-like conditions. These models are expected to reproduce better at least part of the complex modulation of STDP as an emergent property of the underlying molecular pathways. Elucidation of the mechanisms underlying STDP modulation and its consequences on network dynamics is of critical importance and will allow better understanding of the major mechanisms of memory storage and recall both in health and disease

Learning cortical representations through perturbed and adversarial dreaming.

Humans and other animals learn to extract general concepts from sensory experience without extensive teaching. This ability is thought to be facilitated by offline states like sleep where previous experiences are systemically replayed. However, the characteristic creative nature of dreams suggests that learning semantic representations may go beyond merely replaying previous experiences. We support this hypothesis by implementing a cortical architecture inspired by generative adversarial networks (GANs). Learning in our model is organized across three different global brain states mimicking wakefulness, NREM and REM sleep, optimizing different, but complementary objective functions. We train the model on standard datasets of natural images and evaluate the quality of the learned representations. Our results suggest that generating new, virtual sensory inputs via adversarial dreaming during REM sleep is essential for extracting semantic concepts, while replaying episodic memories via perturbed dreaming during NREM sleep improves the robustness of latent representations. The model provides a new computational perspective on sleep states, memory replay and dreams and suggests a cortical implementation of GANs

Corticolimbic catecholamines in stress: A computational model of the appraisal of controllability

Appraisal of a stressful situation and the possibility to control or avoid it is thought to involve frontal-cortical mechanisms. The precise mechanism underlying this appraisal and its translation into effective stress coping (the regulation of physiological and behavioural responses) are poorly understood. Here, we propose a computational model which involves tuning motivational arousal to the appraised stressing condition. The model provides a causal explanation of the shift from active to passive coping strategies, i.e. from a condition characterised by high motivational arousal, required to deal with a situation appraised as stressful, to a condition characterised by emotional and motivational withdrawal, required when the stressful situation is appraised as uncontrollable/unavoidable. The model is motivated by results acquired via microdialysis recordings in rats and highlights the presence of two competing circuits dominated by different areas of the ventromedial prefrontal cortex: these are shown having opposite effects on several subcortical areas, affecting dopamine outflow in the striatum, and therefore controlling motivation. We start by reviewing published data supporting structure and functioning of the neural model and present the computational model itself with its essential neural mechanisms. Finally, we show the results of a new experiment, involving the condition of repeated inescapable stress, which validate most of the model's prediction