BP(\lambda): Online Learning via Synthetic Gradients

Training recurrent neural networks typically relies on backpropagation through time (BPTT). BPTT depends on forward and backward passes to be completed, rendering the network locked to these computations before loss gradients are available. Recently, Jaderberg et al. proposed synthetic gradients to alleviate the need for full BPTT. In their implementation synthetic gradients are learned through a mixture of backpropagated gradients and bootstrapped synthetic gradients, analogous to the temporal difference (TD) algorithm in Reinforcement Learning (RL). However, as in TD learning, heavy use of bootstrapping can result in bias which leads to poor synthetic gradient estimates. Inspired by the accumulate $\mathrm{TD}(\lambda)$ in RL, we propose a fully online method for learning synthetic gradients which avoids the use of BPTT altogether: accumulate $BP(\lambda)$. As in accumulate $\mathrm{TD}(\lambda)$, we show analytically that accumulate $\mathrm{BP}(\lambda)$ can control the level of bias by using a mixture of temporal difference errors and recursively defined eligibility traces. We next demonstrate empirically that our model outperforms the original implementation for learning synthetic gradients in a variety of tasks, and is particularly suited for capturing longer timescales. Finally, building on recent work we reflect on accumulate $\mathrm{BP}(\lambda)$ as a principle for learning in biological circuits. In summary, inspired by RL principles we introduce an algorithm capable of bias-free online learning via synthetic gradients.Comment: 24 pages, 7 figure

Developmental depression-to-facilitation shift controls excitation-inhibition balance

Changes in the short-term dynamics of excitatory synapses over development have been observed throughout cortex, but their purpose and consequences remain unclear. Here, we propose that developmental changes in synaptic dynamics buffer the effect of slow inhibitory long-term plasticity, allowing for continuously stable neural activity. Using computational modeling we demonstrate that early in development excitatory short-term depression quickly stabilises neural activity, even in the face of strong, unbalanced excitation. We introduce a model of the commonly observed developmental shift from depression to facilitation and show that neural activity remains stable throughout development, while inhibitory synaptic plasticity slowly balances excitation, consistent with experimental observations. Our model predicts changes in the input responses from phasic to phasic-and-tonic and more precise spike timings. We also observe a gradual emergence of short-lasting memory traces governed by short-term plasticity development. We conclude that the developmental depression-to-facilitation shift may control excitation-inhibition balance throughout development with important functional consequences

The short-term plasticity of VIP interneurons in motor cortex

Short-term plasticity is an important feature in the brain for shaping neural dynamics and for information processing. Short-term plasticity is known to depend on many factors including brain region, cortical layer, and cell type. Here we focus on vasoactive-intestinal peptide (VIP) interneurons (INs). VIP INs play a key disinhibitory role in cortical circuits by inhibiting other IN types, including Martinotti cells (MCs) and basket cells (BCs). Despite this prominent role, short-term plasticity at synapses to and from VIP INs is not well described. In this study, we therefore characterized the short-term plasticity at inputs and outputs of genetically targeted VIP INs in mouse motor cortex. To explore inhibitory to inhibitory (I → I) short-term plasticity at layer 2/3 (L2/3) VIP IN outputs onto L5 MCs and BCs, we relied on a combination of whole-cell recording, 2-photon microscopy, and optogenetics, which revealed that VIP IN→MC/BC synapses were consistently short-term depressing. To explore excitatory (E) → I short-term plasticity at inputs to VIP INs, we used extracellular stimulation. Surprisingly, unlike VIP IN outputs, E → VIP IN synapses exhibited heterogeneous short-term dynamics, which we attributed to the target VIP IN cell rather than the input. Computational modeling furthermore linked the diversity in short-term dynamics at VIP IN inputs to a wide variability in probability of release. Taken together, our findings highlight how short-term plasticity at VIP IN inputs and outputs is specific to synapse type. We propose that the broad diversity in short-term plasticity of VIP IN inputs forms a basis to code for a broad range of contrasting signal dynamics

Distributional coding of associative learning in discrete populations of midbrain dopamine neurons

Midbrain dopamine neurons are thought to play key roles in learning by conveying the difference between expected and actual outcomes. Recent evidence suggests diversity in dopamine signaling, yet it remains poorly understood how heterogeneous signals might be organized to facilitate the role of downstream circuits mediating distinct aspects of behavior. Here, we investigated the organizational logic of dopaminergic signaling by recording and labeling individual midbrain dopamine neurons during associative behavior. Our findings show that reward information and behavioral parameters are not only heterogeneously encoded but also differentially distributed across populations of dopamine neurons. Retrograde tracing and fiber photometry suggest that populations of dopamine neurons projecting to different striatal regions convey distinct signals. These data, supported by computational modeling, indicate that such distributional coding can maximize dynamic range and tailor dopamine signals to facilitate specialized roles of different striatal regions

Functional consequences of pre- and postsynaptic expression of synaptic plasticity

Growing experimental evidence shows that both homeostatic and Hebbian synaptic plasticity can be expressed presynaptically as well as postsynaptically. In this review, we start by discussing this evidence and methods used to determine expression loci. Next, we discuss functional consequences of this diversity in pre- and postsynaptic expression of both homeostatic and Hebbian synaptic plasticity. In particular, we explore the functional consequences of a biologically tuned model of pre- and postsynaptically expressed spike-timing-dependent plasticity complemented with postsynaptic homeostatic control. The pre- and postsynaptic expression in this model predicts 1) more reliable receptive fields and sensory perception, 2) rapid recovery of forgotten information (memory savings) and 3) reduced response latencies, compared to a model with postsynaptic expression only. Finally we discuss open questions that will require a considerable research effort to better elucidate how the specific locus of expression of homeostatic and Hebbian plasticity alters synaptic and network computations

Functional consequences of pre- and postsynaptic expression of synaptic plasticity

Growing experimental evidence shows that both homeostatic and Hebbian synaptic plasticity can be expressed presynaptically as well as postsynaptically. In this review, we start by discussing this evidence and methods used to determine expression loci. Next, we discuss the functional consequences of this diversity in pre- and postsynaptic expression of both homeostatic and Hebbian synaptic plasticity. In particular, we explore the functional consequences of a biologically tuned model of pre- and postsynaptically expressed spike-timing-dependent plasticity complemented with postsynaptic homeostatic control. The pre- and postsynaptic expression in this model predicts (i) more reliable receptive fields and sensory perception, (ii) rapid recovery of forgotten information (memory savings), and (iii) reduced response latencies, compared with a model with postsynaptic expression only. Finally, we discuss open questions that will require a considerable research effort to better elucidate how the specific locus of expression of homeostatic and Hebbian plasticity alters synaptic and network computations.This article is part of the themed issue 'Integrating Hebbian and homeostatic plasticity'