13 research outputs found
Supervised Learning in Spiking Neural Networks with Phase-Change Memory Synapses
Spiking neural networks (SNN) are artificial computational models that have
been inspired by the brain's ability to naturally encode and process
information in the time domain. The added temporal dimension is believed to
render them more computationally efficient than the conventional artificial
neural networks, though their full computational capabilities are yet to be
explored. Recently, computational memory architectures based on non-volatile
memory crossbar arrays have shown great promise to implement parallel
computations in artificial and spiking neural networks. In this work, we
experimentally demonstrate for the first time, the feasibility to realize
high-performance event-driven in-situ supervised learning systems using
nanoscale and stochastic phase-change synapses. Our SNN is trained to recognize
audio signals of alphabets encoded using spikes in the time domain and to
generate spike trains at precise time instances to represent the pixel
intensities of their corresponding images. Moreover, with a statistical model
capturing the experimental behavior of the devices, we investigate
architectural and systems-level solutions for improving the training and
inference performance of our computational memory-based system. Combining the
computational potential of supervised SNNs with the parallel compute power of
computational memory, the work paves the way for next-generation of efficient
brain-inspired systems
Training Multi-layer Spiking Neural Networks using NormAD based Spatio-Temporal Error Backpropagation
Spiking neural networks (SNNs) have garnered a great amount of interest for
supervised and unsupervised learning applications. This paper deals with the
problem of training multi-layer feedforward SNNs. The non-linear
integrate-and-fire dynamics employed by spiking neurons make it difficult to
train SNNs to generate desired spike trains in response to a given input. To
tackle this, first the problem of training a multi-layer SNN is formulated as
an optimization problem such that its objective function is based on the
deviation in membrane potential rather than the spike arrival instants. Then,
an optimization method named Normalized Approximate Descent (NormAD),
hand-crafted for such non-convex optimization problems, is employed to derive
the iterative synaptic weight update rule. Next, it is reformulated to
efficiently train multi-layer SNNs, and is shown to be effectively performing
spatio-temporal error backpropagation. The learning rule is validated by
training -layer SNNs to solve a spike based formulation of the XOR problem
as well as training -layer SNNs for generic spike based training problems.
Thus, the new algorithm is a key step towards building deep spiking neural
networks capable of efficient event-triggered learning.Comment: 19 pages, 10 figure
Spiking Neural Networks for Inference and Learning: A Memristor-based Design Perspective
On metrics of density and power efficiency, neuromorphic technologies have
the potential to surpass mainstream computing technologies in tasks where
real-time functionality, adaptability, and autonomy are essential. While
algorithmic advances in neuromorphic computing are proceeding successfully, the
potential of memristors to improve neuromorphic computing have not yet born
fruit, primarily because they are often used as a drop-in replacement to
conventional memory. However, interdisciplinary approaches anchored in machine
learning theory suggest that multifactor plasticity rules matching neural and
synaptic dynamics to the device capabilities can take better advantage of
memristor dynamics and its stochasticity. Furthermore, such plasticity rules
generally show much higher performance than that of classical Spike Time
Dependent Plasticity (STDP) rules. This chapter reviews the recent development
in learning with spiking neural network models and their possible
implementation with memristor-based hardware
Surrogate Gradient Learning in Spiking Neural Networks
Spiking neural networks are nature's versatile solution to fault-tolerant and
energy efficient signal processing. To translate these benefits into hardware,
a growing number of neuromorphic spiking neural network processors attempt to
emulate biological neural networks. These developments have created an imminent
need for methods and tools to enable such systems to solve real-world signal
processing problems. Like conventional neural networks, spiking neural networks
can be trained on real, domain specific data. However, their training requires
overcoming a number of challenges linked to their binary and dynamical nature.
This article elucidates step-by-step the problems typically encountered when
training spiking neural networks, and guides the reader through the key
concepts of synaptic plasticity and data-driven learning in the spiking
setting. To that end, it gives an overview of existing approaches and provides
an introduction to surrogate gradient methods, specifically, as a particularly
flexible and efficient method to overcome the aforementioned challenges
Recommended from our members
Optimization of output spike train encoding for a spiking neuron based on its spatiotemporal input pattern
A common learning task for a spiking neuron is to map a spatiotemporal input pattern to a target output spike train. There is no prescribed method for selection of the target output spike train. However, the precise spiking pattern of the target output spike train (output encoding) can affect the learning performance of the spiking neuron. Therefore, systematic methods of finding the optimum spiking pattern for a target output spike train that can be learned by spiking neurons are needed. Here, a method is proposed to adaptively adjust an initial sub-optimal output encoding during different learning epochs to find the optimal output encoding. A time varying value of a local event called a spike trace is used to calculate the amount of a required adjustment. The Remote Supervised Method (ReSuMe) learning algorithm is used to train the weights, and the proposed method is used for finding optimized output encoding (optimized desired spikes). Experimental results show that optimizing the output encoding during the learning phase increases the accuracy. The proposed method was applied to find optimized output encoding in classification tasks and the results revealed improvements up to 16.5% in accuracy compared to when using the non-adapted method. It also increases the accuracy in a classification task from 90% to 100%