13 research outputs found
SuperSpike: Supervised learning in multi-layer spiking neural networks
A vast majority of computation in the brain is performed by spiking neural
networks. Despite the ubiquity of such spiking, we currently lack an
understanding of how biological spiking neural circuits learn and compute
in-vivo, as well as how we can instantiate such capabilities in artificial
spiking circuits in-silico. Here we revisit the problem of supervised learning
in temporally coding multi-layer spiking neural networks. First, by using a
surrogate gradient approach, we derive SuperSpike, a nonlinear voltage-based
three factor learning rule capable of training multi-layer networks of
deterministic integrate-and-fire neurons to perform nonlinear computations on
spatiotemporal spike patterns. Second, inspired by recent results on feedback
alignment, we compare the performance of our learning rule under different
credit assignment strategies for propagating output errors to hidden units.
Specifically, we test uniform, symmetric and random feedback, finding that
simpler tasks can be solved with any type of feedback, while more complex tasks
require symmetric feedback. In summary, our results open the door to obtaining
a better scientific understanding of learning and computation in spiking neural
networks by advancing our ability to train them to solve nonlinear problems
involving transformations between different spatiotemporal spike-time patterns
Spiking PointNet: Spiking Neural Networks for Point Clouds
Recently, Spiking Neural Networks (SNNs), enjoying extreme energy efficiency,
have drawn much research attention on 2D visual recognition and shown gradually
increasing application potential. However, it still remains underexplored
whether SNNs can be generalized to 3D recognition. To this end, we present
Spiking PointNet in the paper, the first spiking neural model for efficient
deep learning on point clouds. We discover that the two huge obstacles limiting
the application of SNNs in point clouds are: the intrinsic optimization
obstacle of SNNs that impedes the training of a big spiking model with large
time steps, and the expensive memory and computation cost of PointNet that
makes training a big spiking point model unrealistic. To solve the problems
simultaneously, we present a trained-less but learning-more paradigm for
Spiking PointNet with theoretical justifications and in-depth experimental
analysis. In specific, our Spiking PointNet is trained with only a single time
step but can obtain better performance with multiple time steps inference,
compared to the one trained directly with multiple time steps. We conduct
various experiments on ModelNet10, ModelNet40 to demonstrate the effectiveness
of Spiking PointNet. Notably, our Spiking PointNet even can outperform its ANN
counterpart, which is rare in the SNN field thus providing a potential research
direction for the following work. Moreover, Spiking PointNet shows impressive
speedup and storage saving in the training phase.Comment: Accepted by NeurIP
Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networks
The emergence of brain-inspired neuromorphic computing as a paradigm for edge
AI is motivating the search for high-performance and efficient spiking neural
networks to run on this hardware. However, compared to classical neural
networks in deep learning, current spiking neural networks lack competitive
performance in compelling areas. Here, for sequential and streaming tasks, we
demonstrate how a novel type of adaptive spiking recurrent neural network
(SRNN) is able to achieve state-of-the-art performance compared to other
spiking neural networks and almost reach or exceed the performance of classical
recurrent neural networks (RNNs) while exhibiting sparse activity. From this,
we calculate a 100x energy improvement for our SRNNs over classical RNNs on
the harder tasks. To achieve this, we model standard and adaptive
multiple-timescale spiking neurons as self-recurrent neural units, and leverage
surrogate gradients and auto-differentiation in the PyTorch Deep Learning
framework to efficiently implement backpropagation-through-time, including
learning of the important spiking neuron parameters to adapt our spiking
neurons to the tasks.Comment: 11 pages,5 figure
Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networks
The emergence of brain-inspired neuromorphic computing as a paradigm for edge AI is motivating the search for high-performance and efficient spiking neural networks to run on this hardware. However, compared to classical neural networks in deep learning, current spiking neural networks lack competitive performance in compelling areas. Here, for sequential and streaming tasks, we demonstrate how a novel type of adaptive spiking recurrent neural network (SRNN) is able to achieve state-of-the-art performance compared to other spiking neural networks and almost reach or exceed the performance of classical recurrent neural networks (RNNs) while exhibiting sparse activity. From this, we calculate a >100x energy improvement for our SRNNs over classical RNNs on the harder tasks. To achieve this, we model standard and adaptive multiple-timescale spiking neurons as self-recurrent neural units, and leverage surrogate gradients and auto-differentiation in the PyTorch Deep Learning framework to efficiently implement backpropagation-through-time, including learning of the important spiking neuron parameters to adapt our spiking neurons to the tasks
Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks
Inspired by detailed modelling of biological neurons, spiking neural networks (SNNs) are investigated as biologically plausible and high-performance models of neural computation. The sparse and binary communication between spiking neurons potentially enables powerful and energy-efficient neural networks. The performance of SNNs, however, has remained lacking compared with artificial neural networks. Here we demonstrate how an activity-regularizing surrogate gradient combined with recurrent networks of tunable and adaptive spiking neurons yields the state of the art for SNNs on challenging benchmarks in the time domain, such as speech and gesture recognition. This also exceeds the performance of standard classical recurrent neural networks and approaches that of the best modern artificial neural networks. As these SNNs exhibit sparse spiking, we show that they are theoretically one to three orders of magnitude more computationally efficient compared to recurrent neural networks with similar performance. Together, this positions SNNs as an attractive solution for AI hardware implementations
Surrogate Gradient Learning in Spiking Neural Networks
Spiking neural networks are nature's versatile solution to fault-tolerant and
energy efficient signal processing. To translate these benefits into hardware,
a growing number of neuromorphic spiking neural network processors attempt to
emulate biological neural networks. These developments have created an imminent
need for methods and tools to enable such systems to solve real-world signal
processing problems. Like conventional neural networks, spiking neural networks
can be trained on real, domain specific data. However, their training requires
overcoming a number of challenges linked to their binary and dynamical nature.
This article elucidates step-by-step the problems typically encountered when
training spiking neural networks, and guides the reader through the key
concepts of synaptic plasticity and data-driven learning in the spiking
setting. To that end, it gives an overview of existing approaches and provides
an introduction to surrogate gradient methods, specifically, as a particularly
flexible and efficient method to overcome the aforementioned challenges
Direct Learning-Based Deep Spiking Neural Networks: A Review
The spiking neural network (SNN), as a promising brain-inspired computational
model with binary spike information transmission mechanism, rich
spatially-temporal dynamics, and event-driven characteristics, has received
extensive attention. However, its intricately discontinuous spike mechanism
brings difficulty to the optimization of the deep SNN. Since the surrogate
gradient method can greatly mitigate the optimization difficulty and shows
great potential in directly training deep SNNs, a variety of direct
learning-based deep SNN works have been proposed and achieved satisfying
progress in recent years. In this paper, we present a comprehensive survey of
these direct learning-based deep SNN works, mainly categorized into accuracy
improvement methods, efficiency improvement methods, and temporal dynamics
utilization methods. In addition, we also divide these categorizations into
finer granularities further to better organize and introduce them. Finally, the
challenges and trends that may be faced in future research are prospected.Comment: Accepted by Frontiers in Neuroscienc