132 research outputs found
Training Deep Spiking Neural Networks Using Backpropagation
Deep spiking neural networks (SNNs) hold the potential for improving the latency and energy efficiency of deep neural networks through data-driven event-based computation. However, training such networks is difficult due to the non-differentiable nature of spike events. In this paper, we introduce a novel technique, which treats the membrane potentials of spiking neurons as differentiable signals, where discontinuities at spike times are considered as noise. This enables an error backpropagation mechanism for deep SNNs that follows the same principles as in conventional deep networks, but works directly on spike signals and membrane potentials. Compared with previous methods relying on indirect training and conversion, our technique has the potential to capture the statistics of spikes more precisely. We evaluate the proposed framework on artificially generated events from the original MNIST handwritten digit benchmark, and also on the N-MNIST benchmark recorded with an event-based dynamic vision sensor, in which the proposed method reduces the error rate by a factor of more than three compared to the best previous SNN, and also achieves a higher accuracy than a conventional convolutional neural network (CNN) trained and tested on the same data. We demonstrate in the context of the MNIST task that thanks to their event-driven operation, deep SNNs (both fully connected and convolutional) trained with our method achieve accuracy equivalent with conventional neural networks. In the N-MNIST example, equivalent accuracy is achieved with about five times fewer computational operations
Neuroinspired unsupervised learning and pruning with subquantum CBRAM arrays.
Resistive RAM crossbar arrays offer an attractive solution to minimize off-chip data transfer and parallelize on-chip computations for neural networks. Here, we report a hardware/software co-design approach based on low energy subquantum conductive bridging RAM (CBRAMĀ®) devices and a network pruning technique to reduce network level energy consumption. First, we demonstrate low energy subquantum CBRAM devices exhibiting gradual switching characteristics important for implementing weight updates in hardware during unsupervised learning. Then we develop a network pruning algorithm that can be employed during training, different from previous network pruning approaches applied for inference only. Using a 512 kbit subquantum CBRAM array, we experimentally demonstrate high recognition accuracy on the MNIST dataset for digital implementation of unsupervised learning. Our hardware/software co-design approach can pave the way towards resistive memory based neuro-inspired systems that can autonomously learn and process information in power-limited settings
Algorithm and Hardware Design of Discrete-Time Spiking Neural Networks Based on Back Propagation with Binary Activations
We present a new back propagation based training algorithm for discrete-time
spiking neural networks (SNN). Inspired by recent deep learning algorithms on
binarized neural networks, binary activation with a straight-through gradient
estimator is used to model the leaky integrate-fire spiking neuron, overcoming
the difficulty in training SNNs using back propagation. Two SNN training
algorithms are proposed: (1) SNN with discontinuous integration, which is
suitable for rate-coded input spikes, and (2) SNN with continuous integration,
which is more general and can handle input spikes with temporal information.
Neuromorphic hardware designed in 40nm CMOS exploits the spike sparsity and
demonstrates high classification accuracy (>98% on MNIST) and low energy
(48.4-773 nJ/image).Comment: 2017 IEEE Biomedical Circuits and Systems (BioCAS
Training Probabilistic Spiking Neural Networks with First-to-spike Decoding
Third-generation neural networks, or Spiking Neural Networks (SNNs), aim at
harnessing the energy efficiency of spike-domain processing by building on
computing elements that operate on, and exchange, spikes. In this paper, the
problem of training a two-layer SNN is studied for the purpose of
classification, under a Generalized Linear Model (GLM) probabilistic neural
model that was previously considered within the computational neuroscience
literature. Conventional classification rules for SNNs operate offline based on
the number of output spikes at each output neuron. In contrast, a novel
training method is proposed here for a first-to-spike decoding rule, whereby
the SNN can perform an early classification decision once spike firing is
detected at an output neuron. Numerical results bring insights into the optimal
parameter selection for the GLM neuron and on the accuracy-complexity trade-off
performance of conventional and first-to-spike decoding.Comment: A shorter version will be published on Proc. IEEE ICASSP 201
Fast and Efficient Information Transmission with Burst Spikes in Deep Spiking Neural Networks
The spiking neural networks (SNNs) are considered as one of the most
promising artificial neural networks due to their energy efficient computing
capability. Recently, conversion of a trained deep neural network to an SNN has
improved the accuracy of deep SNNs. However, most of the previous studies have
not achieved satisfactory results in terms of inference speed and energy
efficiency. In this paper, we propose a fast and energy-efficient information
transmission method with burst spikes and hybrid neural coding scheme in deep
SNNs. Our experimental results showed the proposed methods can improve
inference energy efficiency and shorten the latency.Comment: Accepted to DAC 201
Training Dynamic Exponential Family Models with Causal and Lateral Dependencies for Generalized Neuromorphic Computing
Neuromorphic hardware platforms, such as Intel's Loihi chip, support the
implementation of Spiking Neural Networks (SNNs) as an energy-efficient
alternative to Artificial Neural Networks (ANNs). SNNs are networks of neurons
with internal analogue dynamics that communicate by means of binary time
series. In this work, a probabilistic model is introduced for a generalized
set-up in which the synaptic time series can take values in an arbitrary
alphabet and are characterized by both causal and instantaneous statistical
dependencies. The model, which can be considered as an extension of exponential
family harmoniums to time series, is introduced by means of a hybrid
directed-undirected graphical representation. Furthermore, distributed learning
rules are derived for Maximum Likelihood and Bayesian criteria under the
assumption of fully observed time series in the training set.Comment: Published in IEEE ICASSP 2019. Author's Accepted Manuscrip
- ā¦