13,449 research outputs found
Spike-based computation using classical recurrent neural networks
Spiking neural networks are a type of artificial neural networks in which
communication between neurons is only made of events, also called spikes. This
property allows neural networks to make asynchronous and sparse computations
and therefore to drastically decrease energy consumption when run on
specialized hardware. However, training such networks is known to be difficult,
mainly due to the non-differentiability of the spike activation, which prevents
the use of classical backpropagation. This is because state-of-the-art spiking
neural networks are usually derived from biologically-inspired neuron models,
to which are applied machine learning methods for training. Nowadays, research
about spiking neural networks focuses on the design of training algorithms
whose goal is to obtain networks that compete with their non-spiking version on
specific tasks. In this paper, we attempt the symmetrical approach: we modify
the dynamics of a well-known, easily trainable type of recurrent neural network
to make it event-based. This new RNN cell, called the Spiking Recurrent Cell,
therefore communicates using events, i.e. spikes, while being completely
differentiable. Vanilla backpropagation can thus be used to train any network
made of such RNN cell. We show that this new network can achieve performance
comparable to other types of spiking networks in the MNIST benchmark and its
variants, the Fashion-MNIST and the Neuromorphic-MNIST. Moreover, we show that
this new cell makes the training of deep spiking networks achievable.Comment: 12 pages, 3 figure
Biologically plausible deep learning -- but how far can we go with shallow networks?
Training deep neural networks with the error backpropagation algorithm is
considered implausible from a biological perspective. Numerous recent
publications suggest elaborate models for biologically plausible variants of
deep learning, typically defining success as reaching around 98% test accuracy
on the MNIST data set. Here, we investigate how far we can go on digit (MNIST)
and object (CIFAR10) classification with biologically plausible, local learning
rules in a network with one hidden layer and a single readout layer. The hidden
layer weights are either fixed (random or random Gabor filters) or trained with
unsupervised methods (PCA, ICA or Sparse Coding) that can be implemented by
local learning rules. The readout layer is trained with a supervised, local
learning rule. We first implement these models with rate neurons. This
comparison reveals, first, that unsupervised learning does not lead to better
performance than fixed random projections or Gabor filters for large hidden
layers. Second, networks with localized receptive fields perform significantly
better than networks with all-to-all connectivity and can reach backpropagation
performance on MNIST. We then implement two of the networks - fixed, localized,
random & random Gabor filters in the hidden layer - with spiking leaky
integrate-and-fire neurons and spike timing dependent plasticity to train the
readout layer. These spiking models achieve > 98.2% test accuracy on MNIST,
which is close to the performance of rate networks with one hidden layer
trained with backpropagation. The performance of our shallow network models is
comparable to most current biologically plausible models of deep learning.
Furthermore, our results with a shallow spiking network provide an important
reference and suggest the use of datasets other than MNIST for testing the
performance of future models of biologically plausible deep learning.Comment: 14 pages, 4 figure
Recommended from our members
A biologically inspired spiking model of visual processing for image feature detection
To enable fast reliable feature matching or tracking in scenes, features need to be discrete and meaningful, and hence edge or corner features, commonly called interest points are often used for this purpose. Experimental research has illustrated that biological vision systems use neuronal circuits to extract particular features such as edges or corners from visual scenes. Inspired by this biological behaviour, this paper proposes a biologically inspired spiking neural network for the purpose of image feature extraction. Standard digital images are processed and converted to spikes in a manner similar to the processing that transforms light into spikes in the retina. Using a hierarchical spiking network, various types of biologically inspired receptive fields are used to extract progressively complex image features. The performance of the network is assessed by examining the repeatability of extracted features with visual results presented using both synthetic and real images
- …