39,633 research outputs found
Deep supervised learning using local errors
Error backpropagation is a highly effective mechanism for learning
high-quality hierarchical features in deep networks. Updating the features or
weights in one layer, however, requires waiting for the propagation of error
signals from higher layers. Learning using delayed and non-local errors makes
it hard to reconcile backpropagation with the learning mechanisms observed in
biological neural networks as it requires the neurons to maintain a memory of
the input long enough until the higher-layer errors arrive. In this paper, we
propose an alternative learning mechanism where errors are generated locally in
each layer using fixed, random auxiliary classifiers. Lower layers could thus
be trained independently of higher layers and training could either proceed
layer by layer, or simultaneously in all layers using local error information.
We address biological plausibility concerns such as weight symmetry
requirements and show that the proposed learning mechanism based on fixed,
broad, and random tuning of each neuron to the classification categories
outperforms the biologically-motivated feedback alignment learning technique on
the MNIST, CIFAR10, and SVHN datasets, approaching the performance of standard
backpropagation. Our approach highlights a potential biological mechanism for
the supervised, or task-dependent, learning of feature hierarchies. In
addition, we show that it is well suited for learning deep networks in custom
hardware where it can drastically reduce memory traffic and data communication
overheads
Deep learning with asymmetric connections and Hebbian updates
We show that deep networks can be trained using Hebbian updates yielding
similar performance to ordinary back-propagation on challenging image datasets.
To overcome the unrealistic symmetry in connections between layers, implicit in
back-propagation, the feedback weights are separate from the feedforward
weights. The feedback weights are also updated with a local rule, the same as
the feedforward weights - a weight is updated solely based on the product of
activity of the units it connects. With fixed feedback weights as proposed in
Lillicrap et. al (2016) performance degrades quickly as the depth of the
network increases. If the feedforward and feedback weights are initialized with
the same values, as proposed in Zipser and Rumelhart (1990), they remain the
same throughout training thus precisely implementing back-propagation. We show
that even when the weights are initialized differently and at random, and the
algorithm is no longer performing back-propagation, performance is comparable
on challenging datasets. We also propose a cost function whose derivative can
be represented as a local Hebbian update on the last layer. Convolutional
layers are updated with tied weights across space, which is not biologically
plausible. We show that similar performance is achieved with untied layers,
also known as locally connected layers, corresponding to the connectivity
implied by the convolutional layers, but where weights are untied and updated
separately. In the linear case we show theoretically that the convergence of
the error to zero is accelerated by the update of the feedback weights
Direct Feedback Alignment with Sparse Connections for Local Learning
Recent advances in deep neural networks (DNNs) owe their success to training
algorithms that use backpropagation and gradient-descent. Backpropagation,
while highly effective on von Neumann architectures, becomes inefficient when
scaling to large networks. Commonly referred to as the weight transport
problem, each neuron's dependence on the weights and errors located deeper in
the network require exhaustive data movement which presents a key problem in
enhancing the performance and energy-efficiency of machine-learning hardware.
In this work, we propose a bio-plausible alternative to backpropagation drawing
from advances in feedback alignment algorithms in which the error computation
at a single synapse reduces to the product of three scalar values. Using a
sparse feedback matrix, we show that a neuron needs only a fraction of the
information previously used by the feedback alignment algorithms. Consequently,
memory and compute can be partitioned and distributed whichever way produces
the most efficient forward pass so long as a single error can be delivered to
each neuron. Our results show orders of magnitude improvement in data movement
and improvement in multiply-and-accumulate operations over
backpropagation. Like previous work, we observe that any variant of feedback
alignment suffers significant losses in classification accuracy on deep
convolutional neural networks. By transferring trained convolutional layers and
training the fully connected layers using direct feedback alignment, we
demonstrate that direct feedback alignment can obtain results competitive with
backpropagation. Furthermore, we observe that using an extremely sparse
feedback matrix, rather than a dense one, results in a small accuracy drop
while yielding hardware advantages. All the code and results are available
under https://github.com/bcrafton/ssdfa.Comment: 15 pages, 8 figure
SuperSpike: Supervised learning in multi-layer spiking neural networks
A vast majority of computation in the brain is performed by spiking neural
networks. Despite the ubiquity of such spiking, we currently lack an
understanding of how biological spiking neural circuits learn and compute
in-vivo, as well as how we can instantiate such capabilities in artificial
spiking circuits in-silico. Here we revisit the problem of supervised learning
in temporally coding multi-layer spiking neural networks. First, by using a
surrogate gradient approach, we derive SuperSpike, a nonlinear voltage-based
three factor learning rule capable of training multi-layer networks of
deterministic integrate-and-fire neurons to perform nonlinear computations on
spatiotemporal spike patterns. Second, inspired by recent results on feedback
alignment, we compare the performance of our learning rule under different
credit assignment strategies for propagating output errors to hidden units.
Specifically, we test uniform, symmetric and random feedback, finding that
simpler tasks can be solved with any type of feedback, while more complex tasks
require symmetric feedback. In summary, our results open the door to obtaining
a better scientific understanding of learning and computation in spiking neural
networks by advancing our ability to train them to solve nonlinear problems
involving transformations between different spatiotemporal spike-time patterns
Biologically plausible deep learning -- but how far can we go with shallow networks?
Training deep neural networks with the error backpropagation algorithm is
considered implausible from a biological perspective. Numerous recent
publications suggest elaborate models for biologically plausible variants of
deep learning, typically defining success as reaching around 98% test accuracy
on the MNIST data set. Here, we investigate how far we can go on digit (MNIST)
and object (CIFAR10) classification with biologically plausible, local learning
rules in a network with one hidden layer and a single readout layer. The hidden
layer weights are either fixed (random or random Gabor filters) or trained with
unsupervised methods (PCA, ICA or Sparse Coding) that can be implemented by
local learning rules. The readout layer is trained with a supervised, local
learning rule. We first implement these models with rate neurons. This
comparison reveals, first, that unsupervised learning does not lead to better
performance than fixed random projections or Gabor filters for large hidden
layers. Second, networks with localized receptive fields perform significantly
better than networks with all-to-all connectivity and can reach backpropagation
performance on MNIST. We then implement two of the networks - fixed, localized,
random & random Gabor filters in the hidden layer - with spiking leaky
integrate-and-fire neurons and spike timing dependent plasticity to train the
readout layer. These spiking models achieve > 98.2% test accuracy on MNIST,
which is close to the performance of rate networks with one hidden layer
trained with backpropagation. The performance of our shallow network models is
comparable to most current biologically plausible models of deep learning.
Furthermore, our results with a shallow spiking network provide an important
reference and suggest the use of datasets other than MNIST for testing the
performance of future models of biologically plausible deep learning.Comment: 14 pages, 4 figure
- …