5 research outputs found
PyTorch-Hebbian : facilitating local learning in a deep learning framework
Recently, unsupervised local learning, based on Hebb's idea that change in
synaptic efficacy depends on the activity of the pre- and postsynaptic neuron
only, has shown potential as an alternative training mechanism to
backpropagation. Unfortunately, Hebbian learning remains experimental and
rarely makes it way into standard deep learning frameworks. In this work, we
investigate the potential of Hebbian learning in the context of standard deep
learning workflows. To this end, a framework for thorough and systematic
evaluation of local learning rules in existing deep learning pipelines is
proposed. Using this framework, the potential of Hebbian learned feature
extractors for image classification is illustrated. In particular, the
framework is used to expand the Krotov-Hopfield learning rule to standard
convolutional neural networks without sacrificing accuracy compared to
end-to-end backpropagation. The source code is available at
https://github.com/Joxis/pytorch-hebbian.Comment: Presented as a poster at the NeurIPS 2020 Beyond Backpropagation
worksho
Convergence guarantees for forward gradient descent in the linear regression model
Renewed interest in the relationship between artificial and biological neural networks motivates the study of gradient-free methods. Considering the linear regression model with random design, we theoretically analyze in this work the biologically motivated (weight-perturbed) forward gradient scheme that is based on random linear combination of the gradient. If d denotes the number of parameters and k the number of samples, we prove that the mean squared error of this method converges for k≳d2log(d) with rate d2log(d)/k. Compared to the dimension dependence d for stochastic gradient descent, an additional factor dlog(d) occurs.</p