4,214 research outputs found
Recurrent backpropagation and the dynamical approach to adaptive neural computation
Error backpropagation in feedforward neural network models is a popular learning algorithm that has its roots in nonlinear estimation and optimization. It is being used routinely to calculate error gradients in nonlinear systems with hundreds of thousands of parameters. However, the classical architecture for backpropagation has severe restrictions. The extension of backpropagation to networks with recurrent connections will be reviewed. It is now possible to efficiently compute the error gradients for networks that have temporal dynamics, which opens applications to a host of problems in systems identification and control
Deep learning with asymmetric connections and Hebbian updates
We show that deep networks can be trained using Hebbian updates yielding
similar performance to ordinary back-propagation on challenging image datasets.
To overcome the unrealistic symmetry in connections between layers, implicit in
back-propagation, the feedback weights are separate from the feedforward
weights. The feedback weights are also updated with a local rule, the same as
the feedforward weights - a weight is updated solely based on the product of
activity of the units it connects. With fixed feedback weights as proposed in
Lillicrap et. al (2016) performance degrades quickly as the depth of the
network increases. If the feedforward and feedback weights are initialized with
the same values, as proposed in Zipser and Rumelhart (1990), they remain the
same throughout training thus precisely implementing back-propagation. We show
that even when the weights are initialized differently and at random, and the
algorithm is no longer performing back-propagation, performance is comparable
on challenging datasets. We also propose a cost function whose derivative can
be represented as a local Hebbian update on the last layer. Convolutional
layers are updated with tied weights across space, which is not biologically
plausible. We show that similar performance is achieved with untied layers,
also known as locally connected layers, corresponding to the connectivity
implied by the convolutional layers, but where weights are untied and updated
separately. In the linear case we show theoretically that the convergence of
the error to zero is accelerated by the update of the feedback weights
Learning long-range spatial dependencies with horizontal gated-recurrent units
Progress in deep learning has spawned great successes in many engineering
applications. As a prime example, convolutional neural networks, a type of
feedforward neural networks, are now approaching -- and sometimes even
surpassing -- human accuracy on a variety of visual recognition tasks. Here,
however, we show that these neural networks and their recent extensions
struggle in recognition tasks where co-dependent visual features must be
detected over long spatial ranges. We introduce the horizontal gated-recurrent
unit (hGRU) to learn intrinsic horizontal connections -- both within and across
feature columns. We demonstrate that a single hGRU layer matches or outperforms
all tested feedforward hierarchical baselines including state-of-the-art
architectures which have orders of magnitude more free parameters. We further
discuss the biological plausibility of the hGRU in comparison to anatomical
data from the visual cortex as well as human behavioral data on a classic
contour detection task.Comment: Published at NeurIPS 2018
https://papers.nips.cc/paper/7300-learning-long-range-spatial-dependencies-with-horizontal-gated-recurrent-unit
- …