70,104 research outputs found
Recurrent backpropagation and the dynamical approach to adaptive neural computation
Error backpropagation in feedforward neural network models is a popular learning algorithm that has its roots in nonlinear estimation and optimization. It is being used routinely to calculate error gradients in nonlinear systems with hundreds of thousands of parameters. However, the classical architecture for backpropagation has severe restrictions. The extension of backpropagation to networks with recurrent connections will be reviewed. It is now possible to efficiently compute the error gradients for networks that have temporal dynamics, which opens applications to a host of problems in systems identification and control
Unbiasing Truncated Backpropagation Through Time
Truncated Backpropagation Through Time (truncated BPTT) is a widespread
method for learning recurrent computational graphs. Truncated BPTT keeps the
computational benefits of Backpropagation Through Time (BPTT) while relieving
the need for a complete backtrack through the whole data sequence at every
step. However, truncation favors short-term dependencies: the gradient estimate
of truncated BPTT is biased, so that it does not benefit from the convergence
guarantees from stochastic gradient theory. We introduce Anticipated Reweighted
Truncated Backpropagation (ARTBP), an algorithm that keeps the computational
benefits of truncated BPTT, while providing unbiasedness. ARTBP works by using
variable truncation lengths together with carefully chosen compensation factors
in the backpropagation equation. We check the viability of ARTBP on two tasks.
First, a simple synthetic task where careful balancing of temporal dependencies
at different scales is needed: truncated BPTT displays unreliable performance,
and in worst case scenarios, divergence, while ARTBP converges reliably.
Second, on Penn Treebank character-level language modelling, ARTBP slightly
outperforms truncated BPTT
Analog hardware for delta-backpropagation neural networks
This is a fully parallel analog backpropagation learning processor which comprises a plurality of programmable resistive memory elements serving as synapse connections whose values can be weighted during learning with buffer amplifiers, summing circuits, and sample-and-hold circuits arranged in a plurality of neuron layers in accordance with delta-backpropagation algorithms modified so as to control weight changes due to circuit drift
Backpropagation training in adaptive quantum networks
We introduce a robust, error-tolerant adaptive training algorithm for
generalized learning paradigms in high-dimensional superposed quantum networks,
or \emph{adaptive quantum networks}. The formalized procedure applies standard
backpropagation training across a coherent ensemble of discrete topological
configurations of individual neural networks, each of which is formally merged
into appropriate linear superposition within a predefined, decoherence-free
subspace. Quantum parallelism facilitates simultaneous training and revision of
the system within this coherent state space, resulting in accelerated
convergence to a stable network attractor under consequent iteration of the
implemented backpropagation algorithm. Parallel evolution of linear superposed
networks incorporating backpropagation training provides quantitative,
numerical indications for optimization of both single-neuron activation
functions and optimal reconfiguration of whole-network quantum structure.Comment: Talk presented at "Quantum Structures - 2008", Gdansk, Polan
One step backpropagation through time for learning input mapping in reservoir computing applied to speech recognition
Recurrent neural networks are very powerful engines for processing information that is coded in time, however, many problems with common training algorithms, such as Backpropagation Through Time, remain. Because of this, another important learning setup known as Reservoir Computing has appeared in recent years, where one uses an essentially untrained network to perform computations. Though very successful in many applications, using a random network can be quite inefficient when considering the required number of neurons and the associated computational costs. In this paper we introduce a highly simplified version of Backpropagation Through Time by basically truncating the error backpropagation to one step back in time, and we combine this with the classic Reservoir Computing setup using an instantaneous linear readout. We apply this setup to a spoken digit recognition task and show it to give very good results for small networks
- …