8,172 research outputs found
Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation
We introduce Equilibrium Propagation, a learning framework for energy-based
models. It involves only one kind of neural computation, performed in both the
first phase (when the prediction is made) and the second phase of training
(after the target or prediction error is revealed). Although this algorithm
computes the gradient of an objective function just like Backpropagation, it
does not need a special computation or circuit for the second phase, where
errors are implicitly propagated. Equilibrium Propagation shares similarities
with Contrastive Hebbian Learning and Contrastive Divergence while solving the
theoretical issues of both algorithms: our algorithm computes the gradient of a
well defined objective function. Because the objective function is defined in
terms of local perturbations, the second phase of Equilibrium Propagation
corresponds to only nudging the prediction (fixed point, or stationary
distribution) towards a configuration that reduces prediction error. In the
case of a recurrent multi-layer supervised network, the output units are
slightly nudged towards their target in the second phase, and the perturbation
introduced at the output layer propagates backward in the hidden layers. We
show that the signal 'back-propagated' during this second phase corresponds to
the propagation of error derivatives and encodes the gradient of the objective
function, when the synaptic update corresponds to a standard form of
spike-timing dependent plasticity. This work makes it more plausible that a
mechanism similar to Backpropagation could be implemented by brains, since
leaky integrator neural computation performs both inference and error
back-propagation in our model. The only local difference between the two phases
is whether synaptic changes are allowed or not
Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors
Restricted Boltzmann Machines are described by the Gibbs measure of a
bipartite spin glass, which in turn corresponds to the one of a generalised
Hopfield network. This equivalence allows us to characterise the state of these
systems in terms of retrieval capabilities, both at low and high load. We study
the paramagnetic-spin glass and the spin glass-retrieval phase transitions, as
the pattern (i.e. weight) distribution and spin (i.e. unit) priors vary
smoothly from Gaussian real variables to Boolean discrete variables. Our
analysis shows that the presence of a retrieval phase is robust and not
peculiar to the standard Hopfield model with Boolean patterns. The retrieval
region is larger when the pattern entries and retrieval units get more peaked
and, conversely, when the hidden units acquire a broader prior and therefore
have a stronger response to high fields. Moreover, at low load retrieval always
exists below some critical temperature, for every pattern distribution ranging
from the Boolean to the Gaussian case.Comment: 18 pages, 9 figures; typos adde
- …