23,331 research outputs found
A Deterministic and Generalized Framework for Unsupervised Learning with Restricted Boltzmann Machines
Restricted Boltzmann machines (RBMs) are energy-based neural-networks which
are commonly used as the building blocks for deep architectures neural
architectures. In this work, we derive a deterministic framework for the
training, evaluation, and use of RBMs based upon the Thouless-Anderson-Palmer
(TAP) mean-field approximation of widely-connected systems with weak
interactions coming from spin-glass theory. While the TAP approach has been
extensively studied for fully-visible binary spin systems, our construction is
generalized to latent-variable models, as well as to arbitrarily distributed
real-valued spin systems with bounded support. In our numerical experiments, we
demonstrate the effective deterministic training of our proposed models and are
able to show interesting features of unsupervised learning which could not be
directly observed with sampling. Additionally, we demonstrate how to utilize
our TAP-based framework for leveraging trained RBMs as joint priors in
denoising problems
Attention in a family of Boltzmann machines emerging from modern Hopfield networks
Hopfield networks and Boltzmann machines (BMs) are fundamental energy-based
neural network models. Recent studies on modern Hopfield networks have broaden
the class of energy functions and led to a unified perspective on general
Hopfield networks including an attention module. In this letter, we consider
the BM counterparts of modern Hopfield networks using the associated energy
functions, and study their salient properties from a trainability perspective.
In particular, the energy function corresponding to the attention module
naturally introduces a novel BM, which we refer to as attentional BM (AttnBM).
We verify that AttnBM has a tractable likelihood function and gradient for a
special case and is easy to train. Moreover, we reveal the hidden connections
between AttnBM and some single-layer models, namely the Gaussian--Bernoulli
restricted BM and denoising autoencoder with softmax units. We also investigate
BMs introduced by other energy functions, and in particular, observe that the
energy function of dense associative memory models gives BMs belonging to
Exponential Family Harmoniums.Comment: 12 pages, 1 figur
Metric-Free Natural Gradient for Joint-Training of Boltzmann Machines
This paper introduces the Metric-Free Natural Gradient (MFNG) algorithm for
training Boltzmann Machines. Similar in spirit to the Hessian-Free method of
Martens [8], our algorithm belongs to the family of truncated Newton methods
and exploits an efficient matrix-vector product to avoid explicitely storing
the natural gradient metric . This metric is shown to be the expected second
derivative of the log-partition function (under the model distribution), or
equivalently, the variance of the vector of partial derivatives of the energy
function. We evaluate our method on the task of joint-training a 3-layer Deep
Boltzmann Machine and show that MFNG does indeed have faster per-epoch
convergence compared to Stochastic Maximum Likelihood with centering, though
wall-clock performance is currently not competitive
- …