73 research outputs found
Approximate Message Passing with Restricted Boltzmann Machine Priors
Approximate Message Passing (AMP) has been shown to be an excellent
statistical approach to signal inference and compressed sensing problem. The
AMP framework provides modularity in the choice of signal prior; here we
propose a hierarchical form of the Gauss-Bernouilli prior which utilizes a
Restricted Boltzmann Machine (RBM) trained on the signal support to push
reconstruction performance beyond that of simple iid priors for signals whose
support can be well represented by a trained binary RBM. We present and analyze
two methods of RBM factorization and demonstrate how these affect signal
reconstruction performance within our proposed algorithm. Finally, using the
MNIST handwritten digit dataset, we show experimentally that using an RBM
allows AMP to approach oracle-support performance
A Deterministic and Generalized Framework for Unsupervised Learning with Restricted Boltzmann Machines
Restricted Boltzmann machines (RBMs) are energy-based neural-networks which
are commonly used as the building blocks for deep architectures neural
architectures. In this work, we derive a deterministic framework for the
training, evaluation, and use of RBMs based upon the Thouless-Anderson-Palmer
(TAP) mean-field approximation of widely-connected systems with weak
interactions coming from spin-glass theory. While the TAP approach has been
extensively studied for fully-visible binary spin systems, our construction is
generalized to latent-variable models, as well as to arbitrarily distributed
real-valued spin systems with bounded support. In our numerical experiments, we
demonstrate the effective deterministic training of our proposed models and are
able to show interesting features of unsupervised learning which could not be
directly observed with sampling. Additionally, we demonstrate how to utilize
our TAP-based framework for leveraging trained RBMs as joint priors in
denoising problems
Inferring Sparsity: Compressed Sensing using Generalized Restricted Boltzmann Machines
In this work, we consider compressed sensing reconstruction from
measurements of -sparse structured signals which do not possess a writable
correlation model. Assuming that a generative statistical model, such as a
Boltzmann machine, can be trained in an unsupervised manner on example signals,
we demonstrate how this signal model can be used within a Bayesian framework of
signal reconstruction. By deriving a message-passing inference for general
distribution restricted Boltzmann machines, we are able to integrate these
inferred signal models into approximate message passing for compressed sensing
reconstruction. Finally, we show for the MNIST dataset that this approach can
be very effective, even for .Comment: IEEE Information Theory Workshop, 201
Unsupervised Generative Modeling Using Matrix Product States
Generative modeling, which learns joint probability distribution from data
and generates samples according to it, is an important task in machine learning
and artificial intelligence. Inspired by probabilistic interpretation of
quantum physics, we propose a generative model using matrix product states,
which is a tensor network originally proposed for describing (particularly
one-dimensional) entangled quantum states. Our model enjoys efficient learning
analogous to the density matrix renormalization group method, which allows
dynamically adjusting dimensions of the tensors and offers an efficient direct
sampling approach for generative tasks. We apply our method to generative
modeling of several standard datasets including the Bars and Stripes, random
binary patterns and the MNIST handwritten digits to illustrate the abilities,
features and drawbacks of our model over popular generative models such as
Hopfield model, Boltzmann machines and generative adversarial networks. Our
work sheds light on many interesting directions of future exploration on the
development of quantum-inspired algorithms for unsupervised machine learning,
which are promisingly possible to be realized on quantum devices.Comment: 11 pages, 12 figures (not including the TNs) GitHub Page:
https://congzlwag.github.io/UnsupGenModbyMPS
Mean-field message-passing equations in the Hopfield model and its generalizations
International audienceMotivated by recent progress in using restricted Boltzmann machines as preprocess-ing algorithms for deep neural network, we revisit the mean-field equations (belief-propagation and TAP equations) in the best understood such machine, namely the Hopfield model of neural networks, and we explicit how they can be used as iterative message-passing algorithms, providing a fast method to compute the local polariza-tions of neurons. In the "retrieval phase" where neurons polarize in the direction of one memorized pattern, we point out a major difference between the belief propagation and TAP equations : the set of belief propagation equations depends on the pattern which is retrieved, while one can use a unique set of TAP equations. This makes the latter method much better suited for applications in the learning process of restricted Boltzmann machines. In the case where the patterns memorized in the Hopfield model are not independent, but are correlated through a combinatorial structure, we show that the TAP equations have to be modified. This modification can be seen either as an alteration of the reaction term in TAP equations, or, more interestingly, as the consequence of message passing on a graphical model with several hidden layers, where the number of hidden layers depends on the depth of the correlations in the memorized patterns. This layered structure is actually necessary when one deals with more general restricted Boltzmann machines
Variational Cumulant Expansions for Intractable Distributions
Intractable distributions present a common difficulty in inference within the
probabilistic knowledge representation framework and variational methods have
recently been popular in providing an approximate solution. In this article, we
describe a perturbational approach in the form of a cumulant expansion which,
to lowest order, recovers the standard Kullback-Leibler variational bound.
Higher-order terms describe corrections on the variational approach without
incurring much further computational cost. The relationship to other
perturbational approaches such as TAP is also elucidated. We demonstrate the
method on a particular class of undirected graphical models, Boltzmann
machines, for which our simulation results confirm improved accuracy and
enhanced stability during learning
Neural Networks retrieving Boolean patterns in a sea of Gaussian ones
Restricted Boltzmann Machines are key tools in Machine Learning and are
described by the energy function of bipartite spin-glasses. From a statistical
mechanical perspective, they share the same Gibbs measure of Hopfield networks
for associative memory. In this equivalence, weights in the former play as
patterns in the latter. As Boltzmann machines usually require real weights to
be trained with gradient descent like methods, while Hopfield networks
typically store binary patterns to be able to retrieve, the investigation of a
mixed Hebbian network, equipped with both real (e.g., Gaussian) and discrete
(e.g., Boolean) patterns naturally arises. We prove that, in the challenging
regime of a high storage of real patterns, where retrieval is forbidden, an
extra load of Boolean patterns can still be retrieved, as long as the ratio
among the overall load and the network size does not exceed a critical
threshold, that turns out to be the same of the standard
Amit-Gutfreund-Sompolinsky theory. Assuming replica symmetry, we study the case
of a low load of Boolean patterns combining the stochastic stability and
Hamilton-Jacobi interpolating techniques. The result can be extended to the
high load by a non rigorous but standard replica computation argument.Comment: 16 pages, 1 figur
Unsupervised hierarchical clustering using the learning dynamics of RBMs
Datasets in the real world are often complex and to some degree hierarchical,
with groups and sub-groups of data sharing common characteristics at different
levels of abstraction. Understanding and uncovering the hidden structure of
these datasets is an important task that has many practical applications. To
address this challenge, we present a new and general method for building
relational data trees by exploiting the learning dynamics of the Restricted
Boltzmann Machine (RBM). Our method is based on the mean-field approach,
derived from the Plefka expansion, and developed in the context of disordered
systems. It is designed to be easily interpretable. We tested our method in an
artificially created hierarchical dataset and on three different real-world
datasets (images of digits, mutations in the human genome, and a homologous
family of proteins). The method is able to automatically identify the
hierarchical structure of the data. This could be useful in the study of
homologous protein sequences, where the relationships between proteins are
critical for understanding their function and evolution.Comment: Version accepted in Physical Review
Fast and Functional Structured Data Generators Rooted in Out-of-Equilibrium Physics
In this study, we address the challenge of using energy-based models to
produce high-quality, label-specific data in complex structured datasets, such
as population genetics, RNA or protein sequences data. Traditional training
methods encounter difficulties due to inefficient Markov chain Monte Carlo
mixing, which affects the diversity of synthetic data and increases generation
times. To address these issues, we use a novel training algorithm that exploits
non-equilibrium effects. This approach, applied on the Restricted Boltzmann
Machine, improves the model's ability to correctly classify samples and
generate high-quality synthetic data in only a few sampling steps. The
effectiveness of this method is demonstrated by its successful application to
four different types of data: handwritten digits, mutations of human genomes
classified by continental origin, functionally characterized sequences of an
enzyme protein family, and homologous RNA sequences from specific taxonomies.Comment: 15 page
- âŠ