294 research outputs found
Extending Factor Graphs so as to Unify Directed and Undirected Graphical Models
The two most popular types of graphical model are directed models (Bayesian
networks) and undirected models (Markov random fields, or MRFs). Directed and
undirected models offer complementary properties in model construction,
expressing conditional independencies, expressing arbitrary factorizations of
joint distributions, and formulating message-passing inference algorithms. We
show that the strengths of these two representations can be combined in a
single type of graphical model called a 'factor graph'. Every Bayesian network
or MRF can be easily converted to a factor graph that expresses the same
conditional independencies, expresses the same factorization of the joint
distribution, and can be used for probabilistic inference through application
of a single, simple message-passing algorithm. In contrast to chain graphs,
where message-passing is implemented on a hypergraph, message-passing can be
directly implemented on the factor graph. We describe a modified 'Bayes-ball'
algorithm for establishing conditional independence in factor graphs, and we
show that factor graphs form a strict superset of Bayesian networks and MRFs.
In particular, we give an example of a commonly-used 'mixture of experts' model
fragment, whose independencies cannot be represented in a Bayesian network or
an MRF, but can be represented in a factor graph. We finish by giving examples
of real-world problems that are not well suited to representation in Bayesian
networks and MRFs, but are well-suited to representation in factor graphs.Comment: Appears in Proceedings of the Nineteenth Conference on Uncertainty in
Artificial Intelligence (UAI2003
PixelGAN Autoencoders
In this paper, we describe the "PixelGAN autoencoder", a generative
autoencoder in which the generative path is a convolutional autoregressive
neural network on pixels (PixelCNN) that is conditioned on a latent code, and
the recognition path uses a generative adversarial network (GAN) to impose a
prior distribution on the latent code. We show that different priors result in
different decompositions of information between the latent code and the
autoregressive decoder. For example, by imposing a Gaussian distribution as the
prior, we can achieve a global vs. local decomposition, or by imposing a
categorical distribution as the prior, we can disentangle the style and content
information of images in an unsupervised fashion. We further show how the
PixelGAN autoencoder with a categorical prior can be directly used in
semi-supervised settings and achieve competitive semi-supervised classification
results on the MNIST, SVHN and NORB datasets
Winner-Take-All Autoencoders
In this paper, we propose a winner-take-all method for learning hierarchical
sparse representations in an unsupervised fashion. We first introduce
fully-connected winner-take-all autoencoders which use mini-batch statistics to
directly enforce a lifetime sparsity in the activations of the hidden units. We
then propose the convolutional winner-take-all autoencoder which combines the
benefits of convolutional architectures and autoencoders for learning
shift-invariant sparse representations. We describe a way to train
convolutional autoencoders layer by layer, where in addition to lifetime
sparsity, a spatial sparsity within each feature map is achieved using
winner-take-all activation functions. We will show that winner-take-all
autoencoders can be used to to learn deep sparse representations from the
MNIST, CIFAR-10, ImageNet, Street View House Numbers and Toronto Face datasets,
and achieve competitive classification performance
k-Sparse Autoencoders
Recently, it has been observed that when representations are learnt in a way
that encourages sparsity, improved performance is obtained on classification
tasks. These methods involve combinations of activation functions, sampling
steps and different kinds of penalties. To investigate the effectiveness of
sparsity by itself, we propose the k-sparse autoencoder, which is an
autoencoder with linear activation function, where in hidden layers only the k
highest activities are kept. When applied to the MNIST and NORB datasets, we
find that this method achieves better classification results than denoising
autoencoders, networks trained with dropout, and RBMs. k-sparse autoencoders
are simple to train and the encoding stage is very fast, making them
well-suited to large problem sizes, where conventional sparse coding algorithms
cannot be applied
Cumulative distribution networks and the derivative-sum-product algorithm
We introduce a new type of graphical model called a "cumulative distribution
network" (CDN), which expresses a joint cumulative distribution as a product of
local functions. Each local function can be viewed as providing evidence about
possible orderings, or rankings, of variables. Interestingly, we find that the
conditional independence properties of CDNs are quite different from other
graphical models. We also describe a messagepassing algorithm that efficiently
computes conditional cumulative distributions. Due to the unique independence
properties of the CDN, these messages do not in general have a one-to-one
correspondence with messages exchanged in standard algorithms, such as belief
propagation. We demonstrate the application of CDNs for structured ranking
learning using a previously-studied multi-player gaming dataset.Comment: Appears in Proceedings of the Twenty-Fourth Conference on Uncertainty
in Artificial Intelligence (UAI2008
Learning Generative Models of Similarity Matrices
We describe a probabilistic (generative) view of affinity matrices along with
inference algorithms for a subclass of problems associated with data
clustering. This probabilistic view is helpful in understanding different
models and algorithms that are based on affinity functions OF the data. IN
particular, we show how(greedy) inference FOR a specific probabilistic model IS
equivalent TO the spectral clustering algorithm.It also provides a framework
FOR developing new algorithms AND extended models. AS one CASE, we present new
generative data clustering models that allow us TO infer the underlying
distance measure suitable for the clustering problem at hand. These models seem
to perform well in a larger class of problems for which other clustering
algorithms (including spectral clustering) usually fail. Experimental
evaluation was performed in a variety point data sets, showing excellent
performance.Comment: Appears in Proceedings of the Nineteenth Conference on Uncertainty in
Artificial Intelligence (UAI2003
Training Restricted Boltzmann Machine by Perturbation
A new approach to maximum likelihood learning of discrete graphical models
and RBM in particular is introduced. Our method, Perturb and Descend (PD) is
inspired by two ideas (I) perturb and MAP method for sampling (II) learning by
Contrastive Divergence minimization. In contrast to perturb and MAP, PD
leverages training data to learn the models that do not allow efficient MAP
estimation. During the learning, to produce a sample from the current model, we
start from a training data and descend in the energy landscape of the
"perturbed model", for a fixed number of steps, or until a local optima is
reached. For RBM, this involves linear calculations and thresholding which can
be very fast. Furthermore we show that the amount of perturbation is closely
related to the temperature parameter and it can regularize the model by
producing robust features resulting in sparse hidden layer activation
Matrix Tile Analysis
Many tasks require finding groups of elements in a matrix of numbers, symbols
or class likelihoods. One approach is to use efficient bi- or tri-linear
factorization techniques including PCA, ICA, sparse matrix factorization and
plaid analysis. These techniques are not appropriate when addition and
multiplication of matrix elements are not sensibly defined. More directly,
methods like bi-clustering can be used to classify matrix elements, but these
methods make the overly-restrictive assumption that the class of each element
is a function of a row class and a column class. We introduce a general
computational problem, `matrix tile analysis' (MTA), which consists of
decomposing a matrix into a set of non-overlapping tiles, each of which is
defined by a subset of usually nonadjacent rows and columns. MTA does not
require an algebra for combining tiles, but must search over discrete
combinations of tile assignments. Exact MTA is a computationally intractable
integer programming problem, but we describe an approximate iterative technique
and a computationally efficient sum-product relaxation of the integer program.
We compare the effectiveness of these methods to PCA and plaid on hundreds of
randomly generated tasks. Using double-gene-knockout data, we show that MTA
finds groups of interacting yeast genes that have biologically-related
functions.Comment: Appears in Proceedings of the Twenty-Second Conference on Uncertainty
in Artificial Intelligence (UAI2006
A Factorized Variational Technique for Phase Unwrapping in Markov Random Fields
Some types of medical and topographic imaging device produce images in which
the pixel values are "phase-wrapped", i.e. measured modulus a known scalar.
Phase unwrapping can be viewed as the problem of inferring the number of shifts
between each and every pair of neighboring pixels, subject to an a priori
preference for smooth surfaces, and subject to a zero curl constraint, which
requires that the shifts must sum to 0 around every loop. We formulate phase
unwrapping as a mean field inference problem in a Markov network, where the
prior favors the zero curl constraint. We compare our mean field technique with
the least squares method on a synthetic 100x100 image, and give results on a
512x512 synthetic aperture radar image from Sandia National Laboratories.<Long
Text>Comment: Appears in Proceedings of the Seventeenth Conference on Uncertainty
in Artificial Intelligence (UAI2001
Variational Learning in Mixed-State Dynamic Graphical Models
Many real-valued stochastic time-series are locally linear (Gassian), but
globally non-linear. For example, the trajectory of a human hand gesture can be
viewed as a linear dynamic system driven by a nonlinear dynamic system that
represents muscle actions. We present a mixed-state dynamic graphical model in
which a hidden Markov model drives a linear dynamic system. This combination
allows us to model both the discrete and continuous causes of trajectories such
as human gestures. The number of computations needed for exact inference is
exponential in the sequence length, so we derive an approximate variational
inference technique that can also be used to learn the parameters of the
discrete and continuous models. We show how the mixed-state model and the
variational technique can be used to classify human hand gestures made with a
computer mouse.Comment: Appears in Proceedings of the Fifteenth Conference on Uncertainty in
Artificial Intelligence (UAI1999
- …