45,083 research outputs found
Deep Exponential Families
We describe \textit{deep exponential families} (DEFs), a class of latent
variable models that are inspired by the hidden structures used in deep neural
networks. DEFs capture a hierarchy of dependencies between latent variables,
and are easily generalized to many settings through exponential families. We
perform inference using recent "black box" variational inference techniques. We
then evaluate various DEFs on text and combine multiple DEFs into a model for
pairwise recommendation data. In an extensive study, we show that going beyond
one layer improves predictions for DEFs. We demonstrate that DEFs find
interesting exploratory structure in large data sets, and give better
predictive performance than state-of-the-art models
Towards Building Deep Networks with Bayesian Factor Graphs
We propose a Multi-Layer Network based on the Bayesian framework of the
Factor Graphs in Reduced Normal Form (FGrn) applied to a two-dimensional
lattice. The Latent Variable Model (LVM) is the basic building block of a
quadtree hierarchy built on top of a bottom layer of random variables that
represent pixels of an image, a feature map, or more generally a collection of
spatially distributed discrete variables. The multi-layer architecture
implements a hierarchical data representation that, via belief propagation, can
be used for learning and inference. Typical uses are pattern completion,
correction and classification. The FGrn paradigm provides great flexibility and
modularity and appears as a promising candidate for building deep networks: the
system can be easily extended by introducing new and different (in cardinality
and in type) variables. Prior knowledge, or supervised information, can be
introduced at different scales. The FGrn paradigm provides a handy way for
building all kinds of architectures by interconnecting only three types of
units: Single Input Single Output (SISO) blocks, Sources and Replicators. The
network is designed like a circuit diagram and the belief messages flow
bidirectionally in the whole system. The learning algorithms operate only
locally within each block. The framework is demonstrated in this paper in a
three-layer structure applied to images extracted from a standard data set.Comment: Submitted for journal publicatio
Learning the Structure of Deep Sparse Graphical Models
Deep belief networks are a powerful way to model complex probability
distributions. However, learning the structure of a belief network,
particularly one with hidden units, is difficult. The Indian buffet process has
been used as a nonparametric Bayesian prior on the directed structure of a
belief network with a single infinitely wide hidden layer. In this paper, we
introduce the cascading Indian buffet process (CIBP), which provides a
nonparametric prior on the structure of a layered, directed belief network that
is unbounded in both depth and width, yet allows tractable inference. We use
the CIBP prior with the nonlinear Gaussian belief network so each unit can
additionally vary its behavior between discrete and continuous representations.
We provide Markov chain Monte Carlo algorithms for inference in these belief
networks and explore the structures learned on several image data sets.Comment: 20 pages, 6 figures, AISTATS 2010, Revise
- …