7,151 research outputs found
Variational Inference in Nonconjugate Models
Mean-field variational methods are widely used for approximate posterior
inference in many probabilistic models. In a typical application, mean-field
methods approximately compute the posterior with a coordinate-ascent
optimization algorithm. When the model is conditionally conjugate, the
coordinate updates are easily derived and in closed form. However, many models
of interest---like the correlated topic model and Bayesian logistic
regression---are nonconjuate. In these models, mean-field methods cannot be
directly applied and practitioners have had to develop variational algorithms
on a case-by-case basis. In this paper, we develop two generic methods for
nonconjugate models, Laplace variational inference and delta method variational
inference. Our methods have several advantages: they allow for easily derived
variational algorithms with a wide class of nonconjugate models; they extend
and unify some of the existing algorithms that have been derived for specific
models; and they work well on real-world datasets. We studied our methods on
the correlated topic model, Bayesian logistic regression, and hierarchical
Bayesian logistic regression
Efficient Correlated Topic Modeling with Topic Embedding
Correlated topic modeling has been limited to small model and problem sizes
due to their high computational cost and poor scaling. In this paper, we
propose a new model which learns compact topic embeddings and captures topic
correlations through the closeness between the topic vectors. Our method
enables efficient inference in the low-dimensional embedding space, reducing
previous cubic or quadratic time complexity to linear w.r.t the topic size. We
further speedup variational inference with a fast sampler to exploit sparsity
of topic occurrence. Extensive experiments show that our approach is capable of
handling model and data scales which are several orders of magnitude larger
than existing correlation results, without sacrificing modeling quality by
providing competitive or superior performance in document classification and
retrieval.Comment: KDD 2017 oral. The first two authors contributed equall
The Discrete Infinite Logistic Normal Distribution
We present the discrete infinite logistic normal distribution (DILN), a
Bayesian nonparametric prior for mixed membership models. DILN is a
generalization of the hierarchical Dirichlet process (HDP) that models
correlation structure between the weights of the atoms at the group level. We
derive a representation of DILN as a normalized collection of gamma-distributed
random variables, and study its statistical properties. We consider
applications to topic modeling and derive a variational inference algorithm for
approximate posterior inference. We study the empirical performance of the DILN
topic model on four corpora, comparing performance with the HDP and the
correlated topic model (CTM). To deal with large-scale data sets, we also
develop an online inference algorithm for DILN and compare with online HDP and
online LDA on the Nature magazine, which contains approximately 350,000
articles.Comment: This paper will appear in Bayesian Analysis. A shorter version of
this paper appeared at AISTATS 2011, Fort Lauderdale, FL, US
- …