77 research outputs found
Nonparametric variational inference
Variational methods are widely used for approximate posterior inference.
However, their use is typically limited to families of distributions that enjoy
particular conjugacy properties. To circumvent this limitation, we propose a
family of variational approximations inspired by nonparametric kernel density
estimation. The locations of these kernels and their bandwidth are treated as
variational parameters and optimized to improve an approximate lower bound on
the marginal likelihood of the data. Using multiple kernels allows the
approximation to capture multiple modes of the posterior, unlike most other
variational approximations. We demonstrate the efficacy of the nonparametric
approximation with a hierarchical logistic regression model and a nonlinear
matrix factorization model. We obtain predictive performance as good as or
better than more specialized variational methods and sample-based
approximations. The method is easy to apply to more general graphical models
for which standard variational methods are difficult to derive.Comment: ICML201
A Stein variational Newton method
Stein variational gradient descent (SVGD) was recently proposed as a general
purpose nonparametric variational inference algorithm [Liu & Wang, NIPS 2016]:
it minimizes the Kullback-Leibler divergence between the target distribution
and its approximation by implementing a form of functional gradient descent on
a reproducing kernel Hilbert space. In this paper, we accelerate and generalize
the SVGD algorithm by including second-order information, thereby approximating
a Newton-like iteration in function space. We also show how second-order
information can lead to more effective choices of kernel. We observe
significant computational gains over the original SVGD algorithm in multiple
test cases.Comment: 18 pages, 7 figure
Variational Inference in Nonconjugate Models
Mean-field variational methods are widely used for approximate posterior
inference in many probabilistic models. In a typical application, mean-field
methods approximately compute the posterior with a coordinate-ascent
optimization algorithm. When the model is conditionally conjugate, the
coordinate updates are easily derived and in closed form. However, many models
of interest---like the correlated topic model and Bayesian logistic
regression---are nonconjuate. In these models, mean-field methods cannot be
directly applied and practitioners have had to develop variational algorithms
on a case-by-case basis. In this paper, we develop two generic methods for
nonconjugate models, Laplace variational inference and delta method variational
inference. Our methods have several advantages: they allow for easily derived
variational algorithms with a wide class of nonconjugate models; they extend
and unify some of the existing algorithms that have been derived for specific
models; and they work well on real-world datasets. We studied our methods on
the correlated topic model, Bayesian logistic regression, and hierarchical
Bayesian logistic regression
- …