3,462 research outputs found
Fast Variational Inference in the Conjugate Exponential Family
We present a general method for deriving collapsed variational inference
algo- rithms for probabilistic models in the conjugate exponential family. Our
method unifies many existing approaches to collapsed variational inference. Our
collapsed variational inference leads to a new lower bound on the marginal
likelihood. We exploit the information geometry of the bound to derive much
faster optimization methods based on conjugate gradients for these models. Our
approach is very general and is easily applied to any model where the mean
field update equations have been derived. Empirically we show significant
speed-ups for probabilistic models optimized using our bound.Comment: Accepted at NIPS 201
Conditionally conjugate mean-field variational Bayes for logistic models
Variational Bayes (VB) is a common strategy for approximate Bayesian
inference, but simple methods are only available for specific classes of models
including, in particular, representations having conditionally conjugate
constructions within an exponential family. Models with logit components are an
apparently notable exception to this class, due to the absence of conjugacy
between the logistic likelihood and the Gaussian priors for the coefficients in
the linear predictor. To facilitate approximate inference within this widely
used class of models, Jaakkola and Jordan (2000) proposed a simple variational
approach which relies on a family of tangent quadratic lower bounds of logistic
log-likelihoods, thus restoring conjugacy between these approximate bounds and
the Gaussian priors. This strategy is still implemented successfully, but less
attempts have been made to formally understand the reasons underlying its
excellent performance. To cover this key gap, we provide a formal connection
between the above bound and a recent P\'olya-gamma data augmentation for
logistic regression. Such a result places the computational methods associated
with the aforementioned bounds within the framework of variational inference
for conditionally conjugate exponential family models, thereby allowing recent
advances for this class to be inherited also by the methods relying on Jaakkola
and Jordan (2000)
- …