11,907 research outputs found
Variational Bayesian Inference with Stochastic Search
Mean-field variational inference is a method for approximate Bayesian
posterior inference. It approximates a full posterior distribution with a
factorized set of distributions by maximizing a lower bound on the marginal
likelihood. This requires the ability to integrate a sum of terms in the log
joint likelihood using this factorized distribution. Often not all integrals
are in closed form, which is typically handled by using a lower bound. We
present an alternative algorithm based on stochastic optimization that allows
for direct optimization of the variational lower bound. This method uses
control variates to reduce the variance of the stochastic search gradient, in
which existing lower bounds can play an important role. We demonstrate the
approach on two non-conjugate models: logistic regression and an approximation
to the HDP.Comment: Appears in Proceedings of the 29th International Conference on
Machine Learning (ICML 2012
Dropout Inference in Bayesian Neural Networks with Alpha-divergences
To obtain uncertainty estimates with real-world Bayesian deep learning
models, practical inference approximations are needed. Dropout variational
inference (VI) for example has been used for machine vision and medical
applications, but VI can severely underestimates model uncertainty.
Alpha-divergences are alternative divergences to VI's KL objective, which are
able to avoid VI's uncertainty underestimation. But these are hard to use in
practice: existing techniques can only use Gaussian approximating
distributions, and require existing models to be changed radically, thus are of
limited use for practitioners. We propose a re-parametrisation of the
alpha-divergence objectives, deriving a simple inference technique which,
together with dropout, can be easily implemented with existing models by simply
changing the loss of the model. We demonstrate improved uncertainty estimates
and accuracy compared to VI in dropout networks. We study our model's epistemic
uncertainty far away from the data using adversarial images, showing that these
can be distinguished from non-adversarial images by examining our model's
uncertainty
- …