11 research outputs found
Stochastic Gradient Annealed Importance Sampling for Efficient Online Marginal Likelihood Estimation
We consider estimating the marginal likelihood in settings with independent
and identically distributed (i.i.d.) data. We propose estimating the predictive
distributions in a sequential factorization of the marginal likelihood in such
settings by using stochastic gradient Markov Chain Monte Carlo techniques. This
approach is far more efficient than traditional marginal likelihood estimation
techniques such as nested sampling and annealed importance sampling due to its
use of mini-batches to approximate the likelihood. Stability of the estimates
is provided by an adaptive annealing schedule. The resulting stochastic
gradient annealed importance sampling (SGAIS) technique, which is the key
contribution of our paper, enables us to estimate the marginal likelihood of a
number of models considerably faster than traditional approaches, with no
noticeable loss of accuracy. An important benefit of our approach is that the
marginal likelihood is calculated in an online fashion as data becomes
available, allowing the estimates to be used for applications such as online
weighted model combination
Patterns of Scalable Bayesian Inference
Datasets are growing not just in size but in complexity, creating a demand
for rich models and quantification of uncertainty. Bayesian methods are an
excellent fit for this demand, but scaling Bayesian inference is a challenge.
In response to this challenge, there has been considerable recent work based on
varying assumptions about model structure, underlying computational resources,
and the importance of asymptotic correctness. As a result, there is a zoo of
ideas with few clear overarching principles.
In this paper, we seek to identify unifying principles, patterns, and
intuitions for scaling Bayesian inference. We review existing work on utilizing
modern computing resources with both MCMC and variational approximation
techniques. From this taxonomy of ideas, we characterize the general principles
that have proven successful for designing scalable inference procedures and
comment on the path forward