280 research outputs found
Patterns of Scalable Bayesian Inference
Datasets are growing not just in size but in complexity, creating a demand
for rich models and quantification of uncertainty. Bayesian methods are an
excellent fit for this demand, but scaling Bayesian inference is a challenge.
In response to this challenge, there has been considerable recent work based on
varying assumptions about model structure, underlying computational resources,
and the importance of asymptotic correctness. As a result, there is a zoo of
ideas with few clear overarching principles.
In this paper, we seek to identify unifying principles, patterns, and
intuitions for scaling Bayesian inference. We review existing work on utilizing
modern computing resources with both MCMC and variational approximation
techniques. From this taxonomy of ideas, we characterize the general principles
that have proven successful for designing scalable inference procedures and
comment on the path forward
Scalable Bayesian Non-Negative Tensor Factorization for Massive Count Data
We present a Bayesian non-negative tensor factorization model for
count-valued tensor data, and develop scalable inference algorithms (both batch
and online) for dealing with massive tensors. Our generative model can handle
overdispersed counts as well as infer the rank of the decomposition. Moreover,
leveraging a reparameterization of the Poisson distribution as a multinomial
facilitates conjugacy in the model and enables simple and efficient Gibbs
sampling and variational Bayes (VB) inference updates, with a computational
cost that only depends on the number of nonzeros in the tensor. The model also
provides a nice interpretability for the factors; in our model, each factor
corresponds to a "topic". We develop a set of online inference algorithms that
allow further scaling up the model to massive tensors, for which batch
inference methods may be infeasible. We apply our framework on diverse
real-world applications, such as \emph{multiway} topic modeling on a scientific
publications database, analyzing a political science data set, and analyzing a
massive household transactions data set.Comment: ECML PKDD 201
Online but Accurate Inference for Latent Variable Models with Local Gibbs Sampling
We study parameter inference in large-scale latent variable models. We first
propose an unified treatment of online inference for latent variable models
from a non-canonical exponential family, and draw explicit links between
several previously proposed frequentist or Bayesian methods. We then propose a
novel inference method for the frequentist estimation of parameters, that
adapts MCMC methods to online inference of latent variable models with the
proper use of local Gibbs sampling. Then, for latent Dirich-let allocation,we
provide an extensive set of experiments and comparisons with existing work,
where our new approach outperforms all previously proposed methods. In
particular, using Gibbs sampling for latent variable inference is superior to
variational inference in terms of test log-likelihoods. Moreover, Bayesian
inference through variational methods perform poorly, sometimes leading to
worse fits with latent variables of higher dimensionality
Minibatch Markov chain Monte Carlo Algorithms for Fitting Gaussian Processes
Gaussian processes (GPs) are a highly flexible, nonparametric statistical
model that are commonly used to fit nonlinear relationships or account for
correlation between observations. However, the computational load of fitting a
Gaussian process is making them infeasible for use on large
datasets. To make GPs more feasible for large datasets, this research focuses
on the use of minibatching to estimate GP parameters. Specifically, we outline
both approximate and exact minibatch Markov chain Monte Carlo algorithms that
substantially reduce the computation of fitting a GP by only considering small
subsets of the data at a time. We demonstrate and compare this methodology
using various simulations and real datasets.Comment: 19 Pages, 6 Figure
- …