2,919 research outputs found
PASS-GLM: polynomial approximate sufficient statistics for scalable Bayesian GLM inference
Generalized linear models (GLMs) -- such as logistic regression, Poisson
regression, and robust regression -- provide interpretable models for diverse
data types. Probabilistic approaches, particularly Bayesian ones, allow
coherent estimates of uncertainty, incorporation of prior information, and
sharing of power across experiments via hierarchical models. In practice,
however, the approximate Bayesian methods necessary for inference have either
failed to scale to large data sets or failed to provide theoretical guarantees
on the quality of inference. We propose a new approach based on constructing
polynomial approximate sufficient statistics for GLMs (PASS-GLM). We
demonstrate that our method admits a simple algorithm as well as trivial
streaming and distributed extensions that do not compound error across
computations. We provide theoretical guarantees on the quality of point (MAP)
estimates, the approximate posterior, and posterior mean and uncertainty
estimates. We validate our approach empirically in the case of logistic
regression using a quadratic approximation and show competitive performance
with stochastic gradient descent, MCMC, and the Laplace approximation in terms
of speed and multiple measures of accuracy -- including on an advertising data
set with 40 million data points and 20,000 covariates.Comment: In Proceedings of the 31st Annual Conference on Neural Information
Processing Systems (NIPS 2017). v3: corrected typos in Appendix
The Kernel Interaction Trick: Fast Bayesian Discovery of Pairwise Interactions in High Dimensions
Discovering interaction effects on a response of interest is a fundamental
problem faced in biology, medicine, economics, and many other scientific
disciplines. In theory, Bayesian methods for discovering pairwise interactions
enjoy many benefits such as coherent uncertainty quantification, the ability to
incorporate background knowledge, and desirable shrinkage properties. In
practice, however, Bayesian methods are often computationally intractable for
even moderate-dimensional problems. Our key insight is that many hierarchical
models of practical interest admit a particular Gaussian process (GP)
representation; the GP allows us to capture the posterior with a vector of O(p)
kernel hyper-parameters rather than O(p^2) interactions and main effects. With
the implicit representation, we can run Markov chain Monte Carlo (MCMC) over
model hyper-parameters in time and memory linear in p per iteration. We focus
on sparsity-inducing models and show on datasets with a variety of covariate
behaviors that our method: (1) reduces runtime by orders of magnitude over
naive applications of MCMC, (2) provides lower Type I and Type II error
relative to state-of-the-art LASSO-based approaches, and (3) offers improved
computational scaling in high dimensions relative to existing Bayesian and
LASSO-based approaches.Comment: Accepted at ICML 2019. 20 pages, 4 figures, 3 table
Practical bounds on the error of Bayesian posterior approximations: A nonasymptotic approach
Bayesian inference typically requires the computation of an approximation to
the posterior distribution. An important requirement for an approximate
Bayesian inference algorithm is to output high-accuracy posterior mean and
uncertainty estimates. Classical Monte Carlo methods, particularly Markov Chain
Monte Carlo, remain the gold standard for approximate Bayesian inference
because they have a robust finite-sample theory and reliable convergence
diagnostics. However, alternative methods, which are more scalable or apply to
problems where Markov Chain Monte Carlo cannot be used, lack the same
finite-data approximation theory and tools for evaluating their accuracy. In
this work, we develop a flexible new approach to bounding the error of mean and
uncertainty estimates of scalable inference algorithms. Our strategy is to
control the estimation errors in terms of Wasserstein distance, then bound the
Wasserstein distance via a generalized notion of Fisher distance. Unlike
computing the Wasserstein distance, which requires access to the normalized
posterior distribution, the Fisher distance is tractable to compute because it
requires access only to the gradient of the log posterior density. We
demonstrate the usefulness of our Fisher distance approach by deriving bounds
on the Wasserstein error of the Laplace approximation and Hilbert coresets. We
anticipate that our approach will be applicable to many other approximate
inference methods such as the integrated Laplace approximation, variational
inference, and approximate Bayesian computationComment: 22 pages, 2 figure
Truncated Random Measures
Completely random measures (CRMs) and their normalizations are a rich source
of Bayesian nonparametric priors. Examples include the beta, gamma, and
Dirichlet processes. In this paper we detail two major classes of sequential
CRM representations---series representations and superposition
representations---within which we organize both novel and existing sequential
representations that can be used for simulation and posterior inference. These
two classes and their constituent representations subsume existing ones that
have previously been developed in an ad hoc manner for specific processes.
Since a complete infinite-dimensional CRM cannot be used explicitly for
computation, sequential representations are often truncated for tractability.
We provide truncation error analyses for each type of sequential
representation, as well as their normalized versions, thereby generalizing and
improving upon existing truncation error bounds in the literature. We analyze
the computational complexity of the sequential representations, which in
conjunction with our error bounds allows us to directly compare representations
and discuss their relative efficiency. We include numerous applications of our
theoretical results to commonly-used (normalized) CRMs, demonstrating that our
results enable a straightforward representation and analysis of CRMs that has
not previously been available in a Bayesian nonparametric context.Comment: To appear in Bernoulli; 58 pages, 3 figure
- …