16,125 research outputs found
Fast Convergent Algorithms for Expectation Propagation Approximate Bayesian Inference
We propose a novel algorithm to solve the expectation propagation relaxation
of Bayesian inference for continuous-variable graphical models. In contrast to
most previous algorithms, our method is provably convergent. By marrying
convergent EP ideas from (Opper&Winther 05) with covariance decoupling
techniques (Wipf&Nagarajan 08, Nickisch&Seeger 09), it runs at least an order
of magnitude faster than the most commonly used EP solver.Comment: 16 pages, 3 figures, submitted for conference publicatio
Approximate Inference for Robust Gaussian Process Regression
Gaussian process (GP) priors have been successfully used in non-parametric Bayesian regression and classification models. Inference can be performed analytically only for the regression model with Gaussian noise. For all other likelihood models inference is intractable and various approximation techniques have been proposed. In recent years expectation-propagation (EP) has been developed as a general method for approximate inference. This article provides a general summary of how expectation-propagation can be used for approximate inference in Gaussian process models. Furthermore we present a case study describing its implementation for a new robust variant of Gaussian process regression. To gain further insights into the quality of the EP approximation we present experiments in which we compare to results obtained by Markov chain Monte Carlo (MCMC) sampling
Approximate Bayesian Inference for Count Data Modeling
Bayesian inference allows to make conclusions based on some antecedents that depend on prior knowledge. It additionally allows to quantify uncertainty, which is important in Machine Learning in order to make better predictions and model interpretability. However, in real applications, we often deal with complicated models for which is unfeasible to perform full Bayesian inference. This thesis explores the use of approximate Bayesian inference for count data modeling using Expectation Propagation and Stochastic Expectation Propagation.
In Chapter 2, we develop an expectation propagation approach to learn an EDCM finite mixture model. The EDCM distribution is an exponential approximation to the widely used Dirichlet Compound distribution and has shown to offer excellent modeling capabilities in the case of sparse count data. Chapter 3 develops an efficient generative mixture model of EMSD distributions. We use Stochastic Expectation Propagation, which reduces memory consumption, important characteristic when making inference in large datasets.
Finally, Chapter 4 develops a probabilistic topic model using the generalized Dirichlet distribution (LGDA) in order to capture topic correlation while maintaining conjugacy. We make use of Expectation Propagation to approximate the posterior, resulting in a model that achieves more accurate inference compared to variational inference. We show that latent topics can be used as a proxy for improving supervised tasks
The explicit form of expectation propagation for a simple statistical model
© 2016, Institute of Mathematical Statistics. All rights reserved. We derive the explicit form of expectation propagation for approximate deterministic Bayesian inference in a simple statistical model. The model corresponds to a random sample from the Normal distribution. The explicit forms, and their derivation, allow a deeper understanding of the issues and challenges involved in practical implementation of expectation propagation for statistical analyses. No auxiliary approximations are used: we follow the expectation propagation prescription exactly. A simulation study shows expectation propagation to be more accurate than mean field variational Bayes for larger sample sizes, but at the cost of considerably more algebraic and computational effort
Deep Gaussian processes for regression using approximate expectation propagation
Deep Gaussian processes (DGPs) are multi-layer hierarchical generalisations
of Gaussian processes (GPs) and are formally equivalent to neural networks with
multiple, infinitely wide hidden layers. DGPs are nonparametric probabilistic
models and as such are arguably more flexible, have a greater capacity to
generalise, and provide better calibrated uncertainty estimates than
alternative deep models. This paper develops a new approximate Bayesian
learning scheme that enables DGPs to be applied to a range of medium to large
scale regression problems for the first time. The new method uses an
approximate Expectation Propagation procedure and a novel and efficient
extension of the probabilistic backpropagation algorithm for learning. We
evaluate the new method for non-linear regression on eleven real-world
datasets, showing that it always outperforms GP regression and is almost always
better than state-of-the-art deterministic and sampling-based approximate
inference methods for Bayesian neural networks. As a by-product, this work
provides a comprehensive analysis of six approximate Bayesian methods for
training neural networks
Expectation Propagation for Poisson Data
The Poisson distribution arises naturally when dealing with data involving
counts, and it has found many applications in inverse problems and imaging. In
this work, we develop an approximate Bayesian inference technique based on
expectation propagation for approximating the posterior distribution formed
from the Poisson likelihood function and a Laplace type prior distribution,
e.g., the anisotropic total variation prior. The approach iteratively yields a
Gaussian approximation, and at each iteration, it updates the Gaussian
approximation to one factor of the posterior distribution by moment matching.
We derive explicit update formulas in terms of one-dimensional integrals, and
also discuss stable and efficient quadrature rules for evaluating these
integrals. The method is showcased on two-dimensional PET images.Comment: 25 pages, to be published at Inverse Problem
Stochastic expectation propagation
Expectation propagation (EP) is a deterministic approximation algorithm that
is often used to perform approximate Bayesian parameter learning. EP
approximates the full intractable posterior distribution through a set of local
approximations that are iteratively refined for each datapoint. EP can offer
analytic and computational advantages over other approximations, such as
Variational Inference (VI), and is the method of choice for a number of models.
The local nature of EP appears to make it an ideal candidate for performing
Bayesian learning on large models in large-scale dataset settings. However, EP
has a crucial limitation in this context: the number of approximating factors
needs to increase with the number of data-points, N, which often entails a
prohibitively large memory overhead. This paper presents an extension to EP,
called stochastic expectation propagation (SEP), that maintains a global
posterior approximation (like VI) but updates it in a local way (like EP).
Experiments on a number of canonical learning problems using synthetic and
real-world datasets indicate that SEP performs almost as well as full EP, but
reduces the memory consumption by a factor of . SEP is therefore ideally
suited to performing approximate Bayesian learning in the large model, large
dataset setting
Expectation Propagation for Nonlinear Inverse Problems -- with an Application to Electrical Impedance Tomography
In this paper, we study a fast approximate inference method based on
expectation propagation for exploring the posterior probability distribution
arising from the Bayesian formulation of nonlinear inverse problems. It is
capable of efficiently delivering reliable estimates of the posterior mean and
covariance, thereby providing an inverse solution together with quantified
uncertainties. Some theoretical properties of the iterative algorithm are
discussed, and the efficient implementation for an important class of problems
of projection type is described. The method is illustrated with one typical
nonlinear inverse problem, electrical impedance tomography with complete
electrode model, under sparsity constraints. Numerical results for real
experimental data are presented, and compared with that by Markov chain Monte
Carlo. The results indicate that the method is accurate and computationally
very efficient.Comment: Journal of Computational Physics, to appea
- …