70 research outputs found
Variational Probabilistic Inference and the QMR-DT Network
We describe a variational approximation method for efficient inference in
large-scale probabilistic models. Variational methods are deterministic
procedures that provide approximations to marginal and conditional
probabilities of interest. They provide alternatives to approximate inference
methods based on stochastic sampling or search. We describe a variational
approach to the problem of diagnostic inference in the `Quick Medical
Reference' (QMR) network. The QMR network is a large-scale probabilistic
graphical model built on statistical and expert knowledge. Exact probabilistic
inference is infeasible in this model for all but a small set of cases. We
evaluate our variational inference algorithm on a large set of diagnostic test
cases, comparing the algorithm to a state-of-the-art stochastic sampling
method
Recommended from our members
Efficient Variational Inference for Hierarchical Models of Images, Text, and Networks
Variational inference provides a general optimization framework to approximate the posterior distributions of latent variables in probabilistic models. Although effective in simple scenarios, variational inference may be inaccurate or infeasible when the data is high-dimensional, the model structure is complicated, or variable relationships are non-conjugate. We propose solutions to these problems through the smart design and leverage of model structures, the rigorous derivation of variational bounds, and the creation of flexible algorithms for various models with rich, non-conjugate dependencies.Concretely, we first design an interpretable generative model for natural images, in which the hundreds of thousands of pixels per image are split into small patches represented by Gaussian mixture models. Through structured variational inference, the evidence lower bound of this model automatically recovers the popular expected patch log-likelihood method for image processing. A nonparametric extension using hierarchical Dirichlet processes further enables self-similarities to be captured and image-specific clusters created during inference, boosting image denoising and inpainting accuracy.Then we move on to text data, and design hierarchical topic graphs that generalize the bipartite noisy-OR models previously used for medical diagnosis. We derive auxiliary bounds to overcome the non-conjugacy of noisy-OR conditionals, and use stochastic variational inference to efficiently train on datasets with hundreds of thousands of documents. We dramatically increase the algorithm speed through a constrained family of variational bounds, so that only the ancestors of the sparse observed tokens of each document need to be considered.Finally, we propose a general-purpose Monte Carlo variational inference strategy that is directly applicable to any model with discrete variables. Compared to REINFORCE-style stochastic gradient updates, our coordinate-ascent updates have lower variance and converge much faster. Compared to auxiliary-variable bounds crafted for each individual model, our algorithm is simpler to derive and may be easily integrated into probabilistic programming languages for broader use. By avoiding auxiliary variables, we also tighten likelihood bounds and increase robustness to local optima. Extensive experiments on real-world models of images, text, and networks illustrate these appealing advantages
Mean Field Methods for a Special Class of Belief Networks
The chief aim of this paper is to propose mean-field approximations for a
broad class of Belief networks, of which sigmoid and noisy-or networks can be
seen as special cases. The approximations are based on a powerful mean-field
theory suggested by Plefka. We show that Saul, Jaakkola and Jordan' s approach
is the first order approximation in Plefka's approach, via a variational
derivation. The application of Plefka's theory to belief networks is not
computationally tractable. To tackle this problem we propose new approximations
based on Taylor series. Small scale experiments show that the proposed schemes
are attractive
- …