9,387 research outputs found
Bayesian Estimation of Mixed Multinomial Logit Models: Advances and Simulation-Based Evaluations
Variational Bayes (VB) methods have emerged as a fast and
computationally-efficient alternative to Markov chain Monte Carlo (MCMC)
methods for scalable Bayesian estimation of mixed multinomial logit (MMNL)
models. It has been established that VB is substantially faster than MCMC at
practically no compromises in predictive accuracy. In this paper, we address
two critical gaps concerning the usage and understanding of VB for MMNL. First,
extant VB methods are limited to utility specifications involving only
individual-specific taste parameters. Second, the finite-sample properties of
VB estimators and the relative performance of VB, MCMC and maximum simulated
likelihood estimation (MSLE) are not known. To address the former, this study
extends several VB methods for MMNL to admit utility specifications including
both fixed and random utility parameters. To address the latter, we conduct an
extensive simulation-based evaluation to benchmark the extended VB methods
against MCMC and MSLE in terms of estimation times, parameter recovery and
predictive accuracy. The results suggest that all VB variants with the
exception of the ones relying on an alternative variational lower bound
constructed with the help of the modified Jensen's inequality perform as well
as MCMC and MSLE at prediction and parameter recovery. In particular, VB with
nonconjugate variational message passing and the delta-method (VB-NCVMP-Delta)
is up to 16 times faster than MCMC and MSLE. Thus, VB-NCVMP-Delta can be an
attractive alternative to MCMC and MSLE for fast, scalable and accurate
estimation of MMNL models
Variational Hamiltonian Monte Carlo via Score Matching
Traditionally, the field of computational Bayesian statistics has been
divided into two main subfields: variational methods and Markov chain Monte
Carlo (MCMC). In recent years, however, several methods have been proposed
based on combining variational Bayesian inference and MCMC simulation in order
to improve their overall accuracy and computational efficiency. This marriage
of fast evaluation and flexible approximation provides a promising means of
designing scalable Bayesian inference methods. In this paper, we explore the
possibility of incorporating variational approximation into a state-of-the-art
MCMC method, Hamiltonian Monte Carlo (HMC), to reduce the required gradient
computation in the simulation of Hamiltonian flow, which is the bottleneck for
many applications of HMC in big data problems. To this end, we use a {\it
free-form} approximation induced by a fast and flexible surrogate function
based on single-hidden layer feedforward neural networks. The surrogate
provides sufficiently accurate approximation while allowing for fast
exploration of parameter space, resulting in an efficient approximate inference
algorithm. We demonstrate the advantages of our method on both synthetic and
real data problems
Variational bayes for estimating the parameters of a hidden Potts model
Hidden Markov random field models provide an appealing representation of images and other spatial problems. The drawback is that inference is not straightforward for these models as the normalisation constant for the likelihood is generally intractable except for very small observation sets. Variational methods are an emerging tool for Bayesian inference and they have already been successfully applied in other contexts. Focusing on the particular case of a hidden Potts model with Gaussian noise, we show how variational Bayesian methods can be applied to hidden Markov random field inference. To tackle the obstacle of the intractable normalising constant for the likelihood, we explore alternative estimation approaches for incorporation into the variational Bayes algorithm. We consider a pseudo-likelihood approach as well as the more recent reduced dependence approximation of the normalisation constant. To illustrate the effectiveness of these approaches we present empirical results from the analysis of simulated datasets. We also analyse a real dataset and compare results with those of previous analyses as well as those obtained from the recently developed auxiliary variable MCMC method and the recursive MCMC method. Our results show that the variational Bayesian analyses can be carried out much faster than the MCMC analyses and produce good estimates of model parameters. We also found that the reduced dependence approximation of the normalisation constant outperformed the pseudo-likelihood approximation in our analysis of real and synthetic datasets
MCMC-Interactive Variational Inference
Leveraging well-established MCMC strategies, we propose MCMC-interactive
variational inference (MIVI) to not only estimate the posterior in a time
constrained manner, but also facilitate the design of MCMC transitions.
Constructing a variational distribution followed by a short Markov chain that
has parameters to learn, MIVI takes advantage of the complementary properties
of variational inference and MCMC to encourage mutual improvement. On one hand,
with the variational distribution locating high posterior density regions, the
Markov chain is optimized within the variational inference framework to
efficiently target the posterior despite a small number of transitions. On the
other hand, the optimized Markov chain with considerable flexibility guides the
variational distribution towards the posterior and alleviates its
underestimation of uncertainty. Furthermore, we prove the optimized Markov
chain in MIVI admits extrapolation, which means its marginal distribution gets
closer to the true posterior as the chain grows. Therefore, the Markov chain
can be used separately as an efficient MCMC scheme. Experiments show that MIVI
not only accurately and efficiently approximates the posteriors but also
facilitates designs of stochastic gradient MCMC and Gibbs sampling transitions.Comment: 25 pages, 7 figures, 3 table
- …