823 research outputs found
A partially collapsed Gibbs sampler for Bayesian quantile regression
We introduce a set of new Gibbs sampler for Bayesian analysis of quantile re-gression model. The new algorithm, which partially collapsing an ordinary Gibbs sampler, is called Partially Collapsed Gibbs (PCG) sampler. Although the Metropolis-Hastings algorithm has been employed in Bayesian quantile regression, including
median regression, PCG has superior convergence properties to an ordinary Gibbs sampler. Moreover, Our PCG sampler algorithm, which is based on a theoretic derivation of an asymmetric Laplace as scale mixtures of normal distributions,
requires less computation than the ordinary Gibbs sampler and can significantly reduce the computation involved in approximating the Bayes Factor and marginal likelihood. Like the ordinary Gibbs sampler, the PCG sample can also be used
to calculate any associated marginal and predictive distributions. The quantile regression PCG sampler is illustrated by analysing simulated data and the data of length of stay in hospital. The latter provides new insight into hospital perfor-mance. C-code along with an R interface for our algorithms is publicly available
on request from the first author.
JEL classification: C11, C14, C21, C31, C52, C53
Sparse Partially Collapsed MCMC for Parallel Inference in Topic Models
Topic models, and more specifically the class of Latent Dirichlet Allocation
(LDA), are widely used for probabilistic modeling of text. MCMC sampling from
the posterior distribution is typically performed using a collapsed Gibbs
sampler. We propose a parallel sparse partially collapsed Gibbs sampler and
compare its speed and efficiency to state-of-the-art samplers for topic models
on five well-known text corpora of differing sizes and properties. In
particular, we propose and compare two different strategies for sampling the
parameter block with latent topic indicators. The experiments show that the
increase in statistical inefficiency from only partial collapsing is smaller
than commonly assumed, and can be more than compensated by the speedup from
parallelization and sparsity on larger corpora. We also prove that the
partially collapsed samplers scale well with the size of the corpus. The
proposed algorithm is fast, efficient, exact, and can be used in more modeling
situations than the ordinary collapsed sampler.Comment: Accepted for publication in Journal of Computational and Graphical
Statistic
Beta-Negative Binomial Process and Exchangeable Random Partitions for Mixed-Membership Modeling
The beta-negative binomial process (BNBP), an integer-valued stochastic
process, is employed to partition a count vector into a latent random count
matrix. As the marginal probability distribution of the BNBP that governs the
exchangeable random partitions of grouped data has not yet been developed,
current inference for the BNBP has to truncate the number of atoms of the beta
process. This paper introduces an exchangeable partition probability function
to explicitly describe how the BNBP clusters the data points of each group into
a random number of exchangeable partitions, which are shared across all the
groups. A fully collapsed Gibbs sampler is developed for the BNBP, leading to a
novel nonparametric Bayesian topic model that is distinct from existing ones,
with simple implementation, fast convergence, good mixing, and state-of-the-art
predictive performance.Comment: in Neural Information Processing Systems (NIPS) 2014. 9 pages + 3
page appendi
Gibbs Max-margin Topic Models with Data Augmentation
Max-margin learning is a powerful approach to building classifiers and
structured output predictors. Recent work on max-margin supervised topic models
has successfully integrated it with Bayesian topic models to discover
discriminative latent semantic structures and make accurate predictions for
unseen testing data. However, the resulting learning problems are usually hard
to solve because of the non-smoothness of the margin loss. Existing approaches
to building max-margin supervised topic models rely on an iterative procedure
to solve multiple latent SVM subproblems with additional mean-field assumptions
on the desired posterior distributions. This paper presents an alternative
approach by defining a new max-margin loss. Namely, we present Gibbs max-margin
supervised topic models, a latent variable Gibbs classifier to discover hidden
topic representations for various tasks, including classification, regression
and multi-task learning. Gibbs max-margin supervised topic models minimize an
expected margin loss, which is an upper bound of the existing margin loss
derived from an expected prediction rule. By introducing augmented variables
and integrating out the Dirichlet variables analytically by conjugacy, we
develop simple Gibbs sampling algorithms with no restricting assumptions and no
need to solve SVM subproblems. Furthermore, each step of the
"augment-and-collapse" Gibbs sampling algorithms has an analytical conditional
distribution, from which samples can be easily drawn. Experimental results
demonstrate significant improvements on time efficiency. The classification
performance is also significantly improved over competitors on binary,
multi-class and multi-label classification tasks.Comment: 35 page
Particle Gibbs with Ancestor Sampling
Particle Markov chain Monte Carlo (PMCMC) is a systematic way of combining
the two main tools used for Monte Carlo statistical inference: sequential Monte
Carlo (SMC) and Markov chain Monte Carlo (MCMC). We present a novel PMCMC
algorithm that we refer to as particle Gibbs with ancestor sampling (PGAS).
PGAS provides the data analyst with an off-the-shelf class of Markov kernels
that can be used to simulate the typically high-dimensional and highly
autocorrelated state trajectory in a state-space model. The ancestor sampling
procedure enables fast mixing of the PGAS kernel even when using seemingly few
particles in the underlying SMC sampler. This is important as it can
significantly reduce the computational burden that is typically associated with
using SMC. PGAS is conceptually similar to the existing PG with backward
simulation (PGBS) procedure. Instead of using separate forward and backward
sweeps as in PGBS, however, we achieve the same effect in a single forward
sweep. This makes PGAS well suited for addressing inference problems not only
in state-space models, but also in models with more complex dependencies, such
as non-Markovian, Bayesian nonparametric, and general probabilistic graphical
models
- …