Premium: An R package for profile regression mixture models using dirichlet processes

PReMiuM is a recently developed R package for Bayesian clustering using a Dirichlet process mixture model. This model is an alternative to regression models, nonparametrically linking a response vector to covariate data through cluster membership (Molitor, Papathomas, Jerrett, and Richardson 2010). The package allows binary, categorical, count and continuous response, as well as continuous and discrete covariates. Additionally, predictions may be made for the response, and missing values for the covariates are handled. Several samplers and label switching moves are implemented along with diagnostic tools to assess convergence. A number of R functions for post-processing of the output are also provided. In addition to fitting mixtures, it may additionally be of interest to determine which covariates actively drive the mixture components. This is implemented in the package as variable selection

General Bayesian inference schemes in infinite mixture models

Bayesian statistical models allow us to formalise our knowledge about the world and reason about our uncertainty, but there is a need for better procedures to accurately encode its inherent complexity. One way to do so is through compositional models, which are formed by combining blocks or components consisting of simpler models. One can increase the complexity of the compositional model by either stacking more blocks or by using a not-so-simple model as a building block. This thesis is an example of the latter. One first aim is to expand the choice of Bayesian nonparametric (BNP) blocks for constructing tractable compositional models. So far, most of the models that have a Bayesian nonparametric component use either a Dirichlet Process or a Pitman–Yor process because of the availability of tractable and compact representations. This thesis shows how to overcome certain intractabilities in order to obtain analogous compact representations for the very wide class of Poisson–Kingman priors which includes the Dirichlet and Pitman–Yor processes. A major impediment to the widespread use of Bayesian nonparametric building blocks is that inference is often costly, intractable or difficult to carry out. This is an active research area since dealing with the model’s infinite dimensional component forbids the direct use of standard simulation-based methods. The main contribution of this thesis is a variety of inference schemes that tackle this problem: Markov chain Monte Carlo and Sequential Monte Carlo methods, which are exact inference methods since they target the true posterior. The contributions of this thesis, in a larger context, provide general purpose exact inference schemes in the flavour or probabilistic programming: the user is able to choose from a variety of models, focusing only on the modelling part. We show how if the wide enough class of Poisson–Kingman priors is used as one of our blocks, this objective is achieved

Supervised gamma process Poisson factorization

textThis thesis develops the supervised gamma process Poisson factorization (S-GPPF) framework, a novel supervised topic model for joint modeling of count matrices and document labels. S-GPPF is fully generative and nonparametric: document labels and count matrices are modeled under a unified probabilistic framework and the number of latent topics is controlled automatically via a gamma process prior. The framework provides for multi-class classification of documents using a generative max-margin classifier. Several recent data augmentation techniques are leveraged to provide for exact inference using a Gibbs sampling scheme. The first portion of this thesis reviews supervised topic modeling and several key mathematical devices used in the formulation of S-GPPF. The thesis then introduces the S-GPPF generative model and derives the conditional posterior distributions of the latent variables for posterior inference via Gibbs sampling. The S-GPPF is shown to exhibit state-of-the-art performance for joint topic modeling and document classification on a dataset of conference abstracts, beating out competing supervised topic models. The unique properties of S-GPPF along with its competitive performance make it a novel contribution to supervised topic modeling.Electrical and Computer Engineerin

Markov chain Monte Carlo for continuous-time discrete-state systems

A variety of phenomena are best described using dynamical models which operate on a discrete state space and in continuous time. Examples include Markov (and semi-Markov) jump processes, continuous-time Bayesian networks, renewal processes and other point processes. These continuous-time, discrete-state models are ideal building blocks for Bayesian models in fields such as systems biology, genetics, chemistry, computing networks, human-computer interactions etc. However, a challenge towards their more widespread use is the computational burden of posterior inference; this typically involves approximations like time discretization and can be computationally intensive. In this thesis, we describe a new class of Markov chain Monte Carlo methods that allow efficient computation while still being exact. The core idea is an auxiliary variable Gibbs sampler that alternately resamples a random discretization of time given the state-trajectory of the system, and then samples a new trajectory given this discretization. We introduce this idea by relating it to a classical idea called uniformization, and use it to develop algorithms that outperform the state-of-the-art for models based on the Markov jump process. We then extend the scope of these samplers to a wider class of models such as nonstationary renewal processes, and semi-Markov jump processes. By developing a more general framework beyond uniformization, we remedy various limitations of the original algorithms, allowing us to develop MCMC samplers for systems with infinite state spaces, unbounded rates, as well as systems indexed by more general continuous spaces than time

Hierarchical Bayesian Nonparametric Models for Power-Law Sequences

Sequence data that exhibits power-law behavior in its marginal and conditional distributions arises frequently from natural processes, with natural language text being a prominent example. We study probabilistic models for such sequences based on a hierarchical non-parametric Bayesian prior, develop inference and learning procedures for making these models useful in practice and applicable to large, real-world data sets, and empirically demonstrate their excellent predictive performance. In particular, we consider models based on the infinite-depth variant of the hierarchical Pitman-Yor process (HPYP) language model [Teh, 2006b] known as the Sequence Memoizer, as well as Sequence Memoizer-based cache language models and hybrid models combining the HPYP with neural language models. We empirically demonstrate that these models performwell on languagemodelling and data compression tasks

On Dependent Processes in Bayesian Nonparametrics: Theory, Methods, and Applications

The main topics of the thesis are dependent processes and their uses in Bayesian nonparametric statistics. With the term dependent processes, we refer to two or more infinite dimensional random objects, i.e., random probability measures, completely random measures, and random partitions, whose joint probability law does not factorize and, thus, encodes non-trivial dependence. We investigate properties and limits of existing nonparametric dependent priors and propose new dependent processes that fill gaps in the existing literature. To do so, we first define a class of priors, namely multivariate species sampling processes, which encompasses many dependent processes used in Bayesian nonparametrics. We derive a series of theoretical results for the priors within this class, keeping as main focus the dependence induced between observations as well as between random probability measures. Then, in light of our theoretical findings, as well as considering specific motivating applications, we develop novel prior processes outside this class, enlarging the types of data structures and prior information that can be handled by the Bayesian nonparametric approach. We propose three new classes of dependent processes: full-range borrowing of information priors, invariant dependent priors (with a focus on symmetric hierarchical Dirichlet processes), and dependent priors for panel count data. Full-range borrowing of information priors are dependent random probability measures that may induce either positive or negative correlation across observations and, thus, they achieve high flexibility in the type of induced dependence. Moreover, they introduce an innovative idea of borrowing of information across samples which differs from classical shrinkage. Invariant dependent priors are instead dependent random probabilities that almost surely satisfy a specified invariance condition, e.g., symmetry. They may be employed both when a priori knowledge on the shape of the unknown distribution is available or, as we do, to flexibly model errors terms in complex models without losing identifiability of other parameters of interest. Finally, dependent priors for panel count data are flexible priors based on completely random measures, that take into account dependence between the observed counts and the frequency of observation in panel count data studies. We study a priori and a posteriori properties of all the proposed models, develop algorithms to derive inference, compare the performances of our proposals with existing methods, and apply these constructions to simulated and real datasets. Through all the thesis, we try to balance theoretical and methodological results with real-world applications