The Infinite Mixture of Infinite Gaussian Mixtures

Dirichlet process mixture of Gaussians (DPMG) has been used in the literature for clustering and density estimation problems. However, many real-world data exhibit cluster distributions that cannot be captured by a single Gaussian. Modeling such data sets by DPMG creates several extraneous clusters even when clusters are relatively well-defined. Herein, we present the infinite mixture of infinite Gaussian mixtures (I2GMM) for more flexible modeling of data sets with skewed and multi-modal cluster distributions. Instead of using a single Gaussian for each cluster as in the standard DPMG model, the generative model of I2GMM uses a single DPMG for each cluster. The individual DPMGs are linked together through centering of their base distributions at the atoms of a higher level DP prior. Inference is performed by a collapsed Gibbs sampler that also enables partial parallelization. Experimental results on several artificial and real-world data sets suggest the proposed I2GMM model can predict clusters more accurately than existing variational Bayes and Gibbs sampler versions of DPMG

Infinite Mixtures of Multivariate Gaussian Processes

This paper presents a new model called infinite mixtures of multivariate Gaussian processes, which can be used to learn vector-valued functions and applied to multitask learning. As an extension of the single multivariate Gaussian process, the mixture model has the advantages of modeling multimodal data and alleviating the computationally cubic complexity of the multivariate Gaussian process. A Dirichlet process prior is adopted to allow the (possibly infinite) number of mixture components to be automatically inferred from training data, and Markov chain Monte Carlo sampling techniques are used for parameter and latent variable inference. Preliminary experimental results on multivariate regression show the feasibility of the proposed model.Comment: Proceedings of the International Conference on Machine Learning and Cybernetics, 2013, pages 1011-101

Sparse covariance estimation in heterogeneous samples

Standard Gaussian graphical models (GGMs) implicitly assume that the conditional independence among variables is common to all observations in the sample. However, in practice, observations are usually collected form heterogeneous populations where such assumption is not satisfied, leading in turn to nonlinear relationships among variables. To tackle these problems we explore mixtures of GGMs; in particular, we consider both infinite mixture models of GGMs and infinite hidden Markov models with GGM emission distributions. Such models allow us to divide a heterogeneous population into homogenous groups, with each cluster having its own conditional independence structure. The main advantage of considering infinite mixtures is that they allow us easily to estimate the number of number of subpopulations in the sample. As an illustration, we study the trends in exchange rate fluctuations in the pre-Euro era. This example demonstrates that the models are very flexible while providing extremely interesting interesting insights into real-life applications

Fast aggregation of Student mixture models

International audienceThis paper deals with probabilistic models, that take the form of mixtures of Student distributions. Student distributions are known to be more statistically robust than Gaussian distributions, with regard to outliers (i.e. data that cannot be reasonnably explained by any component in the mixture and that do not justifiy an extra component. Our contribution is as follows : we show how several mixtures of Student distributions may be agregated into a single mixture, without resorting to sampling. The trick is that, as is well known, a Student distribution may be expressed as an infinite mixture of Gaussians, where the variances follow a Gamma distribution

Multidimensional Membership Mixture Models

We present the multidimensional membership mixture (M3) models where every dimension of the membership represents an independent mixture model and each data point is generated from the selected mixture components jointly. This is helpful when the data has a certain shared structure. For example, three unique means and three unique variances can effectively form a Gaussian mixture model with nine components, while requiring only six parameters to fully describe it. In this paper, we present three instantiations of M3 models (together with the learning and inference algorithms): infinite, finite, and hybrid, depending on whether the number of mixtures is fixed or not. They are built upon Dirichlet process mixture models, latent Dirichlet allocation, and a combination respectively. We then consider two applications: topic modeling and learning 3D object arrangements. Our experiments show that our M3 models achieve better performance using fewer topics than many classic topic models. We also observe that topics from the different dimensions of M3 models are meaningful and orthogonal to each other.Comment: 9 pages, 7 figure

A quantum de Finetti theorem in phase space representation

The quantum versions of de Finetti's theorem derived so far express the convergence of n-partite symmetric states, i.e., states that are invariant under permutations of their n parties, towards probabilistic mixtures of independent and identically distributed (i.i.d.) states. Unfortunately, these theorems only hold in finite-dimensional Hilbert spaces, and their direct generalization to infinite-dimensional Hilbert spaces is known to fail. Here, we address this problem by considering invariance under orthogonal transformations in phase space instead of permutations in state space, which leads to a new type of quantum de Finetti's theorem that is particularly relevant to continuous-variable systems. Specifically, an n-mode bosonic state that is invariant with respect to this continuous symmetry in phase space is proven to converge towards a probabilistic mixture of i.i.d. Gaussian states (actually, n identical thermal states).Comment: 5 page

Fast aggregation of Student mixture models

International audienceThis paper deals with probabilistic models, that take the form of mixtures of Student distributions. Student distributions are known to be more statistically robust than Gaussian distributions, with regard to outliers (i.e. data that cannot be reasonnably explained by any component in the mixture and that do not justifiy an extra component. Our contribution is as follows : we show how several mixtures of Student distributions may be agregated into a single mixture, without resorting to sampling. The trick is that, as is well known, a Student distribution may be expressed as an infinite mixture of Gaussians, where the variances follow a Gamma distribution

A Tutorial on Bayesian Nonparametric Models

A key problem in statistical modeling is model selection, how to choose a model at an appropriate level of complexity. This problem appears in many settings, most prominently in choosing the number ofclusters in mixture models or the number of factors in factor analysis. In this tutorial we describe Bayesian nonparametric methods, a class of methods that side-steps this issue by allowing the data to determine the complexity of the model. This tutorial is a high-level introduction to Bayesian nonparametric methods and contains several examples of their application.Comment: 28 pages, 8 figure

Identifying Mixtures of Mixtures Using Bayesian Estimation

The use of a finite mixture of normal distributions in model-based clustering allows to capture non-Gaussian data clusters. However, identifying the clusters from the normal components is challenging and in general either achieved by imposing constraints on the model or by using post-processing procedures. Within the Bayesian framework we propose a different approach based on sparse finite mixtures to achieve identifiability. We specify a hierarchical prior where the hyperparameters are carefully selected such that they are reflective of the cluster structure aimed at. In addition this prior allows to estimate the model using standard MCMC sampling methods. In combination with a post-processing approach which resolves the label switching issue and results in an identified model, our approach allows to simultaneously (1) determine the number of clusters, (2) flexibly approximate the cluster distributions in a semi-parametric way using finite mixtures of normals and (3) identify cluster-specific parameters and classify observations. The proposed approach is illustrated in two simulation studies and on benchmark data sets.Comment: 49 page