85,416 research outputs found
Deep Gaussian Mixture Models
Deep learning is a hierarchical inference method formed by subsequent
multiple layers of learning able to more efficiently describe complex
relationships. In this work, Deep Gaussian Mixture Models are introduced and
discussed. A Deep Gaussian Mixture model (DGMM) is a network of multiple layers
of latent variables, where, at each layer, the variables follow a mixture of
Gaussian distributions. Thus, the deep mixture model consists of a set of
nested mixtures of linear models, which globally provide a nonlinear model able
to describe the data in a very flexible way. In order to avoid
overparameterized solutions, dimension reduction by factor models can be
applied at each layer of the architecture thus resulting in deep mixtures of
factor analysers.Comment: 19 pages, 4 figure
Variational Inference and Sparsity in High-Dimensional Deep Gaussian Mixture Models
Gaussian mixture models are a popular tool for model-based clustering, and
mixtures of factor analyzers are Gaussian mixture models having parsimonious
factor covariance structure for mixture components. There are several recent
extensions of mixture of factor analyzers to deep mixtures, where the Gaussian
model for the latent factors is replaced by a mixture of factor analyzers. This
construction can be iterated to obtain a model with many layers. These deep
models are challenging to fit, and we consider Bayesian inference using
sparsity priors to further regularize the estimation. A scalable natural
gradient variational inference algorithm is developed for fitting the model,
and we suggest computationally efficient approaches to the architecture choice
using overfitted mixtures where unnecessary components drop out in the
estimation. In a number of simulated and two real examples, we demonstrate the
versatility of our approach for high-dimensional problems, and demonstrate that
the use of sparsity inducing priors can be helpful for obtaining improved
clustering results
Porting concepts from DNNs back to GMMs
Deep neural networks (DNNs) have been shown to outperform Gaussian Mixture Models (GMM) on a variety of speech recognition benchmarks. In this paper we analyze the differences between the DNN and GMM modeling techniques and port the best ideas from the DNN-based modeling to a GMM-based system. By going both deep (multiple layers) and wide (multiple parallel sub-models) and by sharing model parameters, we are able to close the gap between the two modeling techniques on the TIMIT database. Since the 'deep' GMMs retain the maximum-likelihood trained Gaussians as first layer, advanced techniques such as speaker adaptation and model-based noise robustness can be readily incorporated. Regardless of their similarities, the DNNs and the deep GMMs still show a sufficient amount of complementarity to allow effective system combination
Factoring variations in natural images with deep Gaussian mixture models
Generative models can be seen as the swiss army knives of machine learning, as many problems can be written probabilistically in terms of the distribution of the data, including prediction, reconstruction, imputation and simulation. One of the most promising directions for unsupervised learning may lie in Deep Learning methods, given their success in supervised learning. However, one of the cur- rent problems with deep unsupervised learning methods, is that they often are harder to scale. As a result there are some easier, more scalable shallow meth- ods, such as the Gaussian Mixture Model and the Student-t Mixture Model, that remain surprisingly competitive. In this paper we propose a new scalable deep generative model for images, called the Deep Gaussian Mixture Model, that is a straightforward but powerful generalization of GMMs to multiple layers. The parametrization of a Deep GMM allows it to efficiently capture products of vari- ations in natural images. We propose a new EM-based algorithm that scales well to large datasets, and we show that both the Expectation and the Maximization steps can easily be distributed over multiple machines. In our density estimation experiments we show that deeper GMM architectures generalize better than more shallow ones, with results in the same ballpark as the state of the art
Deep Generative Models for Reject Inference in Credit Scoring
Credit scoring models based on accepted applications may be biased and their
consequences can have a statistical and economic impact. Reject inference is
the process of attempting to infer the creditworthiness status of the rejected
applications. In this research, we use deep generative models to develop two
new semi-supervised Bayesian models for reject inference in credit scoring, in
which we model the data generating process to be dependent on a Gaussian
mixture. The goal is to improve the classification accuracy in credit scoring
models by adding reject applications. Our proposed models infer the unknown
creditworthiness of the rejected applications by exact enumeration of the two
possible outcomes of the loan (default or non-default). The efficient
stochastic gradient optimization technique used in deep generative models makes
our models suitable for large data sets. Finally, the experiments in this
research show that our proposed models perform better than classical and
alternative machine learning models for reject inference in credit scoring
- …