18,244 research outputs found
Estimating Mixture Entropy with Pairwise Distances
Mixture distributions arise in many parametric and non-parametric settings --
for example, in Gaussian mixture models and in non-parametric estimation. It is
often necessary to compute the entropy of a mixture, but, in most cases, this
quantity has no closed-form expression, making some form of approximation
necessary. We propose a family of estimators based on a pairwise distance
function between mixture components, and show that this estimator class has
many attractive properties. For many distributions of interest, the proposed
estimators are efficient to compute, differentiable in the mixture parameters,
and become exact when the mixture components are clustered. We prove this
family includes lower and upper bounds on the mixture entropy. The Chernoff
-divergence gives a lower bound when chosen as the distance function,
with the Bhattacharyya distance providing the tightest lower bound for
components that are symmetric and members of a location family. The
Kullback-Leibler divergence gives an upper bound when used as the distance
function. We provide closed-form expressions of these bounds for mixtures of
Gaussians, and discuss their applications to the estimation of mutual
information. We then demonstrate that our bounds are significantly tighter than
well-known existing bounds using numeric simulations. This estimator class is
very useful in optimization problems involving maximization/minimization of
entropy and mutual information, such as MaxEnt and rate distortion problems.Comment: Corrects several errata in published version, in particular in
Section V (bounds on mutual information
Bayesian multivariate mixed-scale density estimation
Although continuous density estimation has received abundant attention in the
Bayesian nonparametrics literature, there is limited theory on multivariate
mixed scale density estimation. In this note, we consider a general framework
to jointly model continuous, count and categorical variables under a
nonparametric prior, which is induced through rounding latent variables having
an unknown density with respect to Lebesgue measure. For the proposed class of
priors, we provide sufficient conditions for large support, strong consistency
and rates of posterior contraction. These conditions allow one to convert
sufficient conditions obtained in the setting of multivariate continuous
density estimation to the mixed scale case. To illustrate the procedure a
rounded multivariate nonparametric mixture of Gaussians is introduced and
applied to a crime and communities dataset
Entropy and Wigner Functions
The properties of an alternative definition of quantum entropy, based on
Wigner functions, are discussed. Such definition emerges naturally from the
Wigner representation of quantum mechanics, and can easily quantify the amount
of entanglement of a quantum state. It is shown that smoothing of the Wigner
function induces an increase in entropy. This fact is used to derive some
simple rules to construct positive definite probability distributions which are
also admissible Wigner functionsComment: 18 page
A Unifying review of linear gaussian models
Factor analysis, principal component analysis, mixtures of gaussian clusters, vector quantization, Kalman filter models, and hidden Markov models can all be unified as variations of unsupervised learning under a single basic generative model. This is achieved by collecting together disparate observations and derivations made by many previous authors and introducing a new way of linking discrete and continuous state models using a simple nonlinearity. Through the use of other nonlinearities, we show how independent component analysis is also a variation of the same basic generative model.We show that factor analysis and mixtures of gaussians can be implemented in autoencoder neural networks and learned using squared error plus the same regularization term. We introduce a new model for static data, known as sensible principal component analysis, as well as a novel concept of spatially adaptive observation noise. We also review some of the literature involving global and local mixtures of the basic models and provide pseudocode for inference and learning for all the basic models
- …