6 research outputs found
On a generalization of the Jensen-Shannon divergence and the JS-symmetrization of distances relying on abstract means
The Jensen-Shannon divergence is a renown bounded symmetrization of the
unbounded Kullback-Leibler divergence which measures the total Kullback-Leibler
divergence to the average mixture distribution. However the Jensen-Shannon
divergence between Gaussian distributions is not available in closed-form. To
bypass this problem, we present a generalization of the Jensen-Shannon (JS)
divergence using abstract means which yields closed-form expressions when the
mean is chosen according to the parametric family of distributions. More
generally, we define the JS-symmetrizations of any distance using generalized
statistical mixtures derived from abstract means. In particular, we first show
that the geometric mean is well-suited for exponential families, and report two
closed-form formula for (i) the geometric Jensen-Shannon divergence between
probability densities of the same exponential family, and (ii) the geometric
JS-symmetrization of the reverse Kullback-Leibler divergence. As a second
illustrating example, we show that the harmonic mean is well-suited for the
scale Cauchy distributions, and report a closed-form formula for the harmonic
Jensen-Shannon divergence between scale Cauchy distributions. We also define
generalized Jensen-Shannon divergences between matrices (e.g., quantum
Jensen-Shannon divergences) and consider clustering with respect to these novel
Jensen-Shannon divergences.Comment: 30 page
The {\alpha}-divergences associated with a pair of strictly comparable quasi-arithmetic means
We generalize the family of -divergences using a pair of strictly
comparable weighted means. In particular, we obtain the -divergence in the
limit case (a generalization of the Kullback-Leibler
divergence) and the -divergence in the limit case (a
generalization of the reverse Kullback-Leibler divergence). We state the
condition for a pair of quasi-arithmetic means to be strictly comparable, and
report the formula for the quasi-arithmetic -divergences and its
subfamily of bipower homogeneous -divergences which belong to the
Csis\'ar's -divergences. Finally, we show that these generalized
quasi-arithmetic -divergences and -divergences can be decomposed as the
sum of generalized cross-entropies minus entropies, and rewritten as conformal
Bregman divergences using monotone embeddings.Comment: 18 page
On a generalization of the Jensen-Shannon divergence
The Jensen-Shannon divergence is a renown bounded symmetrization of the
Kullback-Leibler divergence which does not require probability densities to
have matching supports. In this paper, we introduce a vector-skew
generalization of the scalar -Jensen-Bregman divergences and derive
thereof the vector-skew -Jensen-Shannon divergences. We study the
properties of these novel divergences and show how to build parametric families
of symmetric Jensen-Shannon-type divergences. Finally, we report an iterative
algorithm to numerically compute the Jensen-Shannon-type centroids for a set of
probability densities belonging to a mixture family: This includes the case of
the Jensen-Shannon centroid of a set of categorical distributions or normalized
histograms.Comment: 19 pages, 3 figure
On Clustering Histograms with k-Means by Using Mixed α-Divergences
Clustering sets of histograms has become popular thanks to the success of the generic method of bag-of-X used in text categorization and in visual categorization applications. In this paper, we investigate the use of a parametric family of distortion measures, called the α-divergences, for clustering histograms. Since it usually makes sense to deal with symmetric divergences in information retrieval systems, we symmetrize the α -divergences using the concept of mixed divergences. First, we present a novel extension of k-means clustering to mixed divergences. Second, we extend the k-means++ seeding to mixed α-divergences and report a guaranteed probabilistic bound. Finally, we describe a soft clustering technique for mixed α-divergences
Information geometry
This Special Issue of the journal Entropy, titled “Information Geometry I”, contains a collection of 17 papers concerning the foundations and applications of information geometry. Based on a geometrical interpretation of probability, information geometry has become a rich mathematical field employing the methods of differential geometry. It has numerous applications to data science, physics, and neuroscience. Presenting original research, yet written in an accessible, tutorial style, this collection of papers will be useful for scientists who are new to the field, while providing an excellent reference for the more experienced researcher. Several papers are written by authorities in the field, and topics cover the foundations of information geometry, as well as applications to statistics, Bayesian inference, machine learning, complex systems, physics, and neuroscience