Guaranteed bounds on the Kullback-Leibler divergence of univariate mixtures using piecewise log-sum-exp inequalities

Information-theoretic measures such as the entropy, cross-entropy and the Kullback-Leibler divergence between two mixture models is a core primitive in many signal processing tasks. Since the Kullback-Leibler divergence of mixtures provably does not admit a closed-form formula, it is in practice either estimated using costly Monte-Carlo stochastic integration, approximated, or bounded using various techniques. We present a fast and generic method that builds algorithmically closed-form lower and upper bounds on the entropy, the cross-entropy and the Kullback-Leibler divergence of mixtures. We illustrate the versatile method by reporting on our experiments for approximating the Kullback-Leibler divergence between univariate exponential mixtures, Gaussian mixtures, Rayleigh mixtures, and Gamma mixtures.Comment: 20 pages, 3 figure

Pseudolikelihood inference for Gibbsian T-tessellations ... and point processes

Recently a new class of planar tessellations, named T-tessellations, was introduced. Splits, merges and a third local modification named flip where shown to be sufficient for exploring the space of T-tessellations. Based on these local transformations and by analogy with point process theory, tools Campbell measures and a general simulation algorithm of Metropolis-Hastings-Green type were translated for random T-tessellations.The current report is concerned with parametric inference for Gibbs models of T-tessellations. The estimation criterion referred to as the pseudolikelihood is derived from Campbell measures of random T-tessellations and the Kullback-Leibler divergence. A detailed algorithm for approximating the pseudolikelihood maximum is provided. A simulation study seems to show that bias and variability of the pseudolikelihood maximum decrease when the tessellated domain grows in size.In the last part of the report, it is shown that an analogous approach based on the Campbell measure and the KL divergence when applied to point processes leads to the well-known pseudo-likelihood introduced by Besag. More surprisingly, the binomial regression method recently proposed by Baddeley and his co-authors for computing the pseudolikelihood maximum can be derived using the same approach starting from a slight modification of the Campbell measure

Geometry of escort distributions

Given an original distribution, its statistical and probabilistic attributs may be scanned by the associated escort distribution introduced by Beck and Schlogl and employed in the formulation of nonextensive statistical mechanics. Here, the geometric structure of the one-parameter family of the escort distributions is studied based on the Kullback-Leibler divergence and the relevant Fisher metric. It is shown that the Fisher metric is given in terms of the generalized bit-variance, which measures fluctuations of the crowding index of a multifractal. The Cramer-Rao inequality leads to the fundamental limit for precision of statistical estimate of the order of the escort distribution. It is also quantitatively discussed how inappropriate it is to use the original distribution instead of the escort distribution for calculating the expectation values of physical quantities in nonextensive statistical mechanics.Comment: 12 pages, no figure

Information Measures: the Curious Case of the Binary Alphabet

Four problems related to information divergence measures defined on finite alphabets are considered. In three of the cases we consider, we illustrate a contrast which arises between the binary-alphabet and larger-alphabet settings. This is surprising in some instances, since characterizations for the larger-alphabet settings do not generalize their binary-alphabet counterparts. Specifically, we show that $f$-divergences are not the unique decomposable divergences on binary alphabets that satisfy the data processing inequality, thereby clarifying claims that have previously appeared in the literature. We also show that KL divergence is the unique Bregman divergence which is also an $f$-divergence for any alphabet size. We show that KL divergence is the unique Bregman divergence which is invariant to statistically sufficient transformations of the data, even when non-decomposable divergences are considered. Like some of the problems we consider, this result holds only when the alphabet size is at least three.Comment: to appear in IEEE Transactions on Information Theor