373 research outputs found
Unifying computational entropies via Kullback-Leibler divergence
We introduce hardness in relative entropy, a new notion of hardness for
search problems which on the one hand is satisfied by all one-way functions and
on the other hand implies both next-block pseudoentropy and inaccessible
entropy, two forms of computational entropy used in recent constructions of
pseudorandom generators and statistically hiding commitment schemes,
respectively. Thus, hardness in relative entropy unifies the latter two notions
of computational entropy and sheds light on the apparent "duality" between
them. Additionally, it yields a more modular and illuminating proof that
one-way functions imply next-block inaccessible entropy, similar in structure
to the proof that one-way functions imply next-block pseudoentropy (Vadhan and
Zheng, STOC '12)
Extropy: Complementary Dual of Entropy
This article provides a completion to theories of information based on
entropy, resolving a longstanding question in its axiomatization as proposed by
Shannon and pursued by Jaynes. We show that Shannon's entropy function has a
complementary dual function which we call "extropy." The entropy and the
extropy of a binary distribution are identical. However, the measure bifurcates
into a pair of distinct measures for any quantity that is not merely an event
indicator. As with entropy, the maximum extropy distribution is also the
uniform distribution, and both measures are invariant with respect to
permutations of their mass functions. However, they behave quite differently in
their assessments of the refinement of a distribution, the axiom which
concerned Shannon and Jaynes. Their duality is specified via the relationship
among the entropies and extropies of course and fine partitions. We also
analyze the extropy function for densities, showing that relative extropy
constitutes a dual to the Kullback-Leibler divergence, widely recognized as the
continuous entropy measure. These results are unified within the general
structure of Bregman divergences. In this context they identify half the
metric as the extropic dual to the entropic directed distance. We describe a
statistical application to the scoring of sequential forecast distributions
which provoked the discovery.Comment: Published at http://dx.doi.org/10.1214/14-STS430 in the Statistical
Science (http://www.imstat.org/sts/) by the Institute of Mathematical
Statistics (http://www.imstat.org
The Burbea-Rao and Bhattacharyya centroids
We study the centroid with respect to the class of information-theoretic
Burbea-Rao divergences that generalize the celebrated Jensen-Shannon divergence
by measuring the non-negative Jensen difference induced by a strictly convex
and differentiable function. Although those Burbea-Rao divergences are
symmetric by construction, they are not metric since they fail to satisfy the
triangle inequality. We first explain how a particular symmetrization of
Bregman divergences called Jensen-Bregman distances yields exactly those
Burbea-Rao divergences. We then proceed by defining skew Burbea-Rao
divergences, and show that skew Burbea-Rao divergences amount in limit cases to
compute Bregman divergences. We then prove that Burbea-Rao centroids are
unique, and can be arbitrarily finely approximated by a generic iterative
concave-convex optimization algorithm with guaranteed convergence property. In
the second part of the paper, we consider the Bhattacharyya distance that is
commonly used to measure overlapping degree of probability distributions. We
show that Bhattacharyya distances on members of the same statistical
exponential family amount to calculate a Burbea-Rao divergence in disguise.
Thus we get an efficient algorithm for computing the Bhattacharyya centroid of
a set of parametric distributions belonging to the same exponential families,
improving over former specialized methods found in the literature that were
limited to univariate or "diagonal" multivariate Gaussians. To illustrate the
performance of our Bhattacharyya/Burbea-Rao centroid algorithm, we present
experimental performance results for -means and hierarchical clustering
methods of Gaussian mixture models.Comment: 13 page
Editorial Comment on the Special Issue of "Information in Dynamical Systems and Complex Systems"
This special issue collects contributions from the participants of the
"Information in Dynamical Systems and Complex Systems" workshop, which cover a
wide range of important problems and new approaches that lie in the
intersection of information theory and dynamical systems. The contributions
include theoretical characterization and understanding of the different types
of information flow and causality in general stochastic processes, inference
and identification of coupling structure and parameters of system dynamics,
rigorous coarse-grain modeling of network dynamical systems, and exact
statistical testing of fundamental information-theoretic quantities such as the
mutual information. The collective efforts reported herein reflect a modern
perspective of the intimate connection between dynamical systems and
information flow, leading to the promise of better understanding and modeling
of natural complex systems and better/optimal design of engineering systems
vsgoftest: An R Package for Goodness-of-Fit Testing Based on Kullback-Leibler Divergence
The R package vsgoftest performs goodness-of-fit (GOF) tests, based on Shannon entropy and Kullback-Leibler divergence, developed by Vasicek (1976) and Song (2002), of various classical families of distributions. The so-called Vasicek-Song (VS) tests are intended to be applied to continuous data - typically drawn from a density distribution, even including ties. Their excellent properties - they exhibit high power in a large variety of situations, make them relevant alternatives to classical GOF tests in any domain of application requiring statistical processing. The theoretical framework of VS tests is summarized and followed by a detailed description of the different features of the package. The power and computational time performances of VS tests are studied through their comparison with other GOF tests. Application to real datasets illustrates the easy-to-use functionalities of the vsgoftest package
On a generalization of the Jensen-Shannon divergence and the JS-symmetrization of distances relying on abstract means
The Jensen-Shannon divergence is a renown bounded symmetrization of the
unbounded Kullback-Leibler divergence which measures the total Kullback-Leibler
divergence to the average mixture distribution. However the Jensen-Shannon
divergence between Gaussian distributions is not available in closed-form. To
bypass this problem, we present a generalization of the Jensen-Shannon (JS)
divergence using abstract means which yields closed-form expressions when the
mean is chosen according to the parametric family of distributions. More
generally, we define the JS-symmetrizations of any distance using generalized
statistical mixtures derived from abstract means. In particular, we first show
that the geometric mean is well-suited for exponential families, and report two
closed-form formula for (i) the geometric Jensen-Shannon divergence between
probability densities of the same exponential family, and (ii) the geometric
JS-symmetrization of the reverse Kullback-Leibler divergence. As a second
illustrating example, we show that the harmonic mean is well-suited for the
scale Cauchy distributions, and report a closed-form formula for the harmonic
Jensen-Shannon divergence between scale Cauchy distributions. We also define
generalized Jensen-Shannon divergences between matrices (e.g., quantum
Jensen-Shannon divergences) and consider clustering with respect to these novel
Jensen-Shannon divergences.Comment: 30 page
Formal Groups and -Entropies
We shall prove that the celebrated R\'enyi entropy is the first example of a
new family of infinitely many multi-parametric entropies. We shall call them
the -entropies. Each of them, under suitable hypotheses, generalizes the
celebrated entropies of Boltzmann and R\'enyi.
A crucial aspect is that every -entropy is composable [1]. This property
means that the entropy of a system which is composed of two or more independent
systems depends, in all the associated probability space, on the choice of the
two systems only. Further properties are also required, to describe the
composition process in terms of a group law.
The composability axiom, introduced as a generalization of the fourth
Shannon-Khinchin axiom (postulating additivity), is a highly non-trivial
requirement. Indeed, in the trace-form class, the Boltzmann entropy and Tsallis
entropy are the only known composable cases. However, in the non-trace form
class, the -entropies arise as new entropic functions possessing the
mathematical properties necessary for information-theoretical applications, in
both classical and quantum contexts.
From a mathematical point of view, composability is intimately related to
formal group theory of algebraic topology. The underlying group-theoretical
structure determines crucially the statistical properties of the corresponding
entropies.Comment: 20 pages, no figure
- …