39 research outputs found
Sumset and Inverse Sumset Inequalities for Differential Entropy and Mutual Information
The sumset and inverse sumset theories of Freiman, Pl\"{u}nnecke and Ruzsa,
give bounds connecting the cardinality of the sumset of two discrete sets , to the cardinalities (or the finer
structure) of the original sets . For example, the sum-difference bound of
Ruzsa states that, , where the difference set . Interpreting the differential entropy of a
continuous random variable as (the logarithm of) the size of the effective
support of , the main contribution of this paper is a series of natural
information-theoretic analogs for these results. For example, the Ruzsa
sum-difference bound becomes the new inequality, , for any pair of independent continuous random variables and .
Our results include differential-entropy versions of Ruzsa's triangle
inequality, the Pl\"{u}nnecke-Ruzsa inequality, and the
Balog-Szemer\'{e}di-Gowers lemma. Also we give a differential entropy version
of the Freiman-Green-Ruzsa inverse-sumset theorem, which can be seen as a
quantitative converse to the entropy power inequality. Versions of most of
these results for the discrete entropy were recently proved by Tao,
relying heavily on a strong, functional form of the submodularity property of
. Since differential entropy is {\em not} functionally submodular, in the
continuous case many of the corresponding discrete proofs fail, in many cases
requiring substantially new proof strategies. We find that the basic property
that naturally replaces the discrete functional submodularity, is the data
processing property of mutual information.Comment: 23 page
Entropy methods for sumset inequalities
In this thesis we present several analogies betweeen sumset inequalities and entropy inequalities. We offer an overview of the different results and techniques that have been developed during the last ten years, starting with a seminal paper by Ruzsa, and also studied by authors such as Bollobás, Madiman, or Tao. After an introduction to the tools from sumset theory and entropy theory, we present and prove many sumset inequalities and their entropy analogues, with a particular emphasis on Plünnecke-type results. Functional submodularity is used to prove many of these, as well as an analogue of the Balog-Szemerédi-Gowers theorem. Partition-determined functions are used to obtain many sumset inequalities analogous to some new entropic results. Their use is generalized to other contexts, such as that of projections or polynomial compound sets. Furthermore, we present a generalization of a tool introduced by Ruzsa by extending it to a much more general setting than that of sumsets. We show how it can be used to obtain many entropy inequalities in a direct and unified way, and we extend its use to more general compound sets. Finally, we show how this device may help in finding new expanders
Recommended from our members
Entropy bounds on abelian groups and the ruzsa divergence
Over the past few years, a family of interesting new inequalities for the
entropies of sums and differences of random variables has been developed by
Ruzsa, Tao and others, motivated by analogous results in additive
combinatorics. The present work extends these earlier results to the case of
random variables taking values in or, more generally, in
arbitrary locally compact and Polish abelian groups. We isolate and study a key
quantity, the Ruzsa divergence between two probability distributions, and we
show that its properties can be used to extend the earlier inequalities to the
present general setting. The new results established include several variations
on the theme that the entropies of the sum and the difference of two
independent random variables severely constrain each other. Although the
setting is quite general, the result are already of interest (and new) for
random vectors in . In that special case, quantitative bounds are
provided for the stability of the equality conditions in the entropy power
inequality; a reverse entropy power inequality for log-concave random vectors
is proved; an information-theoretic analog of the Rogers-Shephard inequality
for convex bodies is established; and it is observed that some of these results
lead to new inequalities for the determinants of positive-definite matrices.
Moreover, by considering the multiplicative subgroups of the complex plane, one
obtains new inequalities for the differential entropies of products and ratios
of nonzero, complex-valued random variables
Conditional R\'enyi entropy and the relationships between R\'enyi capacities
The analogues of Arimoto's definition of conditional R\'enyi entropy and
R\'enyi mutual information are explored for abstract alphabets. These
quantities, although dependent on the reference measure, have some useful
properties similar to those known in the discrete setting. In addition to
laying out some such basic properties and the relations to R\'enyi divergences,
the relationships between the families of mutual informations defined by
Sibson, Augustin-Csisz\'ar, and Lapidoth-Pfister, as well as the corresponding
capacities, are explored.Comment: 17 pages, 1 figur
Information Inequalities via Ideas from Additive Combinatorics
Ruzsa's equivalence theorem provided a framework for converting certain
families of inequalities in additive combinatorics to entropic inequalities
(which sometimes did not possess stand-alone entropic proofs). In this work, we
first establish formal equivalences between some families (different from
Ruzsa) of inequalities in additive combinatorics and entropic ones. As a first
step to further these equivalences, we establish an information-theoretic
characterization of the magnification ratio that could also be of independent
interest.Comment: 15 pages, The authors were made aware that some of the results had
been obtained earlier. The revised version acknowledges and references this
work. A conference version of this was published in the proceeding of IEEE
ISIT 2023.
Countably Infinite Multilevel Source Polarization for Non-Stationary Erasure Distributions
Polar transforms are central operations in the study of polar codes. This
paper examines polar transforms for non-stationary memoryless sources on
possibly infinite source alphabets. This is the first attempt of source
polarization analysis over infinite alphabets. The source alphabet is defined
to be a Polish group, and we handle the Ar{\i}kan-style two-by-two polar
transform based on the group. Defining erasure distributions based on the
normal subgroup structure, we give recursive formulas of the polar transform
for our proposed erasure distributions. As a result, the recursive formulas
lead to concrete examples of multilevel source polarization with countably
infinite levels when the group is locally cyclic. We derive this result via
elementary techniques in lattice theory.Comment: 12 pages, 1 figure, a short version has been accepted by the 2019
IEEE International Symposium on Information Theory (ISIT2019