6 research outputs found
A Poisson Decomposition for Information and the Information-Event Diagram
Information diagram and the I-measure are useful mnemonics where random
variables are treated as sets, and entropy and mutual information are treated
as a signed measure. Although the I-measure has been successful in machine
proofs of entropy inequalities, the theoretical underpinning of the ``random
variables as sets'' analogy has been unclear until the recent works on mappings
from random variables to sets by Ellerman (recovering order- Tsallis entropy
over general probability space), and Down and Mediano (recovering Shannon
entropy over discrete probability space). We generalize these constructions by
designing a mapping which recovers the Shannon entropy (and the information
density) over general probability space. Moreover, it has an intuitive
interpretation based on the arrival time in a Poisson process, allowing us to
understand the union, intersection and difference between (sets corresponding
to) random variables and events. Cross entropy, KL divergence, and conditional
entropy given an event, can be obtained as set intersections. We propose a
generalization of the information diagram that also includes events, and
demonstrate its usage by a diagrammatic proof of Fano's inequality.Comment: 18 pages, 6 figure
Proving Information Inequalities by Gaussian Elimination
The proof of information inequalities and identities under linear constraints
on the information measures is an important problem in information theory. For
this purpose, ITIP and other variant algorithms have been developed and
implemented, which are all based on solving a linear program (LP). In this
paper, we develop a method with symbolic computation. Compared with the known
methods, our approach can completely avoids the use of linear programming which
may cause numerical errors. Our procedures are also more efficient
computationally.Comment: arXiv admin note: text overlap with arXiv:2202.0278