519 research outputs found
Coding on countably infinite alphabets
This paper describes universal lossless coding strategies for compressing
sources on countably infinite alphabets. Classes of memoryless sources defined
by an envelope condition on the marginal distribution provide benchmarks for
coding techniques originating from the theory of universal coding over finite
alphabets. We prove general upper-bounds on minimax regret and lower-bounds on
minimax redundancy for such source classes. The general upper bounds emphasize
the role of the Normalized Maximum Likelihood codes with respect to minimax
regret in the infinite alphabet context. Lower bounds are derived by tailoring
sharp bounds on the redundancy of Krichevsky-Trofimov coders for sources over
finite alphabets. Up to logarithmic (resp. constant) factors the bounds are
matching for source classes defined by algebraically declining (resp.
exponentially vanishing) envelopes. Effective and (almost) adaptive coding
techniques are described for the collection of source classes defined by
algebraically vanishing envelopes. Those results extend ourknowledge concerning
universal coding to contexts where the key tools from parametric inferenceComment: 33 page
Coding on countably infinite alphabets
33 pagesInternational audienceThis paper describes universal lossless coding strategies for compressing sources on countably infinite alphabets. Classes of memoryless sources defined by an envelope condition on the marginal distribution provide benchmarks for coding techniques originating from the theory of universal coding over finite alphabets. We prove general upper-bounds on minimax regret and lower-bounds on minimax redundancy for such source classes. The general upper bounds emphasize the role of the Normalized Maximum Likelihood codes with respect to minimax regret in the infinite alphabet context. Lower bounds are derived by tailoring sharp bounds on the redundancy of Krichevsky-Trofimov coders for sources over finite alphabets. Up to logarithmic (resp. constant) factors the bounds are matching for source classes defined by algebraically declining (resp. exponentially vanishing) envelopes. Effective and (almost) adaptive coding techniques are described for the collection of source classes defined by algebraically vanishing envelopes. Those results extend ourknowledge concerning universal coding to contexts where the key tools from parametric inferenc
Generalizations of Fano's Inequality for Conditional Information Measures via Majorization Theory
Fano's inequality is one of the most elementary, ubiquitous, and important
tools in information theory. Using majorization theory, Fano's inequality is
generalized to a broad class of information measures, which contains those of
Shannon and R\'{e}nyi. When specialized to these measures, it recovers and
generalizes the classical inequalities. Key to the derivation is the
construction of an appropriate conditional distribution inducing a desired
marginal distribution on a countably infinite alphabet. The construction is
based on the infinite-dimensional version of Birkhoff's theorem proven by
R\'{e}v\'{e}sz [Acta Math. Hungar. 1962, 3, 188{\textendash}198], and the
constraint of maintaining a desired marginal distribution is similar to
coupling in probability theory. Using our Fano-type inequalities for Shannon's
and R\'{e}nyi's information measures, we also investigate the asymptotic
behavior of the sequence of Shannon's and R\'{e}nyi's equivocations when the
error probabilities vanish. This asymptotic behavior provides a novel
characterization of the asymptotic equipartition property (AEP) via Fano's
inequality.Comment: 44 pages, 3 figure
Universal Coding on Infinite Alphabets: Exponentially Decreasing Envelopes
This paper deals with the problem of universal lossless coding on a countable
infinite alphabet. It focuses on some classes of sources defined by an envelope
condition on the marginal distribution, namely exponentially decreasing
envelope classes with exponent . The minimax redundancy of
exponentially decreasing envelope classes is proved to be equivalent to
. Then a coding strategy is proposed, with
a Bayes redundancy equivalent to the maximin redundancy. At last, an adaptive
algorithm is provided, whose redundancy is equivalent to the minimax redundanc
Universal Densities Exist for Every Finite Reference Measure
As it is known, universal codes, which estimate the entropy rate
consistently, exist for stationary ergodic sources over finite alphabets but
not over countably infinite ones. We generalize universal coding as the problem
of universal densities with respect to a fixed reference measure on a countably
generated measurable space. We show that universal densities, which estimate
the differential entropy rate consistently, exist for finite reference
measures. Thus finite alphabets are not necessary in some sense. To exhibit a
universal density, we adapt the non-parametric differential (NPD) entropy rate
estimator by Feutrill and Roughan. Our modification is analogous to Ryabko's
modification of prediction by partial matching (PPM) by Cleary and Witten.
Whereas Ryabko considered a mixture over Markov orders, we consider a mixture
over quantization levels. Moreover, we demonstrate that any universal density
induces a strongly consistent Ces\`aro mean estimator of conditional density
given an infinite past. This yields a universal predictor with the loss
for a countable alphabet. Finally, we specialize universal densities to
processes over natural numbers and on the real line. We derive sufficient
conditions for consistent estimation of the entropy rate with respect to
infinite reference measures in these domains.Comment: 28 pages, no figure
Countably Infinite Multilevel Source Polarization for Non-Stationary Erasure Distributions
Polar transforms are central operations in the study of polar codes. This
paper examines polar transforms for non-stationary memoryless sources on
possibly infinite source alphabets. This is the first attempt of source
polarization analysis over infinite alphabets. The source alphabet is defined
to be a Polish group, and we handle the Ar{\i}kan-style two-by-two polar
transform based on the group. Defining erasure distributions based on the
normal subgroup structure, we give recursive formulas of the polar transform
for our proposed erasure distributions. As a result, the recursive formulas
lead to concrete examples of multilevel source polarization with countably
infinite levels when the group is locally cyclic. We derive this result via
elementary techniques in lattice theory.Comment: 12 pages, 1 figure, a short version has been accepted by the 2019
IEEE International Symposium on Information Theory (ISIT2019
Phase transitions for suspension flows
This paper is devoted to study thermodynamic formalism for suspension flows
defined over countable alphabets. We are mostly interested in the regularity
properties of the pressure function. We establish conditions for the pressure
function to be real analytic or to exhibit a phase transition. We also
construct an example of a potential for which the pressure has countably many
phase transitions.Comment: Example 5.2 expanded. Typos corrected. Section 6.1 superced the note
"Thermodynamic formalism for the positive geodesic flow on the modular
surface" arXiv:1009.462
- …