Optimal prefix codes for pairs of geometrically-distributed random variables

Optimal prefix codes are studied for pairs of independent, integer-valued symbols emitted by a source with a geometric probability distribution of parameter $q$, $0{<}q{<}1$. By encoding pairs of symbols, it is possible to reduce the redundancy penalty of symbol-by-symbol encoding, while preserving the simplicity of the encoding and decoding procedures typical of Golomb codes and their variants. It is shown that optimal codes for these so-called two-dimensional geometric distributions are \emph{singular}, in the sense that a prefix code that is optimal for one value of the parameter $q$ cannot be optimal for any other value of $q$. This is in sharp contrast to the one-dimensional case, where codes are optimal for positive-length intervals of the parameter $q$. Thus, in the two-dimensional case, it is infeasible to give a compact characterization of optimal codes for all values of the parameter $q$, as was done in the one-dimensional case. Instead, optimal codes are characterized for a discrete sequence of values of $q$ that provide good coverage of the unit interval. Specifically, optimal prefix codes are described for $q=2^{-1/k}$ ($k\ge 1$), covering the range $q\ge 1/2$, and $q=2^{-k}$ ($k>1$), covering the range $q<1/2$. The described codes produce the expected reduction in redundancy with respect to the one-dimensional case, while maintaining low complexity coding operations.Comment: To appear in IEEE Transactions on Information Theor

Tight Bounds on the R\'enyi Entropy via Majorization with Applications to Guessing and Compression

This paper provides tight bounds on the R\'enyi entropy of a function of a discrete random variable with a finite number of possible values, where the considered function is not one-to-one. To that end, a tight lower bound on the R\'enyi entropy of a discrete random variable with a finite support is derived as a function of the size of the support, and the ratio of the maximal to minimal probability masses. This work was inspired by the recently published paper by Cicalese et al., which is focused on the Shannon entropy, and it strengthens and generalizes the results of that paper to R\'enyi entropies of arbitrary positive orders. In view of these generalized bounds and the works by Arikan and Campbell, non-asymptotic bounds are derived for guessing moments and lossless data compression of discrete memoryless sources.Comment: The paper was published in the Entropy journal (special issue on Probabilistic Methods in Information Theory, Hypothesis Testing, and Coding), vol. 20, no. 12, paper no. 896, November 22, 2018. Online available at https://www.mdpi.com/1099-4300/20/12/89