9 research outputs found
Tight Bounds on the R\'enyi Entropy via Majorization with Applications to Guessing and Compression
This paper provides tight bounds on the R\'enyi entropy of a function of a
discrete random variable with a finite number of possible values, where the
considered function is not one-to-one. To that end, a tight lower bound on the
R\'enyi entropy of a discrete random variable with a finite support is derived
as a function of the size of the support, and the ratio of the maximal to
minimal probability masses. This work was inspired by the recently published
paper by Cicalese et al., which is focused on the Shannon entropy, and it
strengthens and generalizes the results of that paper to R\'enyi entropies of
arbitrary positive orders. In view of these generalized bounds and the works by
Arikan and Campbell, non-asymptotic bounds are derived for guessing moments and
lossless data compression of discrete memoryless sources.Comment: The paper was published in the Entropy journal (special issue on
Probabilistic Methods in Information Theory, Hypothesis Testing, and Coding),
vol. 20, no. 12, paper no. 896, November 22, 2018. Online available at
https://www.mdpi.com/1099-4300/20/12/89
Information-Distilling Quantizers
Let and be dependent random variables. This paper considers the
problem of designing a scalar quantizer for to maximize the mutual
information between the quantizer's output and , and develops fundamental
properties and bounds for this form of quantization, which is connected to the
log-loss distortion criterion. The main focus is the regime of low ,
where it is shown that, if is binary, a constant fraction of the mutual
information can always be preserved using
quantization levels, and there exist distributions for which this many
quantization levels are necessary. Furthermore, for larger finite alphabets , it is established that an -fraction of the
mutual information can be preserved using roughly quantization levels
Bottleneck Problems: Information and Estimation-Theoretic View
Information bottleneck (IB) and privacy funnel (PF) are two closely related
optimization problems which have found applications in machine learning, design
of privacy algorithms, capacity problems (e.g., Mrs. Gerber's Lemma), strong
data processing inequalities, among others. In this work, we first investigate
the functional properties of IB and PF through a unified theoretical framework.
We then connect them to three information-theoretic coding problems, namely
hypothesis testing against independence, noisy source coding and dependence
dilution. Leveraging these connections, we prove a new cardinality bound for
the auxiliary variable in IB, making its computation more tractable for
discrete random variables.
In the second part, we introduce a general family of optimization problems,
termed as \textit{bottleneck problems}, by replacing mutual information in IB
and PF with other notions of mutual information, namely -information and
Arimoto's mutual information. We then argue that, unlike IB and PF, these
problems lead to easily interpretable guarantee in a variety of inference tasks
with statistical constraints on accuracy and privacy. Although the underlying
optimization problems are non-convex, we develop a technique to evaluate
bottleneck problems in closed form by equivalently expressing them in terms of
lower convex or upper concave envelope of certain functions. By applying this
technique to binary case, we derive closed form expressions for several
bottleneck problems
On Data-Processing and Majorization Inequalities for -Divergences with Applications
This paper is focused on derivations of data-processing and majorization
inequalities for -divergences, and their applications in information theory
and statistics. For the accessibility of the material, the main results are
first introduced without proofs, followed by exemplifications of the theorems
with further related analytical results, interpretations, and
information-theoretic applications. One application refers to the performance
analysis of list decoding with either fixed or variable list sizes; some
earlier bounds on the list decoding error probability are reproduced in a
unified way, and new bounds are obtained and exemplified numerically. Another
application is related to a study of the quality of approximating a probability
mass function, induced by the leaves of a Tunstall tree, by an equiprobable
distribution. The compression rates of finite-length Tunstall codes are further
analyzed for asserting their closeness to the Shannon entropy of a memoryless
and stationary discrete source. Almost all the analysis is relegated to the
appendices, which form a major part of this manuscript.Comment: This paper is published in the Entropy journal, vol. 21, no. 10,
paper 1022, pages 1-80, October 21, 2019
(https://www.mdpi.com/1099-4300/21/10/1022
Bounds on the Entropy of a Function of a Random Variable and Their Applications
It is well known that the entropy H(X) of a discrete random variable X is always greater than or equal to the entropy H(f(X)) of a function f of X, with equality if and only if f is one-to-one. In this paper, we give tight bounds on H(f(X)), when the function f is not one-to-one, and we illustrate a few scenarios, where this matters. As an intermediate step toward our main result, we derive a lower bound on the entropy of a probability distribution, when only a bound on the ratio between the maximal and minimal probabilities is known. The lower bound improves on previous results in the literature, and it could find applications outside the present scenario