7 research outputs found
Information-Distilling Quantizers
Let and be dependent random variables. This paper considers the
problem of designing a scalar quantizer for to maximize the mutual
information between the quantizer's output and , and develops fundamental
properties and bounds for this form of quantization, which is connected to the
log-loss distortion criterion. The main focus is the regime of low ,
where it is shown that, if is binary, a constant fraction of the mutual
information can always be preserved using
quantization levels, and there exist distributions for which this many
quantization levels are necessary. Furthermore, for larger finite alphabets , it is established that an -fraction of the
mutual information can be preserved using roughly quantization levels
Bottleneck Problems: Information and Estimation-Theoretic View
Information bottleneck (IB) and privacy funnel (PF) are two closely related
optimization problems which have found applications in machine learning, design
of privacy algorithms, capacity problems (e.g., Mrs. Gerber's Lemma), strong
data processing inequalities, among others. In this work, we first investigate
the functional properties of IB and PF through a unified theoretical framework.
We then connect them to three information-theoretic coding problems, namely
hypothesis testing against independence, noisy source coding and dependence
dilution. Leveraging these connections, we prove a new cardinality bound for
the auxiliary variable in IB, making its computation more tractable for
discrete random variables.
In the second part, we introduce a general family of optimization problems,
termed as \textit{bottleneck problems}, by replacing mutual information in IB
and PF with other notions of mutual information, namely -information and
Arimoto's mutual information. We then argue that, unlike IB and PF, these
problems lead to easily interpretable guarantee in a variety of inference tasks
with statistical constraints on accuracy and privacy. Although the underlying
optimization problems are non-convex, we develop a technique to evaluate
bottleneck problems in closed form by equivalently expressing them in terms of
lower convex or upper concave envelope of certain functions. By applying this
technique to binary case, we derive closed form expressions for several
bottleneck problems