Minimum Rates of Approximate Sufficient Statistics

Given a sufficient statistic for a parametric family of distributions, one can estimate the parameter without access to the data. However, the memory or code size for storing the sufficient statistic may nonetheless still be prohibitive. Indeed, for $n$ independent samples drawn from a $k$-nomial distribution with $d=k-1$ degrees of freedom, the length of the code scales as $d\log n+O(1)$. In many applications, we may not have a useful notion of sufficient statistics (e.g., when the parametric family is not an exponential family) and we also may not need to reconstruct the generating distribution exactly. By adopting a Shannon-theoretic approach in which we allow a small error in estimating the generating distribution, we construct various {\em approximate sufficient statistics} and show that the code length can be reduced to $\frac{d}{2}\log n+O(1)$. We consider errors measured according to the relative entropy and variational distance criteria. For the code constructions, we leverage Rissanen's minimum description length principle, which yields a non-vanishing error measured according to the relative entropy. For the converse parts, we use Clarke and Barron's formula for the relative entropy of a parametrized distribution and the corresponding mixture distribution. However, this method only yields a weak converse for the variational distance. We develop new techniques to achieve vanishing errors and we also prove strong converses. The latter means that even if the code is allowed to have a non-vanishing error, its length must still be at least $\frac{d}{2}\log n$.Comment: To appear in the IEEE Transactions on Information Theor

Finite-Block-Length Analysis in Classical and Quantum Information Theory

Coding technology is used in several information processing tasks. In particular, when noise during transmission disturbs communications, coding technology is employed to protect the information. However, there are two types of coding technology: coding in classical information theory and coding in quantum information theory. Although the physical media used to transmit information ultimately obey quantum mechanics, we need to choose the type of coding depending on the kind of information device, classical or quantum, that is being used. In both branches of information theory, there are many elegant theoretical results under the ideal assumption that an infinitely large system is available. In a realistic situation, we need to account for finite size effects. The present paper reviews finite size effects in classical and quantum information theory with respect to various topics, including applied aspects

Variable-Length Coding with Cost Allowing Non-Vanishing Error Probability

We derive a general formula of the minimum achievable rate for fixed-to-variable length coding with a regular cost function by allowing the error probability up to a constant $\varepsilon$. For a fixed-to-variable length code, we call the set of source sequences that can be decoded without error the dominant set of source sequences. For any two regular cost functions, it is revealed that the dominant set of source sequences for a code attaining the minimum achievable rate with a cost function is also the dominant set for a code attaining the minimum achievable rate with the other cost function. We also give a general formula of the second-order minimum achievable rate.Comment: 7 pages; extended version of a paper accepted by ISITA201

Broadcast Capacity Region of Two-Phase Bidirectional Relaying

In a three-node network a half-duplex relay node enables bidirectional communication between two nodes with a spectral efficient two phase protocol. In the first phase, two nodes transmit their message to the relay node, which decodes the messages and broadcast a re-encoded composition in the second phase. In this work we determine the capacity region of the broadcast phase. In this scenario each receiving node has perfect information about the message that is intended for the other node. The resulting set of achievable rates of the two-phase bidirectional relaying includes the region which can be achieved by applying XOR on the decoded messages at the relay node. We also prove the strong converse for the maximum error probability and show that this implies that the [\eps_1,\eps_2]-capacity region defined with respect to the average error probability is constant for small values of error parameters \eps_1, \eps_2.Comment: 25 pages, 2 figures, submitted to IEEE Transactions on Information Theor

Strongly Secure Privacy Amplification Cannot Be Obtained by Encoder of Slepian-Wolf Code

The privacy amplification is a technique to distill a secret key from a random variable by a function so that the distilled key and eavesdropper's random variable are statistically independent. There are three kinds of security criteria for the key distilled by the privacy amplification: the normalized divergence criterion, which is also known as the weak security criterion, the variational distance criterion, and the divergence criterion, which is also known as the strong security criterion. As a technique to distill a secret key, it is known that the encoder of a Slepian-Wolf (the source coding with full side-information at the decoder) code can be used as a function for the privacy amplification if we employ the weak security criterion. In this paper, we show that the encoder of a Slepian-Wolf code cannot be used as a function for the privacy amplification if we employ the criteria other than the weak one.Comment: 10 pages, no figure, A part of this paper will be presented at 2009 IEEE International Symposium on Information Theory in Seoul, Korea. Version 2 is a published version. The results are not changed from version 1. Explanations are polished and some references are added. In version 3, only style and DOI are edite