Asymptotics of input-constrained binary symmetric channel capacity

We study the classical problem of noisy constrained capacity in the case of the binary symmetric channel (BSC), namely, the capacity of a BSC whose inputs are sequences chosen from a constrained set. Motivated by a result of Ordentlich and Weissman [In Proceedings of IEEE Information Theory Workshop (2004) 117--122], we derive an asymptotic formula (when the noise parameter is small) for the entropy rate of a hidden Markov chain, observed when a Markov chain passes through a BSC. Using this result, we establish an asymptotic formula for the capacity of a BSC with input process supported on an irreducible finite type constraint, as the noise parameter tends to zero.Comment: Published in at http://dx.doi.org/10.1214/08-AAP570 the Annals of Applied Probability (http://www.imstat.org/aap/) by the Institute of Mathematical Statistics (http://www.imstat.org

Asymptotic Estimates in Information Theory with Non-Vanishing Error Probabilities

This monograph presents a unified treatment of single- and multi-user problems in Shannon's information theory where we depart from the requirement that the error probability decays asymptotically in the blocklength. Instead, the error probabilities for various problems are bounded above by a non-vanishing constant and the spotlight is shone on achievable coding rates as functions of the growing blocklengths. This represents the study of asymptotic estimates with non-vanishing error probabilities. In Part I, after reviewing the fundamentals of information theory, we discuss Strassen's seminal result for binary hypothesis testing where the type-I error probability is non-vanishing and the rate of decay of the type-II error probability with growing number of independent observations is characterized. In Part II, we use this basic hypothesis testing result to develop second- and sometimes, even third-order asymptotic expansions for point-to-point communication. Finally in Part III, we consider network information theory problems for which the second-order asymptotics are known. These problems include some classes of channels with random state, the multiple-encoder distributed lossless source coding (Slepian-Wolf) problem and special cases of the Gaussian interference and multiple-access channels. Finally, we discuss avenues for further research.Comment: Further comments welcom

Zero-error capacity of binary channels with memory

We begin a systematic study of the problem of the zero--error capacity of noisy binary channels with memory and solve some of the non--trivial cases.Comment: 10 pages. This paper is the revised version of our previous paper having the same title, published on ArXiV on February 3, 2014. We complete Theorem 2 of the previous version by showing here that our previous construction is asymptotically optimal. This proves that the isometric triangles yield different capacities. The new manuscript differs from the old one by the addition of one more pag

On the BICM Capacity

Optimal binary labelings, input distributions, and input alphabets are analyzed for the so-called bit-interleaved coded modulation (BICM) capacity, paying special attention to the low signal-to-noise ratio (SNR) regime. For 8-ary pulse amplitude modulation (PAM) and for 0.75 bit/symbol, the folded binary code results in a higher capacity than the binary reflected gray code (BRGC) and the natural binary code (NBC). The 1 dB gap between the additive white Gaussian noise (AWGN) capacity and the BICM capacity with the BRGC can be almost completely removed if the input symbol distribution is properly selected. First-order asymptotics of the BICM capacity for arbitrary input alphabets and distributions, dimensions, mean, variance, and binary labeling are developed. These asymptotics are used to define first-order optimal (FOO) constellations for BICM, i.e. constellations that make BICM achieve the Shannon limit -1.59 \tr{dB}. It is shown that the \Eb/N_0 required for reliable transmission at asymptotically low rates in BICM can be as high as infinity, that for uniform input distributions and 8-PAM there are only 72 classes of binary labelings with a different first-order asymptotic behavior, and that this number is reduced to only 26 for 8-ary phase shift keying (PSK). A general answer to the question of FOO constellations for BICM is also given: using the Hadamard transform, it is found that for uniform input distributions, a constellation for BICM is FOO if and only if it is a linear projection of a hypercube. A constellation based on PAM or quadrature amplitude modulation input alphabets is FOO if and only if they are labeled by the NBC; if the constellation is based on PSK input alphabets instead, it can never be FOO if the input alphabet has more than four points, regardless of the labeling.Comment: Submitted to the IEEE Transactions on Information Theor

Concavity of Mutual Information Rate for Input-Restricted Finite-State Memoryless Channels at High SNR

We consider a finite-state memoryless channel with i.i.d. channel state and the input Markov process supported on a mixing finite-type constraint. We discuss the asymptotic behavior of entropy rate of the output hidden Markov chain and deduce that the mutual information rate of such a channel is concave with respect to the parameters of the input Markov processes at high signal-to-noise ratio. In principle, the concavity result enables good numerical approximation of the maximum mutual information rate and capacity of such a channel.Comment: 26 page