Information capacity in the weak-signal approximation

We derive an approximate expression for mutual information in a broad class of discrete-time stationary channels with continuous input, under the constraint of vanishing input amplitude or power. The approximation describes the input by its covariance matrix, while the channel properties are described by the Fisher information matrix. This separation of input and channel properties allows us to analyze the optimality conditions in a convenient way. We show that input correlations in memoryless channels do not affect channel capacity since their effect decreases fast with vanishing input amplitude or power. On the other hand, for channels with memory, properly matching the input covariances to the dependence structure of the noise may lead to almost noiseless information transfer, even for intermediate values of the noise correlations. Since many model systems described in mathematical neuroscience and biophysics operate in the high noise regime and weak-signal conditions, we believe, that the described results are of potential interest also to researchers in these areas.Comment: 11 pages, 4 figures; accepted for publication in Physical Review

Efficient LLR Calculation for Non-Binary Modulations over Fading Channels

Log-likelihood ratio (LLR) computation for non-binary modulations over fading channels is complicated. A measure of LLR accuracy on asymmetric binary channels is introduced to facilitate good LLR approximations for non-binary modulations. Considering piecewise linear LLR approximations, we prove convexity of optimizing the coefficients according to this measure. For the optimized approximate LLRs, we report negligible performance losses compared to true LLRs.Comment: Submitted to IEEE Transactions on Communication

Asymptotic Estimates in Information Theory with Non-Vanishing Error Probabilities

This monograph presents a unified treatment of single- and multi-user problems in Shannon's information theory where we depart from the requirement that the error probability decays asymptotically in the blocklength. Instead, the error probabilities for various problems are bounded above by a non-vanishing constant and the spotlight is shone on achievable coding rates as functions of the growing blocklengths. This represents the study of asymptotic estimates with non-vanishing error probabilities. In Part I, after reviewing the fundamentals of information theory, we discuss Strassen's seminal result for binary hypothesis testing where the type-I error probability is non-vanishing and the rate of decay of the type-II error probability with growing number of independent observations is characterized. In Part II, we use this basic hypothesis testing result to develop second- and sometimes, even third-order asymptotic expansions for point-to-point communication. Finally in Part III, we consider network information theory problems for which the second-order asymptotics are known. These problems include some classes of channels with random state, the multiple-encoder distributed lossless source coding (Slepian-Wolf) problem and special cases of the Gaussian interference and multiple-access channels. Finally, we discuss avenues for further research.Comment: Further comments welcom

A Simple Derivation of the Refined Sphere Packing Bound Under Certain Symmetry Hypotheses

A judicious application of the Berry-Esseen theorem via suitable Augustin information measures is demonstrated to be sufficient for deriving the sphere packing bound with a prefactor that is $\mathit{\Omega}\left(n^{-0.5(1-E_{sp}'(R))}\right)$ for all codes on certain families of channels -- including the Gaussian channels and the non-stationary Renyi symmetric channels -- and for the constant composition codes on stationary memoryless channels. The resulting non-asymptotic bounds have definite approximation error terms. As a preliminary result that might be of interest on its own, the trade-off between type I and type II error probabilities in the hypothesis testing problem with (possibly non-stationary) independent samples is determined up to some multiplicative constants, assuming that the probabilities of both types of error are decaying exponentially with the number of samples, using the Berry-Esseen theorem.Comment: 20 page

Coding in the Finite-Blocklength Regime: Bounds based on Laplace Integrals and their Asymptotic Approximations

In this paper we provide new compact integral expressions and associated simple asymptotic approximations for converse and achievability bounds in the finite blocklength regime. The chosen converse and random coding union bounds were taken from the recent work of Polyanskyi-Poor-Verdu, and are investigated under parallel AWGN channels, the AWGN channels, the BI-AWGN channel, and the BSC. The technique we use, which is a generalization of some recent results available from the literature, is to map the probabilities of interest into a Laplace integral, and then solve (or approximate) the integral by use of a steepest descent technique. The proposed results are particularly useful for short packet lengths, where the normal approximation may provide unreliable results.Comment: 29 pages, 10 figures. Submitted to IEEE Trans. on Information Theory. Matlab code available from http://dgt.dei.unipd.it section Download->Finite Blocklength Regim