On Convergence Properties of Shannon Entropy

Convergence properties of Shannon Entropy are studied. In the differential setting, it is shown that weak convergence of probability measures, or convergence in distribution, is not enough for convergence of the associated differential entropies. A general result for the desired differential entropy convergence is provided, taking into account both compactly and uncompactly supported densities. Convergence of differential entropy is also characterized in terms of the Kullback-Liebler discriminant for densities with fairly general supports, and it is shown that convergence in variation of probability measures guarantees such convergence under an appropriate boundedness condition on the densities involved. Results for the discrete setting are also provided, allowing for infinitely supported probability measures, by taking advantage of the equivalence between weak convergence and convergence in variation in this setting.Comment: Submitted to IEEE Transactions on Information Theor

Sum-Rate Capacity for Symmetric Gaussian Multiple Access Channels with Feedback

The feedback sum-rate capacity is established for the symmetric $J$-user Gaussian multiple-access channel (GMAC). The main contribution is a converse bound that combines the dependence-balance argument of Hekstra and Willems (1989) with a variant of the factorization of a convex envelope of Geng and Nair (2014). The converse bound matches the achievable sum-rate of the Fourier-Modulated Estimate Correction strategy of Kramer (2002).Comment: 16 pages, 2 figures, published in International Symposium on Information Theory (ISIT) 201

CSI-based versus RSS-based Secret-Key Generation under Correlated Eavesdropping

Physical-layer security (PLS) has the potential to strongly enhance the overall system security as an alternative to or in combination with conventional cryptographic primitives usually implemented at higher network layers. Secret-key generation relying on wireless channel reciprocity is an interesting solution as it can be efficiently implemented at the physical layer of emerging wireless communication networks, while providing information-theoretic security guarantees. In this paper, we investigate and compare the secret-key capacity based on the sampling of the entire complex channel state information (CSI) or only its envelope, the received signal strength (RSS). Moreover, as opposed to previous works, we take into account the fact that the eavesdropper's observations might be correlated and we consider the high signal-to-noise ratio (SNR) regime where we can find simple analytical expressions for the secret-key capacity. As already found in previous works, we find that RSS-based secret-key generation is heavily penalized as compared to CSI-based systems. At high SNR, we are able to precisely and simply quantify this penalty: a halved pre-log factor and a constant penalty of about 0.69 bit, which disappears as Eve's channel gets highly correlated

On the tightness of Marton's regions for semi-additive broadcast channels

We study cost constrained side-information channels, where the cost function depends on a state which is known only to the encoder. In the additive noise case, we bound the capacity loss due to not knowing the cost state at the decoder and show that it is small under various assumptions, and goes to zero in the limit of weak noise. This model plays an important role in the (non-degraded) broadcast channel. In the semi-additive noise case, we bound the gap between the best known single letter achievable region and the true capacity region, using tools developed for the first problem. In the limit of weak noise, we show that the bounds coincide, thus we get the complete characterization of the capacity region

On the Capacity of Large-MIMO Block-Fading Channels

We characterize the capacity of Rayleigh block-fading multiple-input multiple-output (MIMO) channels in the noncoherent setting where transmitter and receiver have no a priori knowledge of the realizations of the fading channel. We prove that unitary space-time modulation (USTM) is not capacity-achieving in the high signal-to-noise ratio (SNR) regime when the total number of antennas exceeds the coherence time of the fading channel (expressed in multiples of the symbol duration), a situation that is relevant for MIMO systems with large antenna arrays (large-MIMO systems). This result settles a conjecture by Zheng & Tse (2002) in the affirmative. The capacity-achieving input signal, which we refer to as Beta-variate space-time modulation (BSTM), turns out to be the product of a unitary isotropically distributed random matrix, and a diagonal matrix whose nonzero entries are distributed as the square-root of the eigenvalues of a Beta-distributed random matrix of appropriate size. Numerical results illustrate that using BSTM instead of USTM in large-MIMO systems yields a rate gain as large as 13% for SNR values of practical interest.Comment: To appear in IEEE Journal on Selected Areas in Communicatio

Stability of Bernstein's Theorem and Soft Doubling for Vector Gaussian Channels

The stability of Bernstein's characterization of Gaussian distributions is extended to vectors by utilizing characteristic functions. Stability is used to develop a soft doubling argument that establishes the optimality of Gaussian vectors for certain communications channels with additive Gaussian noise, including two-receiver broadcast channels. One novelty is that the argument does not require the existence of distributions that achieve capacity

Entropy-based goodness-of-fit tests for multivariate distributions

Entropy is one of the most basic and significant descriptors of a probability distribution. It is still a commonly used measure of uncertainty and randomness in information theory and mathematical statistics. We study statistical inference for Shannon and Rényi’s entropy-related functionals of multivariate Gaussian and Student-t distributions. This thesis investigates three themes. First, we provide a non-parametric test of goodness-of-fit for a class of multivariate generalized Gaussian distributions based on maximum entropy principle via using the k-th nearest neighbour (NN) distance estimator of the Shannon entropy. Its asymptotic unbiasedness and consistency are demonstrated. Second, we show a class of estimators of the Rényi entropy based on an independent identical distribution sample drawn from an unknown distribution f on R m. We focus on the maximum Rényi entropy principle for multivariate Student-t and Pearson type II distributions. We also consider the entropy-based test for multivariate Student-t distribution using the k-th NN distances estimator of entropy and employ the properties of entropy estimators derived from NN distances. Third, we introduce a new classes of unimodal rotational invariant directional distributions, which generalize von Mises-Fisher distribution. We propose three types of distributions in which one of them represents the axial data. We provide all of the formula together with a short computational study of parameter estimators for each new type via the method of moments and method of maximum likelihood. We also offer the goodness-of-fit test to detect that the sample entries follow one of the introduced generalized von Mises-Fisher distribution based on the maximum entropy principle using the k-th NN distances estimator of Shannon entropy and to prove its L2 -consistence