Generalization Bounds via Information Density and Conditional Information Density

We present a general approach, based on an exponential inequality, to derive bounds on the generalization error of randomized learning algorithms. Using this approach, we provide bounds on the average generalization error as well as bounds on its tail probability, for both the PAC-Bayesian and single-draw scenarios. Specifically, for the case of subgaussian loss functions, we obtain novel bounds that depend on the information density between the training data and the output hypothesis. When suitably weakened, these bounds recover many of the information-theoretic available bounds in the literature. We also extend the proposed exponential-inequality approach to the setting recently introduced by Steinke and Zakynthinou (2020), where the learning algorithm depends on a randomly selected subset of the available training data. For this setup, we present bounds for bounded loss functions in terms of the conditional information density between the output hypothesis and the random variable determining the subset choice, given all training data. Through our approach, we recover the average generalization bound presented by Steinke and Zakynthinou (2020) and extend it to the PAC-Bayesian and single-draw scenarios. For the single-draw scenario, we also obtain novel bounds in terms of the conditional $\alpha$-mutual information and the conditional maximal leakage.Comment: Published in Journal on Selected Areas in Information Theory (JSAIT). Important note: the proof of the data-dependent bounds provided in the paper contains an error, which is rectified in the following document: https://gdurisi.github.io/files/2021/jsait-correction.pd

High-SNR Capacity of Wireless Communication Channels in the Noncoherent Setting: A Primer

This paper, mostly tutorial in nature, deals with the problem of characterizing the capacity of fading channels in the high signal-to-noise ratio (SNR) regime. We focus on the practically relevant noncoherent setting, where neither transmitter nor receiver know the channel realizations, but both are aware of the channel law. We present, in an intuitive and accessible form, two tools, first proposed by Lapidoth & Moser (2003), of fundamental importance to high-SNR capacity analysis: the duality approach and the escape-to-infinity property of capacity-achieving distributions. Furthermore, we apply these tools to refine some of the results that appeared previously in the literature and to simplify the corresponding proofs.Comment: To appear in Int. J. Electron. Commun. (AE\"U), Aug. 201

Bounds on the Per-Sample Capacity of Zero-Dispersion Simplified Fiber-Optical Channel Models

A number of simplified models, based on perturbation theory, have been proposed for the fiber-optical channel and have been extensively used in the literature. Although these models are mainly developed for the low-power regime, they are used at moderate or high powers as well. It remains unclear to what extent the capacity of these models is affected by the simplifying assumptions under which they are derived. In this paper, we consider single channel data transmission based on three continuous-time optical models i) a regular perturbative channel, ii) a logarithmic perturbative channel, and iii) the stochastic nonlinear Schr\"odinger (NLS) channel. We apply two simplifying assumptions on these channels to obtain analytically tractable discrete-time models. Namely, we neglect the channel memory (fiber dispersion) and we use a sampling receiver. These assumptions bring into question the physical relevance of the models studied in the paper. Therefore, the results should be viewed as a first step toward analyzing more realistic channels. We investigate the per-sample capacity of the simplified discrete-time models. Specifically, i) we establish tight bounds on the capacity of the regular perturbative channel; ii) we obtain the capacity of the logarithmic perturbative channel; and iii) we present a novel upper bound on the capacity of the zero-dispersion NLS channel. Our results illustrate that the capacity of these models departs from each other at high powers because these models yield different capacity pre-logs. Since all three models are based on the same physical channel, our results highlight that care must be exercised in using simplified channel models in the high-power regime

Analysis of Massive MIMO With Hardware Impairments and Different Channel Models

Massive Multiple-Input Multiple-Output (MIMO) is foreseen to be one of the main technology components in next generation cellular communications (5G). In this paper, fundamental limits on the performance of downlink massive MIMO systems are investigated by means of simulations and analytical analysis. Signal-to-noise-and-interference ratio (SINR) and sum rate for a single-cell scenario multi-user MIMO are analyzed for different array sizes, channel models, and precoding schemes. The impact of hardware impairments on performance is also investigated. Simple approximations are derived that show explicitly how the number of antennas, number of served users, transmit power, and magnitude of hardware impairments affect performance.Comment: 5 pages, 5 figure

Improved Sparsity Thresholds Through Dictionary Splitting

Known sparsity thresholds for basis pursuit to deliver the maximally sparse solution of the compressed sensing recovery problem typically depend on the dictionary's coherence. While the coherence is easy to compute, it can lead to rather pessimistic thresholds as it captures only limited information about the dictionary. In this paper, we show that viewing the dictionary as the concatenation of two general sub-dictionaries leads to provably better sparsity thresholds--that are explicit in the coherence parameters of the dictionary and of the individual sub-dictionaries. Equivalently, our results can be interpreted as sparsity thresholds for dictionaries that are unions of two general (i.e., not necessarily orthonormal) sub-dictionaries.Comment: IEEE Information Theory Workshop (ITW), Taormina, Italy, Oct. 2009, to appea

Uncertainty Relations and Sparse Signal Recovery for Pairs of General Signal Sets

We present an uncertainty relation for the representation of signals in two different general (possibly redundant or incomplete) signal sets. This uncertainty relation is relevant for the analysis of signals containing two distinct features each of which can be described sparsely in a suitable general signal set. Furthermore, the new uncertainty relation is shown to lead to improved sparsity thresholds for recovery of signals that are sparse in general dictionaries. Specifically, our results improve on the well-known $(1+1/d)/2$-threshold for dictionaries with coherence $d$ by up to a factor of two. Furthermore, we provide probabilistic recovery guarantees for pairs of general dictionaries that also allow us to understand which parts of a general dictionary one needs to randomize over to "weed out" the sparsity patterns that prohibit breaking the square-root bottleneck.Comment: submitted to IEEE Trans. Inf. Theor

Minimum Energy to Send kk Bits Over Multiple-Antenna Fading Channels

This paper investigates the minimum energy required to transmit $k$ information bits with a given reliability over a multiple-antenna Rayleigh block-fading channel, with and without channel state information (CSI) at the receiver. No feedback is assumed. It is well known that the ratio between the minimum energy per bit and the noise level converges to $-1.59$ dB as $k$ goes to infinity, regardless of whether CSI is available at the receiver or not. This paper shows that lack of CSI at the receiver causes a slowdown in the speed of convergence to $-1.59$ dB as $k\to\infty$ compared to the case of perfect receiver CSI. Specifically, we show that, in the no-CSI case, the gap to $-1.59$ dB is proportional to $((\log k) /k)^{1/3}$, whereas when perfect CSI is available at the receiver, this gap is proportional to $1/\sqrt{k}$. In both cases, the gap to $-1.59$ dB is independent of the number of transmit antennas and of the channel's coherence time. Numerically, we observe that, when the receiver is equipped with a single antenna, to achieve an energy per bit of $- 1.5$ dB in the no-CSI case, one needs to transmit at least $7\times 10^7$ information bits, whereas $6\times 10^4$ bits suffice for the case of perfect CSI at the receiver

Capacity of Underspread Noncoherent WSSUS Fading Channels under Peak Signal Constraints

We characterize the capacity of the general class of noncoherent underspread wide-sense stationary uncorrelated scattering (WSSUS) time-frequency-selective Rayleigh fading channels, under peak constraints in time and frequency and in time only. Capacity upper and lower bounds are found which are explicit in the channel's scattering function and allow to identify the capacity-maximizing bandwidth for a given scattering function and a given peak-to-average power ratio.Comment: To be presented at IEEE Int. Symp. Inf. Theory 2007, Nice, Franc

On the capacity of the block-memoryless phase-noise channel

Bounds are presented on the capacity of the block-memoryless phase-noise channel. The bounds capture the first two terms in the asymptotic expansion of capacity for SNR going to infinity and turn out to be tight for a large range of SNR values of practical interest. Through these bounds, the capacity dependency on the coherence time of the phase-noise process is determined