Source Coding When the Side Information May Be Delayed

For memoryless sources, delayed side information at the decoder does not improve the rate-distortion function. However, this is not the case for more general sources with memory, as demonstrated by a number of works focusing on the special case of (delayed) feedforward. In this paper, a setting is studied in which the encoder is potentially uncertain about the delay with which measurements of the side information are acquired at the decoder. Assuming a hidden Markov model for the sources, at first, a single-letter characterization is given for the set-up where the side information delay is arbitrary and known at the encoder, and the reconstruction at the destination is required to be (near) lossless. Then, with delay equal to zero or one source symbol, a single-letter characterization is given of the rate-distortion region for the case where side information may be delayed or not, unbeknownst to the encoder. The characterization is further extended to allow for additional information to be sent when the side information is not delayed. Finally, examples for binary and Gaussian sources are provided.Comment: revised July 201

Lecture Notes on Network Information Theory

These lecture notes have been converted to a book titled Network Information Theory published recently by Cambridge University Press. This book provides a significantly expanded exposition of the material in the lecture notes as well as problems and bibliographic notes at the end of each chapter. The authors are currently preparing a set of slides based on the book that will be posted in the second half of 2012. More information about the book can be found at http://www.cambridge.org/9781107008731/. The previous (and obsolete) version of the lecture notes can be found at http://arxiv.org/abs/1001.3404v4/

Entropy and Quantum Kolmogorov Complexity: A Quantum Brudno's Theorem

In classical information theory, entropy rate and Kolmogorov complexity per symbol are related by a theorem of Brudno. In this paper, we prove a quantum version of this theorem, connecting the von Neumann entropy rate and two notions of quantum Kolmogorov complexity, both based on the shortest qubit descriptions of qubit strings that, run by a universal quantum Turing machine, reproduce them as outputs.Comment: 26 pages, no figures. Reference to publication added: published in the Communications in Mathematical Physics (http://www.springerlink.com/content/1432-0916/

Fundamental limitations on communication channels with noisy feedback: information flow, capacity and bounds

Since the success of obtaining the capacity (i.e. the maximal achievable transmission rate under which the message can be recovered with arbitrarily small probability of error) for non-feedback point-to-point communication channels by C. Shannon (in 1948), Information Theory has been proved to be a powerful tool to derive fundamental limitations in communication systems. During the last decade, motivated by the emerging of networked systems, information theorists have turned lots of their attention to communication channels with feedback (through another channel from receiver to transmitter). Under the assumption that the feedback channel is noiseless, a large body of notable results have been derived, although much work still needs to be done. However, when this ideal assumption is removed, i.e., the feedback channel is noisy, only few valuable results can be found in the literature and many challenging problems are still open. This thesis aims to address some of these long-standing noisy feedback problems, with concentration on the channel capacity. First of all, we analyze the fundamental information flow in noisy feedback channels. We introduce a new notion, the residual directed information, in order to characterize the noisy feedback channel capacity for which the standard directed information can not be used. As an illustration, finite-alphabet noisy feedback channels have been studied in details. Next, we provide an information flow decomposition equality which serves as a foundation of other novel results in this thesis. With the result of information flow decomposition in hand, we next investigate time-varying Gaussian channels with additive Gaussian noise feedback. Following the notable Cover-Pombra results in 1989, we define the n-block noisy feedback capacity and derive a pair of n-block upper and lower bounds on the n-block noisy feedback capacity. These bounds can be obtained by efficiently solving convex optimization problems. Under the assumption of stationarity on the additive Gaussian noises, we show that the limits of these n-block bounds can be characterized in a power spectral optimization form. In addition, two computable lower bounds are derived for the Shannon capacity. Next, we consider a class of channels where feedback could not increase the capacity and thus the noisy feedback capacity equals to the non-feedback capacity. We derive a necessary condition (characterized by the directed information) for the capacity-achieving channel codes. The condition implies that using noisy feedback is detrimental to achievable rate, i.e, the capacity can not be achieved by using noisy feedback. Finally, we introduce a new framework of communication channels with noisy feedback where the feedback information received by the transmitter is also available to the decoder with some finite delays. We investigate the capacity and linear coding schemes for this extended noisy feedback channels. To summarize, this thesis firstly provides a foundation (i.e. information flow analysis) for analyzing communications channels with noisy feedback. In light of this analysis, we next present a sequence of novel results, e.g. channel coding theorem, capacity bounds, etc., which result in a significant step forward to address the long-standing noisy feedback problem

Upper Bounds on the Capacities of Noncontrollable Finite-State Channels with/without Feedback

Noncontrollable finite-state channels (FSCs) are FSCs in which the channel inputs have no influence on the channel states, i.e., the channel states evolve freely. Since single-letter formulae for the channel capacities are rarely available for general noncontrollable FSCs, computable bounds are usually utilized to numerically bound the capacities. In this paper, we take the delayed channel state as part of the channel input and then define the {\em directed information rate} from the new channel input (including the source and the delayed channel state) sequence to the channel output sequence. With this technique, we derive a series of upper bounds on the capacities of noncontrollable FSCs with/without feedback. These upper bounds can be achieved by conditional Markov sources and computed by solving an average reward per stage stochastic control problem (ARSCP) with a compact state space and a compact action space. By showing that the ARSCP has a uniformly continuous reward function, we transform the original ARSCP into a finite-state and finite-action ARSCP that can be solved by a value iteration method. Under a mild assumption, the value iteration algorithm is convergent and delivers a near-optimal stationary policy and a numerical upper bound.Comment: 15 pages, Two columns, 6 figures; appears in IEEE Transaction on Information Theor

Proceedings of the Fifth Workshop on Information Theoretic Methods in Science and Engineering

These are the online proceedings of the Fifth Workshop on Information Theoretic Methods in Science and Engineering (WITMSE), which was held in the Trippenhuis, Amsterdam, in August 2012