Lecture Notes on Network Information Theory

These lecture notes have been converted to a book titled Network Information Theory published recently by Cambridge University Press. This book provides a significantly expanded exposition of the material in the lecture notes as well as problems and bibliographic notes at the end of each chapter. The authors are currently preparing a set of slides based on the book that will be posted in the second half of 2012. More information about the book can be found at http://www.cambridge.org/9781107008731/. The previous (and obsolete) version of the lecture notes can be found at http://arxiv.org/abs/1001.3404v4/

LACE: A light-weight, causal model for enhancing coded speech through adaptive convolutions

Classical speech coding uses low-complexity postfilters with zero lookahead to enhance the quality of coded speech, but their effectiveness is limited by their simplicity. Deep Neural Networks (DNNs) can be much more effective, but require high complexity and model size, or added delay. We propose a DNN model that generates classical filter kernels on a per-frame basis with a model of just 300~K parameters and 100~MFLOPS complexity, which is a practical complexity for desktop or mobile device CPUs. The lack of added delay allows it to be integrated into the Opus codec, and we demonstrate that it enables effective wideband encoding for bitrates down to 6 kb/s.Comment: 5 pages, accepted at WASPAA 202

Rate-distortion function for finite block codes: Analysis of symmetric binary hamming problem

Shannon's rate-distortion theory provides an asymptotic analysis, where delays are allowed to grow unbounded. In practice, real-time applications, such as video streaming and network storage, are subject to certain maximum delay. Accordingly, it is imperative to develop a finite-delay framework for analyzing the rate-distortion limit. In this backdrop, we propose an intuitive generalization of Shannon's asymptotic operational framework to finite block codes. In view of the extreme complexity of such framework, we obtain insight by specializing to the symmetric binary hamming problem. Even upon such specialization, the proposed framework is computationally so intensive that accurate evaluation of the finite-delay rate-distortion function is practical only upto a block length of three. In order to obtain further insight, we then propose a lower-complexity lower bound, based on the partition function of natural numbers, whose computation is practical upto a block length of six. Finally, using a simple combinatorial argument, we propose an upper bound to localize the desired rate-distortion function between our lower and upper bounds

File compression using probabilistic grammars and LR parsing

Data compression, the reduction in size of the physical representation of data being stored or transmitted, has long been of interest both as a research topic and as a practical technique. Different methods are used for encoding different classes of data files. The purpose of this research is to compress a class of highly redundant data files whose contents are partially described by a context-free grammar (i.e. text files containing computer programs). An encoding technique is developed for the removal of structural dependancy due to the context-free structure of such files. The technique depends on a type of LR parsing method called LALR(K) (Lookahead LRM). The encoder also pays particular attention to the encoding of editing characters, comments, names and constants. The encoded data maintains the exact information content of the original data. Hence, a decoding technique (depending on the same parsing method) is developed to recover the original information from its compressed representation. The technique is demonstrated by compressing Pascal programs. An optimal coding scheme (based on Huffman codes) is used to encode the parsing alternatives in each parsing state. The decoder uses these codes during the decoding phase. Also Huffman codes, based on the probability of the symbols c oncerned, are used when coding editing characterst comments, names and constants. The sizes of the parsing tables (and subsequently the encoding tables) were considerably reduced by splitting them into a number of sub-tables. The minimum and the average code length of the average program are derived from two different matrices. These matrices are constructed from a probabilistic grammar, and the language generated by this grammar. Finally, various comparisons are made with a related encoding method by using a simple context-free language

The Capacity of Online (Causal) qq-ary Error-Erasure Channels

In the $q$-ary online (or "causal") channel coding model, a sender wishes to communicate a message to a receiver by transmitting a codeword $\mathbf{x} =(x_1,\ldots,x_n) \in \{0,1,\ldots,q-1\}^n$ symbol by symbol via a channel limited to at most $pn$ errors and/or $p^{*} n$ erasures. The channel is "online" in the sense that at the $i$th step of communication the channel decides whether to corrupt the $i$th symbol or not based on its view so far, i.e., its decision depends only on the transmitted symbols $(x_1,\ldots,x_i)$. This is in contrast to the classical adversarial channel in which the corruption is chosen by a channel that has a full knowledge on the sent codeword $\mathbf{x}$. In this work we study the capacity of $q$-ary online channels for a combined corruption model, in which the channel may impose at most $pn$ {\em errors} and at most $p^{*} n$ {\em erasures} on the transmitted codeword. The online channel (in both the error and erasure case) has seen a number of recent studies which present both upper and lower bounds on its capacity. In this work, we give a full characterization of the capacity as a function of $q,p$, and $p^{*}$.Comment: This is a new version of the binary case, which can be found at arXiv:1412.637