Estimation of the Rate-Distortion Function

Motivated by questions in lossy data compression and by theoretical considerations, we examine the problem of estimating the rate-distortion function of an unknown (not necessarily discrete-valued) source from empirical data. Our focus is the behavior of the so-called "plug-in" estimator, which is simply the rate-distortion function of the empirical distribution of the observed data. Sufficient conditions are given for its consistency, and examples are provided to demonstrate that in certain cases it fails to converge to the true rate-distortion function. The analysis of its performance is complicated by the fact that the rate-distortion function is not continuous in the source distribution; the underlying mathematical problem is closely related to the classical problem of establishing the consistency of maximum likelihood estimators. General consistency results are given for the plug-in estimator applied to a broad class of sources, including all stationary and ergodic ones. A more general class of estimation problems is also considered, arising in the context of lossy data compression when the allowed class of coding distributions is restricted; analogous results are developed for the plug-in estimator in that case. Finally, consistency theorems are formulated for modified (e.g., penalized) versions of the plug-in, and for estimating the optimal reproduction distribution.Comment: 18 pages, no figures [v2: removed an example with an error; corrected typos; a shortened version will appear in IEEE Trans. Inform. Theory

Rate Distortion Behavior of Sparse Sources

This paper studies the rate distortion behavior of sparse memoryless sources that serve as models of sparse signal representations. For the Hamming distortion criterion, $R(D)$ is shown to be essentially linear. For the mean squared error measure, two models are analyzed: the mixed discrete/continuous spike processes and Gaussian mixtures. The latter are shown to be a better model for ``natural'' data such as sparse wavelet coefficients. Finally, the geometric mean of a continuous random variable is introduced as a sparseness measure. It yields upper and lower bounds on the entropy and thus characterizes high-rate $R(D)$

Information Theory and Machine Learning

The recent successes of machine learning, especially regarding systems based on deep neural networks, have encouraged further research activities and raised a new set of challenges in understanding and designing complex machine learning algorithms. New applications require learning algorithms to be distributed, have transferable learning results, use computation resources efficiently, convergence quickly on online settings, have performance guarantees, satisfy fairness or privacy constraints, incorporate domain knowledge on model structures, etc. A new wave of developments in statistical learning theory and information theory has set out to address these challenges. This Special Issue, "Machine Learning and Information Theory", aims to collect recent results in this direction reflecting a diverse spectrum of visions and efforts to extend conventional theories and develop analysis tools for these complex machine learning systems

To code or not to code

It is well known and surprising that the uncoded transmission of an independent and identically distributed Gaussian source across an additive white Gaussian noise channel is optimal: No amount of sophistication in the coding strategy can ever perform better. What makes uncoded transmission optimal? In this thesis, it is shown that the optimality of uncoded transmission can be understood as the perfect match of four involved measures: the probability distribution of the source, its distortion measure, the conditional probability distribution of the channel, and its input cost function. More generally, what makes a source-channel communication system optimal? Inspired by, and in extension of, the results about uncoded transmission, this can again be understood as the perfect match, now of six quantities: the above, plus the encoding and the decoding functions. The matching condition derived in this thesis is explicit and closed-form. This fact is exploited in various ways, for example to analyze the optimality of source-channel coding systems of finite block length, and involving feedback. In the shape of an intermezzo, the potential impact of our findings on the understanding of biological communication is outlined: owing to its simplicity, uncoded transmission must be an interesting strategy, e.g., for neural communication. The matching condition of this thesis shows that, apart from being simple, uncoded transmission may also be information-theoretically optimal. Uncoded transmission is also a useful point of view in network information theory. In this thesis, it is used to determine network source-channel communication results, including a single-source broadcast scenario, to establish capacity results for Gaussian relay networks, and to give a new example of the fact that separate source and channel coding does not lead to optimal performance in general networks

Proceedings of the Eighth Workshop on Information Theoretic Methods in Science and Engineering

Proceedings of the Eighth Workshop on Information Theoretic Methods in Science and Engineering (WITMSE 2015) held in Copenhagen, Denmark, 24-26 June 2015; published in the series of the Department of Computer Science, University of Helsinki.Peer reviewe