Variable-rate source coding theorems for stationary nonergodic sources

For a stationary ergodic source, the source coding theorem and its converse imply that the optimal performance theoretically achievable by a fixed-rate or variable-rate block quantizer is equal to the distortion-rate function, which is defined as the infimum of an expected distortion subject to a mutual information constraint. For a stationary nonergodic source, however, the. Distortion-rate function cannot in general be achieved arbitrarily closely by a fixed-rate block code. We show, though, that for any stationary nonergodic source with a Polish alphabet, the distortion-rate function can be achieved arbitrarily closely by a variable-rate block code. We also show that the distortion-rate function of a stationary nonergodic source has a decomposition as the average of the distortion-rate functions of the source's stationary ergodic components, where the average is taken over points on the component distortion-rate functions having the same slope. These results extend previously known results for finite alphabets

Multi-resolution source coding theorems

For stationary sources on Polish alphabets, we describe the family of achievable rate and distortion vectors (R1,…,RL) and (D1,…,DL) for an L-resolution source code, where the description at the first resolution is given at rate R1 and distortion D1, the description at the second resolution includes both the first description and a refining description of rate R2 and distortion D2, and so on. We consider performance bounds for fixed- and variable-rate source codes on discrete-time stationary ergodic and stationary nonergodic sources for any integer number of resolutions L ≥ 1. For L > 1, the results extend previous results for discrete memoryless sources

Practical multi-resolution source coding: TSVQ revisited

Consider a multi-resolution source code for describing a stationary source at L resolutions. The description at the first resolution is given at rate R1 and achieves an expected distortion no greater than D1. The description at the second resolution includes both the first description and a refining description of rate R2 and achieves expected distortion no greater than D2, and so on. Previously derived multi-resolution source coding bounds describe the family of achievable rate and distortion vectors ((R1, R2, ..., RL ), (D1, D2, DL)). By examining these multi-resolution rate-distortion bounds, we gain insight into the problem of practical multi-resolution source coding. These insights lead to a new multi-resolution source code based on the tree-structured vector quantizer. This paper covers the algorithm, its optimal design, and preliminary experimental results

The ergodic decomposition of asymptotically mean stationary random sources

It is demonstrated how to represent asymptotically mean stationary (AMS) random sources with values in standard spaces as mixtures of ergodic AMS sources. This an extension of the well known decomposition of stationary sources which has facilitated the generalization of prominent source coding theorems to arbitrary, not necessarily ergodic, stationary sources. Asymptotic mean stationarity generalizes the definition of stationarity and covers a much larger variety of real-world examples of random sources of practical interest. It is sketched how to obtain source coding and related theorems for arbitrary, not necessarily ergodic, AMS sources, based on the presented ergodic decomposition.Comment: Submitted to IEEE Transactions on Information Theory, Apr. 200

A vector quantization approach to universal noiseless coding and quantization

A two-stage code is a block code in which each block of data is coded in two stages: the first stage codes the identity of a block code among a collection of codes, and the second stage codes the data using the identified code. The collection of codes may be noiseless codes, fixed-rate quantizers, or variable-rate quantizers. We take a vector quantization approach to two-stage coding, in which the first stage code can be regarded as a vector quantizer that “quantizes” the input data of length n to one of a fixed collection of block codes. We apply the generalized Lloyd algorithm to the first-stage quantizer, using induced measures of rate and distortion, to design locally optimal two-stage codes. On a source of medical images, two-stage variable-rate vector quantizers designed in this way outperform standard (one-stage) fixed-rate vector quantizers by over 9 dB. The tail of the operational distortion-rate function of the first-stage quantizer determines the optimal rate of convergence of the redundancy of a universal sequence of two-stage codes. We show that there exist two-stage universal noiseless codes, fixed-rate quantizers, and variable-rate quantizers whose per-letter rate and distortion redundancies converge to zero as (k/2)n -1 log n, when the universe of sources has finite dimension k. This extends the achievability part of Rissanen's theorem from universal noiseless codes to universal quantizers. Further, we show that the redundancies converge as O(n-1) when the universe of sources is countable, and as O(n-1+ϵ) when the universe of sources is infinite-dimensional, under appropriate conditions

Joint Universal Lossy Coding and Identification of Stationary Mixing Sources With General Alphabets

Abstract-In this paper, we consider the problem of joint universal variable-rate lossy coding and identification for parametric classes of stationary -mixing sources with general (Polish) alphabets. Compression performance is measured in terms of Lagrangians, while identification performance is measured by the variational distance between the true source and the estimated source. Provided that the sources are mixing at a sufficiently fast rate and satisfy certain smoothness and Vapnik-Chervonenkis (VC) learnability conditions, it is shown that, for bounded metric distortions, there exist universal schemes for joint lossy compression and identification whose Lagrangian redundancies converge to zero as V n log n=n as the block length n tends to infinity, where V n is the VC dimension of a certain class of decision regions defined by the n-dimensional marginal distributions of the sources; furthermore, for each n, the decoder can identify n-dimensional marginal of the active source up to a ball of radius O( V n log n=n) in variational distance, eventually with probability one. The results are supplemented by several examples of parametric sources satisfying the regularity conditions

Metric mean dimension and analog compression

Wu and Verd\'u developed a theory of almost lossless analog compression, where one imposes various regularity conditions on the compressor and the decompressor with the input signal being modelled by a (typically infinite-entropy) stationary stochastic process. In this work we consider all stationary stochastic processes with trajectories in a prescribed set of (bi-)infinite sequences and find uniform lower and upper bounds for certain compression rates in terms of metric mean dimension and mean box dimension. An essential tool is the recent Lindenstrauss-Tsukamoto variational principle expressing metric mean dimension in terms of rate-distortion functions. We obtain also lower bounds on compression rates for a fixed stationary process in terms of the rate-distortion dimension rates and study several examples.Comment: v3: Accepted for publication in IEEE Transactions on Information Theory. Additional examples were added. Material have been reorganized (with some parts removed). Minor mistakes were correcte