For a stationary ergodic source, the source coding theorem and its converse imply that the optimal performance theoretically achievable by a fixed-rate or variable-rate block quantizer is equal to the distortion-rate function, which is defined as the infimum of an expected distortion subject to a mutual information constraint. For a stationary nonergodic source, however, the. Distortion-rate function cannot in general be achieved arbitrarily closely by a fixed-rate block code. We show, though, that for any stationary nonergodic source with a Polish alphabet, the distortion-rate function can be achieved arbitrarily closely by a variable-rate block code. We also show that the distortion-rate function of a stationary nonergodic source has a decomposition as the average of the distortion-rate functions of the source's stationary ergodic components, where the average is taken over points on the component distortion-rate functions having the same slope. These results extend previously known results for finite alphabets

Chou, P. A.

Effros, M.

Gray, R. M.

English

The source coding theorem and its converse imply that the optimal performance theoretically achievable by a fixed- or variable-rate block quantizer on a stationary ergodic source equals the distortion-rate function. While a fixed-rate block code cannot achieve arbitrarily closely the distortion-rate function on an arbitrary stationary nonergodic source, the authors show for the case of Polish alphabets that a variable-rate block code can. They also show that the distortion-rate function of a stationary nonergodic source has a decomposition as the average over points of equal slope on the distortion-rate functions of the source's stationary ergodic components. These results extend earlier finite alphabet results

Caltech Authors - Main

Variable-Rate Source Coding Theorems for Stationary Nonergodic 
Sources 
M .  Effros,t P.A. Chou: and R. M.  Gray t1  
'Information Systems Laboratory, Stanford, CA 94305-4055 USA 
!Xerox Palo Alto Research Center, 3333 Coyote Road, Palo Alto, CA 94304 USA 
Abstract ~ The source coding theorem and its con- 
verse imply that the optimal performance theoret- 
ically achievable by a Axed- or variable-rate block 
quantizer on a stationary ergodic source equals the 
distortion-rate function. While a Axed-rate block 
code cannot achieve arbitrarily closely the distortion- 
rate function on an arbitrary stationary nonergodic 
source, we show for the case of Polish alphabets that 
a variable-rate block code can. We also show that 
the distortion-rate function of a stationary nonergodic 
source has a decomposition as the average over points 
of equal slope on the distortion-rate functions of the 
source's stationary ergodic components. These results 
extend earlier Anite alphabet results. 
I .  INTRODUCTION 
In [l], Shields et al. show that for any stationary nonergodic fi- 
nite alphabet source, the distortion-rate function D(R) equals 
the infimum of the average of the distortion-rate functions of 
the source's stationary ergodic components, where the average 
is taken over points on the component distortion-rate func- 
tions whose rates average to at most R. The achievability of 
this bound by variable-rate block codes is shown in [Z]. 
We extend these variable rate quantization results from fi- 
nite alphabets to complete separable metric spaces, or Polish 
alphabets. We employ a simplified variable-rate and variable- 
distortion using a Lagrangian formulation. 
11. RESULTS 
Let ( A m , B m , p , T )  be a stationary dynamical system with 
Polish alphabet A. That is, let A be a complete separable 
metric space, let B be the Bore1 o-algebra generated by the 
open sets of A, let A" be the set of one-sided sequences z = 
(21, zz, . . .) from A, let B" be the o-algebra of subsets of A" 
generated by finite-dimensional rectangles with components 
in 8, let T be the left shift operator on A", and let p be a 
measure on the measurable space (Am,Bm), stationary with 
respect to T. 
Now let p ( z l ,  yl) < 00 be a fed-vdu+ nonnegative distor- 
tion measure for 21 E A, y~ E A, where A is an abstract repro- 
duction alphabet. Assume that p ( z ~ , y ~ )  is continuous in z1 
for each y1 E A and that there exists a reference letter y; such 
that E,p(Xi, Y;) < 00. Define p ( z N ,  a r N )  = Cf, p?,,y,). 
Finally let Q be a variable-rate block quantizer with block- 
length N .  That  is, let Q be  a map from A N  onto some finite 
or countable set of codewords {yN} C AN composing a code- 
book C = {(yN,IyNI)) in which each codeword yN has an 
associated variable-length binary description, with length de- 
noted l ~ ~ [ .  The description lengths must satisfy the Kraft 
inequality xyNEC 2 4 ' 1  I 1.
'This material is based upon work partially supported by an 
AT&T Ph.D. Scholarship, by a grant from the Center for Telecom- 
munications at Stanford, and by an NSF Graduate Fellowship. 
The optimal performance theoretically achievable by any 
variable-rate block quantizer is the operational distortion-rate 
function 6"(R,p) = infN6Lr(R,p), where 6gr(R,p)  is the 
N t h  order operational distortion-rate function 
6Lr(R,p) = i;f { ; E " p ( X N ,  Q ( X N ) )  ; iEpIQ(xN)I IE ] .  
Here, the infimum IS taken over all variable-rate block quantiz- 
ers Q with blocklength N .  We contrast this with the optimal 
performance theoretically achievable by fixed-rate block quan- 
tizers, 6fr(R, P )  = infN &(R, p ) ,  in which &(R, p )  is defined 
as 6Lr(R,p) but with the infimum taken over all fixed-rate 
block quantizers with blocklength N .  
The Shannon distortion-rate function is defined similarly, 
as D(R,p)  = infNDN(R, p ) ,  where D N ( R , ~ )  is the Nth  or- 
der distortion-rate function 
DN(R,  p )  = i;f { ?E,,p(XN, 1 Y N )  : k I p U ( X N ;  Y N )  I R }  . 
Here, U is a conditional probability or test channel from A N  
to  AN defining, with p ,  a joint probability or hookup pu on 
X N  and Y N ,  and I is the mutual information. 
I t  is well-known that both D(R, p )  and 6fr(R, f i )  are convex 
in R [3]; SVr(R,p) is likewise convex in R, by a timesharing 
argument. Hence 6"(R,p) and D ( R , p )  can be characterized 
by their support function& [4, p. 1351 the weighted opera- 
tional distortion-rate function !(A, p )  = infR [6"(R, p )  + XR] 
and the weighted Shannon distortion-rate function L(X, p )  = 
infR[D(R, P )  + XR] . 
The source coding theorem and its converse imply that 
when p is ergodic, 6"'(R,p) = &(R,p)  = D ( R , p )  for all 
R 2 0 (and hence P ( X , p )  = L ( X , p )  for all X 2 0). When 
p is nonergodic, let { p =  : z E A") denote the ergodic de- 
composition of p.  The ergodic decomposition exists since A 
Polish implies ( A , B )  standard [5, Theorem 3.3.11, and hence 
(Am, 8") is standard [5, Lemma 2.4.11 which gives the de- 
sired property by [5, Theorem 7.4.11. The main results of this 
paper are that under the conditions given above 
Theorem 1 ((A, p )  = 
Theorem 2 L ( X , p )  = s L ( X , p = ) d p ( z )  VX > 0, 
Theorem 3 C ( X , p )  = L(X, p )  VX 2 0, and hence hVr(R,p) = 
D(R, p )  VR 2 0. 
!(A, p.)dp(z)  VX 2 0, 
REFERENCES 
[l] P. C. Shields, D. L. Neuhoff, L. D. Davisson, and F. Ledrap 
pier, "The distortion-rate function for nonergodic souTces,I1 The 
Annals of Probability, 6(1):138-143, 1978. 
(21 A. Leon-Garcia, L. D. Davisson, and D. L. Neuhoff, "New re- 
sults on coding of stationary nonergodic sources," IEEE Trans- 
actions on Informafion Theory, 25(2):137-144, March 1979. 
[3] R. M. Gray, Entropy and Information Theory, Springer-Verlag, 
New York, 1990. 
[4] D. G. Luenberger, Optimization by Vector Space Methods, John 
Wiley and Sons, New York, 1969. 
[5] R. M. Gray, Probability, Random Processes, and Ergodic Prop- 
erties, Springer-Verlag, New York, 1988. 
0 - 7803-2015-8/94~S4.00 (01994 IEEE - 266 - 


Variable-rate source coding theorems for stationary nonergodic sources

D. G. Luenberger, Optimization by Vector Space Methods. New York: Wiley, 1969.

Englewood Cliffs, NJ: Prentice-Hall, 1971.

Entropy and Information Theory.

Minimax learning in iterated games via distributional majorization,” Ph.D. Dissertation,

New York: Wiley, 1968. T. Berger, Rate Distortion Theory: A Mathematical Basis for Data Compression. J. C. Kieffer, “On the optimum average distortion attainable by a fixed-rate coding of a nonergodic source,” IEEE Trans. Znform.

Optimization by Vector Space Methods.

Probability, Random Processes, and Ergodic Properties.

R. M. Gray and L. D., Davisson, “Source coding theorems without the ergodic assumption,” ZEEE Trans. Znform. Theory, vol. IT-20, pp. 502-526, July 1974.

R. M. Gray, Entropy and Information Theory. New York: Springer-Verlag, 1990. R. G. Gallager, Information Theory and Reliable Communication.

R. M. Gray, Probability, Random Processes, and Ergodic Properties. New York Springer-Verlag, 1988.

Rate Distortion Theory: A Mathematical Basis for Data Compression.

Source coding theorems without the ergodic assumption,”

T. H. Chung, “Minimax learning in iterated games via distributional majorization,” Ph.D. Dissertation, Stanford University, 1994.

The distortion-rate function for nonergodic sources,”

Theory, vol. IT-21, pp. 190-193, Mar. 1975.

Theory, vol. IT-25 no. 2, pp 137-144, Mar. 1979.

https://authors.library.caltech.edu/7341/1/EFFisit94b.pdf

Variable-rate source coding theorems for stationary nonergodic sources

Abstract

Similar works

Full text

Available Versions

Caltech Authors - Main