A new algorithm for variable dimension weighted universal coding is introduced. Combining the multi-codebook system of weighted universal vector quantization (WUVQ), the partitioning technique of variable dimension vector quantization, and the optimal design strategy common to both, variable dimension WUVQ allows mixture sources to be effectively carved into their component subsources, each of which can then be encoded with the codebook best matched to that source. Application of variable dimension WUVQ to a sequence of medical images provides up to 4.8 dB improvement in signal to quantization noise ratio over WUVQ and up to 11 dB improvement over a standard full-search vector quantizer followed by an entropy code. The optimal partitioning technique can likewise be applied with a collection of noiseless codes, as found in weighted universal noiseless coding (WUNC). The resulting algorithm for variable dimension WUNC is also described

Chou, P. A.

Effros, M.

Gray, R. M.

English

Caltech Authors - Main

Variable Dimension Weighted Universal 
Vector Quantization and Noiseless 
Coding * 
M. Effros,* P. A. Chou,+ and R. M. Gray: 
* Information Systems Labomto y, Stanford, CA 94305-4055, 
phone: (415) 723-2675, fax: (415) 723-8473, email: effros@isl.stanford.edu 
t Xeroz Palo Alto Research Center, 3333 Coyote Road, Palo Alto, CA 94304 
phone: (415) 812-4496, faz: (415) 812-4241, email: chou@parc.zeroz.com 
phone: (415) 723-4001, fax: (415) 723-8473, emad: gmy@isl.stanford.edu 
Infomation Systems Labomto y, Stanford, CA 94905-4055, 
Abstract 
A new algorithm for variable dimension weighted universal coding is in- 
troduced. Combining the multi-codebook system of weighted universal vector 
quantization (WUVQ), the partitioning technique of variable dimension vec- 
tor quantization, and the optimal design strategy common to both, variable 
dimension WUVQ allows mixture sources to be effectively carved into their 
component subsources, each of which can then be encoded with the codebook 
best matched to that source. Application of variable dimension WUVQ to a 
sequence of medical images provides up to 4.8 dB improvement in signal to 
quantization noise ratio over WUVQ and up to 11 dB improvement over a 
standard full-search vector quantizer followed by an entropy code. The opti- 
mal partitioning technique can likewise be applied with a collection of noiseless 
codes, as found in weighted universal noiseless coding (WUNC). The resulting 
algorithm for variable dimension WUNC is also described. 
1 Introduction 
Weighted universal vector quantizers and weighted universal noiseless codes (1, 2, 31 
are tivestage codes. In each, an incoming data string is broken into supervectors, 
which are blocks of vectors. All of the vectors in a particular supervector are encoded 
with a single code. Given some collection o l  codes, the encoder chooses the code that 
'This material is based upon work partially supported by an ATLT Bell Laboratories Ph.D. 
Scholarship and by a grant from the Center for Telecommunications at Stanford. The authors 
would also like to thank Professor Eve Riskin Cor the use of an HP 720 donated by Hewlett Packard 
Laboratories. 
2 1068-0314494 $3.00 Q 1994 IEEE 
3 
most closely matches the statistics of the vectors in the supervector at hand. The 
first stage of a two-stage code describes the chosen quantizer or noiseless code, and 
the second stage describes the data using that code. By keeping a variety of codes 
and allowing the code in use to change from supervector to supervector, WUVQs 
and WUNCs can roughly track local variation in source statistics. The expense of 
tracking this information is the side information necessary to describe the code used 
on each supervector. 
(VDWUVQ) and variable dimension WUNC (VDWUNC) employ the optimal parti- 
tioning algorithm of variable dimension VQ (VDVQ) [4, 51 to break the image into 
variable sized supervectors in an optimal fashion. The basic algorithm proceeds as fol- 
lows. An incoming data stream is partitioned into variable length supervectors. The 
encoder then describes these supervectors, one by one, to the decoder. To describe a 
supervector, the encoder first sends the length of the supervector, then describes the 
index of the code with which that supervector's component vectors will be encoded, 
and finally describes the component vectors using the chosen code. The optimal par- 
tition is the partition that minimizes the overall distortion subject to a constraint 
on the rate, which now includes the rate associated with describing the length of 
each supervector. The employment of optimally chosen supervectors of variable sizes 
allows the VDWUVQ to better carve the data into its component subsources, each 
of which is then encoded separately by an encoder matched to that source. 
Section 2 reviews the basic WUVQ and WUNC algorithms. The VDWUVQ and 
VDWUNC algorithms are then described in section 3. The final section presents re- 
sults achieved by applying the VDWUVQ algorithm to a set of medical images. While 
both fixed- and variable-rate versions of VDWUVQ and VDWUNC are possible, this 
paper focuses on the variable-rate case. 
While WUVQ and WUNC use fixed supervector lengths, variable dimension WUVQ 
2 The WUVQ and WUNC Algorithms 
Given a class, A, of process measures and a distribution over that class, a weighted 
universal code is a sequence of block codes for which the expected performance over 
all 0 E A approaches, with increasing block size, the theoretically optimal expected 
performance. To be more exact, let { X i }  be a random process with alphabet X 
and process measure PO, 0 E A,  and let p" o 9" be a length-n block code with 
encoder an : X" --+ S and decoder p" : S --+ X " ,  where S c {0,1}* is a binary 
prefix code, lan(rn)l is the length of the code for z", and k is the reproduction 
alphabet. In weighted universal quantization, k is not required to equal X ,  X may be 
uncountably infinite, and p"(an(zn)) = 2" with distortion dn(zn, 2") = ,Ci dl(zi, si), 
where, &(zj,&) is an additive fidelity criterion. In noiseless coding, X = X ,  X is 
finite or countably infinite, and ,P(a"(z")) = zn for all zn E X .  A weighted universal 
source code is a sequence of block codes {(a",P")} such that for each 0 E A there 
exists a corresponding sequence of points {(Rn,o, Dn,e)} on the graph of the nth order 
4 
I = I  I 
channel 
xi, ..., xfv 
I 
I decoder I 
L _ _ _ _ _ _ _  2 I encoder I L _ _ _ _ - - _ _ _ _ - >  
Figure 1: Weighted Universal VQ 
operational distortion-rate function for Po for which the per letter "rate" redundancy 
and the per-letter "distortion" redundancy 
each go to zero in expectation with respect to a probability measure (weighting) on 8. 
Notice that the distortion redundancy requirement is trivially satisfied for all noiseless 
codes. 
In [6], Neuhoff, Gray, and Davisson show that a weighted universal source code 
can be constructed from a sequence of multi-codebook vector quantizers; in [l], Chou 
develops an algorithm for the design of weighted universal vector quantizers, which 
are locally optimal multi-codebook vector quantizers for weighted universal source 
coding; and in [7], Chou and Effros provide bounds on the rates at which the expected 
rate and distortion redundancies go to zero for WUVQ. Two-stage universal noiseless 
codes were earlier developed in [SI and [9]. In this section we describe WUVQs and 
WUNCs and the algorithm by which they can be optimally designed. 
The WUVQ and WUNC algorithms [l, 2, 31 treat the optimal design of two- 
stage weighted universal vector quantizers and noiseless coders. In both cases, the 
first stage of the system is a "quantizer" which maps the input space of possible 
data vectors to the output space of possible block codes; the chosen second-stage 
block code then maps the data vector to the output space of possible reproductions. 
In universal variable-rate quantization the block codes are variable-rate quantizers, 
while in universal noiseless coding, these block codes are block noiseless codes. 
Figure 1 shows a two-stage coder. The vector sequence xi,. . . , xfv E IR' is parti- 
tioned evenly into supervectors yl,. . . , y~ E IR'", where each supervector contains n 
1-vectors. For simplicity it is assumed that n evenly divides N, so that K = N/n is 
an integer. The first-stage quantizer contains an encoder ti : X'" -+ s which maps 
the input space of possible supervectors to the output space of code descriptors. The 
first-stage description of the supervector y;, 6 = &(y;) E S ,  indexes the chosen length- 
1 block code used to code the /-vectors of y;. We use 1.51 to denote the first-stage 
description length. The second-stage block code contains an encoder and a decoder. 
Let {ai : 6 E s} and {pi : S E s} be the collections of block encoders and decoders 
5 
respectively. For any 5 E S ,  a: : X‘  + Ss and : Sj -+ kl. Denote the component 
vectors of supervector y; by Y; ,~ ,  . .. , y;+. If s” is the first-stage description of y;, then 
ss = a&j) E Si is the second-stage description of component vector y;,j, 1sil is the 
length of the second-stage description of yi,j, and dl(y;,j, P i ( a i ( y i , j ) ) )  is the distortion 
incurred by using the block code indexed by 5 on y;,j. The distortion is always zero in 
noiseless coding. The total distortion and rate associated with encoding supervector 
y; (a block of n I-vectors) with codebook B are 
and 
For two-stage noiseless codes and two-stage variable-rate quantizers, S and Si are 
Huffman codes matched to the probabilities of the block codes and codewords, re- 
spectively. 
Two-stage source codes are designed using an extension of the generalized Lloyd 
algorithm to minimize 
K 
Jx(y;, s ” ) l ( q Y i ) = s ” )  
i=l jES 
on a training sequence, where 1(A) is 1 if statement A is true and 0 otherwise, the 
Lagrangian functional Jx is defined as 
and X is the Lagrange multiplier. In noiseless coding, where the distortion is always 
equal to zero, X is chosen to equal one, so that Jx equals the total rate. In variable- 
rate vector quantization, nonnegative X values are chosen based on the desired final 
rate of the system. The optimization process consists of a nested descent algorithm in 
which the Lloyd algorithm’s nearest neighbor and centroid conditions are employed on 
both stages of the system. In the first-stage quantizer, satisfying the nearest neighbor 
condition means mapping each supervector to the codebook that will best encode that 
supervector’s component vectors. In the second stage, satisfying the nearest neighbor 
condition means mapping each vector to the best codeword. In both cases, “best” 
is measured by the functional Jx. By satisfying the centroid condition, we optimize 
the decoder for a given encoder. The first-stage quantizer meets this condition by 
redesigning each codebook in its collection of codebooks to match the statistics of 
the data mapped to that codebook. Each second-stage decoder likewise meets this 
condition by redesigning each of its codewords to match the data mapped to it. 
3 The VDWUVQ and VDWUNC Algorithms 
VDWUVQ and VDWUNC are two-stage coding systems where the length of the 
supervectors is allowed to vary. An incoming data sequence, xi,. . . , zfv, is broken 
6 
into supervectors of varying length, say yl,. . . , y ~ ,  of lengths Iyll , .  . . , l y ~ l  vectors. 
The number of supervectors and their lengths are subject only to the constraint that 
the sum of the lengths must equal the initial data length (xi lyil = N ) .  Using xiN 
to denote the sequence zi, . . . xh, let {y;} 4 zi" mean that {y;} partitions xi" and 
satisfies the above constraint. Since the supervector length is allowed to vary, it must 
be described to the decoder along with the first- and second-stage code information. 
The encoder thus uses an entropy code y to describe that length to the decoder, 
where Iy(n)l is the rate associated with describing a supervector length of n. Notice 
that if Iy(.)l is zero for length n and infinity otherwise, VDWUVQ and VDWUNC 
will behave exactly like WUVQ and WUNC. Thus VDWUVQ and VDWUNC can 
only exceed their predecessors in performance, but they do so at the expense of the 
additional complexity necessary for obtaining the optimal partition. 
The optimal design algorithm for VDWUVQs and VDWUNCs is again a variation 
on the generalized Lloyd algorithm, which iteratively achieves descent on the weighted 
sum of the total distortion and rate 
K 
J x ( Y i , q l ( ~ ( Y i ) = s " ) ,  
i=l ges 
where the Lagrangian is now 
Let I{y;}l denote the number of elements in the partition, then the training algorithm 
proceeds as follows: 
1. Optimally parse the data for fixed y, tu, 3, and {(a;,@;,&)} and then update 
y to match the new partition. The optimal partition satisfies 
Thus update y to satisfy Iy(n)l = [-log, [I, l(Iy;l=n)/ ({y;}(]l. 
2. Optimize the first-stage encoder Cy for fixed {y;}, y, S ,  and {(as,&&)} and 
then redesign the prefix code. The resulting encoder is 
tu(yi)  = argmin[d(y;,S) + X ~ ( y i , g ) ] ,  
IES 
and for each B E S its new prefix code satisfies t i l  = [-log, [ x i  l(Cy(y;)=i)/I{y;}l]l 
3. Optimize each block code for fixed {y;}, y, Cy, and S .  This is accomplished for 
each B E S by iteratively optimizing a;,@;, and S,- until convergence, where 
the optimal encoder, decoder, and prefix code respectively satisfy the following 
three conditions. (Note that the change to the decoder has no effect in the case 
of noiseless coding.) 
4 y i j )  = a r g ~ ~ [ d l ( y ; , j , @ s ( s ; ) )  + XlsaI] Vyi,j s.t. &(Y;) = 2. 
7 
4. Iterate steps 1-3 until convergence. 
The optimal partition in step 1 of the above described algorithm is accomplished 
using the dynamic programming argument of VDVQ [ 5 ] .  The argument proceeds as 
follows. Let J,, = miniY,}+ x i  Jx(y,, ii(yi)) be the score of the best partitioning 
of the partial data sequence z:, . . . ,zk. If B is the maximum allowed supervector 
length, then for 1 5 n 5 N ,  
where JO = 0 and J,, = 00 for n < 0. Now let b, be the last su ervector in the optimal 
partition of z;, that is the supervector of length a rgminlsbs~  [J,,-h + minjEJ JA(zE...b+1, i)} 
ending with the nth vector. Then the optimal partition of z:” can be found by back- 
tracking: 
{Yi> = {. . . b N - ( b , v - l b N - l b N i I ,  b N - l b ~ l l  b N ) .  
While partitioning and encoding have been described as separate processes, they can 
be accomplished simultaneously (at the expense of greater storage requirements) by 
tracking not only the optimal performance and last supervector length at each time 
n but also the encoding information for that supervector. 
4 Experimental Results 
We here compare the performance of VDWUVQ with that of WUVQ and standard 
full-search VQ. The data sequence considered contains 25 magnetic resonance (MR) 
brain scans at an original 9 bpp. The first 20 images are used for training and the 
final 5 are used for testing. All of the vector quantizers use vectors of dimension 2 x 2 
and the vectors are ordered according to a Peano scan (e.g., [lo]). The supervectors 
in VDWUVQ are allowed to contain up to 16 vectors while the WUVQ has a constant 
supervector size of 4. All multi-codebook systems consisted of at most 256 codebooks, 
each with no more than 4 codewords. 
Figure 2 shows the performance of variable-rate VDWUVQ and WUVQ with 
varying values of X and the performance of standard full-search VQ. All rates are 
reported in terms of entropy. Signal to quantization noise ratio (SQNR) is calculated 
as -lOlog(D/DO), where D is the current distortion, Do is the zero rate distortion of 
a standard full-search VQ, and all distortion is measured by the squared error fidelity 
criterion. VDWUVQ shows its greatest improvement over WUVQ at very low rates, 
below .1 bpp, with an SQNR up to 4.8 dB higher. Both VDWUVQ and WUVQ 
achieve their greatest gains over standard full-search VQ at slightly higher rates. 
8 
rate 
Figure 2: VDWUVQ, WUVQ, and standard full-search VQ on medical test sequence. 
... variable-rate VDWUVQ with maximum supervector size 256, 
- variable-rate VDWUVQ with maximum supervector size 16, 
- - variable-rate WUVQ, -.- standard full-search VQ followed by an entropy code 
VDWUVQ shows up to 11 dB improvement over standard VQ. Allowing supervector 
lengths up to 256 vectors improves performance by about another dB at  the expense 
of increased computation. 
Figure 3 compares the images resulting from quantizing the data at around .2 
bpp, while figure 4 demonstrates the sizes and shapes of the supervectors used by the 
VDWUVQ algorithm in the previous figure. 
References 
(11 P. A. Chou. Code clustering for weighted universal VQ and other applications. 
In Proceedings of the IEEE International Symposium on Information Theory, 
page 253, Budapest, Hungary, June 1991. 
[2] B. D. Andrews, M. Effros, P. A. Chou, and R. M. Gray. A mean-removed varia- 
tion of weighted universal vector quantization for image coding. In Proceedings 
of the Data Compression Conference, pages 302-309, Snow Bird, UT, April 1993. 
IEEE Computer Society. 
[3] P.A. Chou, M. Effros, and R.M. Gray. A vector quantization approach to uni- 
IEEE Transactions on Information versal noiseless coding and quantization. 
Theory. In preparation. 
[4] P. A. Chou and T. Lookabaugh. Locally optimal variable-to-variable length 
source source coding with respect to a fidelity criterion. In Proceedings of the 
IEEE International Symposium on Information Theory, page 238, Budapest, 
Hungary, June 1991. 
9 
Figure 3: From top left to bottom right: The original 9 bpp image, standard VQ 
at .25 bpp and SQNR = 6.75 dB, WUVQ at  ,184 bpp and SQNR = 11.6 dB, and 
VDWUVQ at  ,166 bpp and SQNR = 13.2 dB. 
10 
Figure 4: Supervector sizes and shapes for the .166 bpp image. The picture on the 
left shows the size of the supervector in which each pixel was coded, with supervector 
sizes from 1 to 16 vectors denoted in shades from black to white. The picture on the 
right gives a random shade to each supervector in order to demonstrate supervector 
shapes. 
11 
(51 P.A. Chou and T. Lookabaugh. Variable dimension vector quantization of linear 
predictive coefficients of speech. In Proceedings of the IEEE International Con- 
ference on Acoustics, Speech, and Signal Processing, Adelaide, Australia, April 
1994. 
[6] D. L. Neuhoff, R. M. Gray, and L. D. Davisson. Fixed rate universal block source 
IEEE Transactions on Information Theory, coding with a fidelity criterion. 
21(5):511-523, September 1975. 
[7] P. A. Chou and M. Effros. Rate and distortion redundancies for source cod- 
ing with respect to a fidelity criterion. In IEEE International Symposium on 
Information Theory, page 53, San Antonio, Texas, January 1993. 
[8] R. F. Rice and J.  R. Plaunt. Adaptive variable-length coding for efficient com- 
pression of spacecraft television data. IEEE Transactions on Communications, 
19(12):889-897, December 1971. 
[9] L. D. Davisson. Universal noiseless coding. IEEE Transactions on Information 
Theory, 19(6):783-795, November 1973. 
[lo] R. J. Stevens, A. F. Lehar, and F. H. Preston. Manipulation and presenta- 
tion of multidimensional image data using the peano scan. IEEE Transactions 
on Pattern Analysis and Machine Intelligence, PAMI-5(5):520-533, September 
1983. 


Variable dimension weighted universal vector quantization and noiseless coding

A mean-removed variation of weighted universal vector quantization for image coding.

A vector quantization approach to uniIEEE Transactions on Information versal noiseless coding and quantization. Theory. In preparation.

Code clustering for weighted universal VQ and other applications.

Locally optimal variable-to-variable length source source coding with respect to a fidelity criterion.

http://authors.library.caltech.edu/7305/1/EFFdcc94.pdf

Variable dimension weighted universal vector quantization and noiseless coding

Abstract

Similar works

Full text

Available Versions

Caltech Authors - Main