Weighted universal vector quantization uses traditional codeword design techniques to design locally optimal multi-codebook systems. Application of this technique to a sequence of medical images produces a 10.3 dB improvement over standard full search vector quantization followed by entropy coding at the cost of increased complexity. In this proposed variation each codebook in the system is given a mean or 'prediction' value which is subtracted from all supervectors that map to the given codebook. The chosen codebook's codewords are then used to encode the resulting residuals. Application of the mean-removed system to the medical data set achieves up to 0.5 dB improvement at no rate expense

Andrews, Barry D.

Chou, Philip A.

Effros, Michelle

Gray, Robert M.

English

Caltech Authors - Main

A Mean-Removed Variation of 
Weighted Universal Vector Quantization 
for Image Coding * 
Barry D. Andrews 
Information Systems Laboratory 
Stanford University 
Stanford, CA 94305-4055 
email: andrews@isl.stanford.edu 
Philip A. Chou 
Xerox Palo Alto Research Center 
3333 Coyote Hill Road 
Palo Alto, CA 94304 
email: chou@parc.xerox.com 
Michelle Effros 
Information Systems Laboratory 
Stanford University 
Stanford, CA 94305-4055 
email: effros@isl.stanford.edu 
Robert M. Gray 
Information Systems Laboratory 
Stanford University 
Stanford, CA 94305-4055 
email: gray@isl.stanford.edu 
Abstract - Approaching a multi-codebook system as 
a codebook of codebooks, weighted universal vector quantiza- 
tion (WUVQ) uses traditional codeword design techniques to 
design locally optimal multi-codebook systems. Application 
of this technique to a sequence of medical images produces a 
10.3 dB improvement over standard full search vector quan- 
tization followed by entropy coding at the cost of increased 
complexity. In this paper we propose a mean-removed vari- 
ation of WUVQ. Each codebook in the system is given a 
“mean” or “prediction” value which is subtracted from all 
supervectors that map to the given codebook. The chosen 
codebook’s codewords are then used to encode the result- 
ing residuals. Application of the mean-removed system to 
the medical data set achieves up to .5 dB improvement over 
WUVQ at no rate expense. 
‘This material is based upon work supported under a Natural Sciences and Engineering Research 
Council of Canada Scholarship and a Sony Corporation Fellowship, a National Science Foundation 
Graduate Fellowship, and National Science Foundation Grant MIP-9016974. 
302 0-8186-3392-1193 $3.00 8 1993 IIjIiE 
303 
1 Introduction 
In multi-codebook vector quantization systems, blocks or vectors of incoming 
data samples are characterized and sent to one of some number of codebooks. One 
example of a multi-codebook vector quantization system is a finite-state vector quan- 
tizer, in which the state, or the index of the current codebook, is determined from 
indices of previously quantized vectors. Thus the state can be tracked by the decoder 
without side information. Another example of a multi-codebook vector quantization 
system is classified vector quantization, in which the “state” or index of the current 
codebook is determined from characteristics of the current vector and its context. 
In this case the state must be transmitted as side information. Weighted univer- 
sal vector quantization (WUVQ) is a multi-codebook vector quantization system in 
which the “state” or codebook index is periodically transmitted as side information, 
and held constant over the period. Thus all vectors within the supervector consti- 
tuting the period are quantized with the same codebook. Moreover, the codebook 
is selected to reproduce the supervector with the lowest distortion. This increases 
the delay associated with encoding any single vector (the entire supervector must be 
encoded with each codebook before a codebook can be chosen), but decreases the 
amount of side information that needs to be sent. WUVQ can be regarded as a two 
stage system: the encoder sends the state or index of the codebook with the lowest 
distortion on the supervector, and then encodes each vector in the supervector with 
the given codebook. 
In the proposed mean-removed variation of WUVQ, we attempt to exploit prop- 
erties common to those supervectors that map to a single codebook. We therefore 
include a “mean” or “prediction” supervector for each codebook, and subtract that 
value from each incoming data supervector. The residual supervector is then broken 
into its corresponding vectors and encoded using the given codebook. The inclusion 
of a single “prediction” for each codebook costs nothing in terms of rate and the 
mean-removed variation includes as a special case (with zero-valued “predictions”) 
the WUVQ system. 
2 Weighted Universal Vector Quantization 
Given a class A of source distributions, a weighted universal quantizer is 
a series of block codes whose expected performance on any single source 8 E A 
approaches, as the block size increases to infinity, the asymptotic performance of a 
system designed explicitly for &distributed processes. To be more exact, let { X i }  
be a random process over alphabet X with process measure Po, 8 E A, and let 
,BN o aN be a length-N block code with encoder aN : XN -+ C and decoder ,BN : 
C 3 J”, where C = {q,.  . , CM} {0,1}* is some binary prefix code and y is the 
reproduction alphabet, typically equal to X. A weighted universal source code with 
respect to a fidelity criterion d ( X N ,  Y ” )  = Ci d ( X ; ,  y t )  is a sequence of block codes 
{ (aN,  p”)} such that for each 8 E A there exists a corresponding sequence of points 
{ ( R N , @ ,  D N , o ) }  on the graph of the Nth  order cperational distortion-rate function for 
304 
r - - - - - - - - - - 1  r - - - - - - - 1  
channel 
I decoder I 
L - - _ - - - -  J I encoder I L - - - - - - _ - - - d  
Figure 1: Weighted Universal VQ 
Po for which the per letter "rate" redundancy 
1 
- -Eo(aN(XN)(  - R N , ~  N 
and the per-letter "distortion" redundancy 
LEod(XN, N p " ( a N ( X N ) ) )  - DN,B (2) 
each go to zero in expectation with respect to a probability measure (weighting) on 
0. Here laN(XN)I is used to denote the length of the binary code a N ( X N ) .  
In [I], Neuhoff, Gray, and Davisson show that a weighted universal source 
code can be constructed from a sequence of multi-codebook vector quantizers; in [2], 
Chou develops an algorithm for the design of weighted universal vector quantizers, 
which are locally optimal multi-codebook vector quantizers for weighted universal 
source coding; and in [3], Chou and Effros provide bounds on the rates at  which the 
expected rate and distortion redundancies (1) and (2) go to zero for WUVQ. In this 
section we describe WUVQ and its design algorithm. 
Consider the set of codebooks {pS 1 s E S} as a codebook of codebooks, where 
S = {SI,. . . , s ~ }  C (0, l}' is a binary prefix code. In the WUVQ system, shown in 
Figure 1, the encoder breaks the source sequence into supervectors X k n ,  consisting 
of n vectors X,", . . . , X,", each of dimension k. The encoder a maps each X k n  to a 
codebook Ps, and sends the binary code s to the decoder. Each k-dimensional vector 
in the supervector is then encoded with the encoder a, associated with the chosen 
codebook ,Bs. The decoder receives s and then decodes each k-dimensional vector 
using ,Bs. 
Given a codebook /3, the distortion associated with coding the supervector 
X k n  is 
and the corresponding 
The function a maps the supervector X k n  to a codebook index s E S, which is sent 
to the decoder along with the encoded data. The per symbol average distortion in 
305 
this system is 
1 
D k n  = GE[d(xk", p a ( X k n ) ) ] ,  
and the per symbol average rate is 
1 
R k n  nE[b(xk")l + r ( X k n ,  p4xkn))1, 
i.e., the length of the codebook description plus the length of the data description 
given the codebook. 
Locally optimal design for both fixed and variable rate WUVQ systems is 
achieved using the generalized Lloyd algorithm. For a fixed rate WUVQ, with fixed 
length s and ps, the optimal encoder satisfies the nearest neighbor condition 
a(Xkn) = arg ?in d(Xkn,  ps) 
and the optimal decoder satisfies the centroid condition 
os = argminE[d(Xk",P)la(Xk") = SI. 
P 
The generalized Lloyd algorithm iteratively optimizes the encoder for a given decoder 
and then the decoder for the resulting encoder until the system converges to an 
optimal solution. 
For a variable rate (entropy-constrained) WUVQ, the optimal encoder satisfies 
a ( x k n )  = argmjn [d(xkn, ,A) + X[ISI + r(xkn, A)]] 
and the optimal decoder satisfies 
pS = arg min E [ d(xk", p)  + X [ I S ~ +  r (xkn . ,  p)ll a(Xkn> = s] , 
where X is a Lagrangian constant chosen as a function of the desired final rate. Here 
too, we employ the generalized Lloyd algorithm to iteratively optimize the encoder 
and decoder in a manner parallel to that of [4]. 
P 
3 A Mean-Removed Variation of WUVQ 
In the mean-removed variation of WUVQ, we alter the WUVQ structure (Fig- 
ure 1) by associating with each codebook pS, a kn-dimensional supervector predic- 
tion P,"". The encoder a maps each supervector X k n  to a codebook/prediction pair 
(ps,P,k"). Given an index s, we encode a supervector Xk" by first subtracting P,"" 
and then encoding the component vectors of the residual supervector ( X k n  - P p )  
using ps. Since only one supervector prediction is included with each codebook, the 
addition of predictions to the system requires no additional side information to be 
sent from encoder to decoder. 
306 
We now assume that d is the squared-error fidelity criterion. The distortion 
and rate associated with coding the supervector X k n  with the codebook/prediction 
pair (@,,P,"") are 
n 
d(Xk",(PS,P,"")) = C d ( X , "  - p;;,Ps(Qs(x; - p;i>>) 
i=l 
and 
n 
Gkn, ( P s  PP)) = 14x; - p:i)l, 
i=1 
and the corresponding per symbol average rate and distortion are 
1 
Dkn = G E [ d ( x k n ,  ( P a ( X L n ) ,  P;&kn)))]t 
and 
1 
Rkn = - E [ b ( x k " ) l  kn + T(Xkn ,  ( P a ( X L n ) ,  p $ i k n ) ) ) ] ,  
respectively. 
To design a locally optimal fixed rate mean-removed system, we again use a 
variation of the generalized Lloyd algorithm, iteratively improving the encoder for 
an existing set of predictions and codebooks, each prediction for a given encoder 
and codebook, and then each codebook for an existing encoder and prediction. For 
a given number of codebooks and a given fixed rate per codebook, the optimal 
(a ,  {Ps, P,""}) minimizes E[d(Xk",  (,Ba(Xkn), P,"ikn,))]. We therefore employ a descent 
algorithm on E [ d ( X k n ,  (Pa(Xkn), P,kixkn,))] to achieve a locally optimal solution. Using 
6,  PF, and fls to represent the updated versions of a,  P,"", and Ps, the descent 
algorithm proceeds as follows: 
1. ; i (Xkn) = arg min, d ( X k n ,  (Ps,  PJkn)) 
2. P$ = E[Xf - ps(xf - P;J;i(Xk") = s] 
3. j s  = arg minp E [ ~ ( x ' " ,  ( P ,  P P ) ) l ; i ( ~ k n )  = s] 
4. Iterate until convergence. 
Given a target rate, the optimal variable rate mean-removed system is the sys- 
tem that minimizes the distortion subject to a constraint on the rate. The generalized 
Lloyd algorithm is thus employed to minimize the Lagrangian 
E[d(Xk",  (A, PP)) + X G k " ,  ( P s ,  eP))l, 
where X is the Lagrangian coefficient and rate is calculated in terms of entropy. Thus 
the iterative design strategy becomes 
1. ;i(xkn) = arg mins[d(Xkn, (ps, ~ J k n ) )  + x [ ~ s I  + r-(xkn, (Ps, p,kn))ll 
2. Pti = E [ X ;  - Ps(X," - Pt;)I;i(Xkn) = s] 
3.  fls = arg minp E [ d ( ~ k " ,  ( P ,  Pp)) + ~ [ l s l +  T ( ~ k n ,  (P ,  P P ) ) ] ~  i t ( ~ k . 1  = s] 
4. Iterate until convergence. 
307 
Average Rate (bpp) 
Figure 2: Entropy constrained mean-removed WUVQ, entropy constrained WUVQ, 
and standard full-search VQ on medical test sequence. 
- variable rate mean-removed WUVQ,- - variable rate WUVQ, 
-.- standard full search VQ 
4 Experimental Results 
WUVQ and its mean-removed variation were tested on a sequence of medical 
images and compared to  standard full search VQ. The training sequence consisted 
of 20 magnetic resonance (MR) images and the test sequence consisted of 5 MR 
images outside of the training sequence. Performance results are reported in terms 
of signal-to-quantization-noise ratio (SQNR), where 
VAR(X) SQNR = 10 loglo ___ D '  
X is the pixel intensity, and D is the average squared-error between the 9-bpp original 
and the quantized reproduction. 
For both WUVQ and the mean-removed variation of WUVQ, supervectors 
consisted of 16 2 x 2  vectors and thus the dimension of the predictions for the mean- 
removed case was 64 (8x8). All multi-codebook systems consisted of at most 32 
codebooks, each with no more than 16 codewords. 
Figure 2 shows the performance of variable rate WUVQ and the variable rate 
mean-removed variation of WUVQ with varying values of A. Also in that figure is the 
performance of standard full searc,h VQ with a 2 x 2  vector dimension. All rates are 
reported in terms of entropy. At low rates WUVQ shows a 10.3 dB improvement over 
standard VQ. The mean-removed variation of WUVQ shows up to .5 dB improvement 
over WUVQ. 
Figure 3 shows an original test image alongside the same image compressed to 
308 
Figure 3: Original and compressed using the mean-removed variation of WUVQ at 
.51 bpp, SQNR = 15.67 dB 
.51 bpp using the mean-removed variation of WUVQ. 
5 Conclusion 
A mean-removed variation of WUVQ has been presented. The algorithm 
exploits the similarity between supervectors mapped to a specific codebook in a 
WUVQ system by subtracting a prediction from each supervector before encoding. 
A single index is used to specify both the codebook and the prediction, and thus no 
additional side information is required. The algorithm achieves at low rates a 10.5 
dB improvement over standard full search VQ followed by entropy coding and up to 
.5 dB improvement over WUVQ. Both WUVQ and the mean-removed variation can 
be generalized to multi-level systems. 
References 
[l] D. L. Neuhoff, R. M. Gray, and L. D. Davisson. Fixed rate universal block 
source coding with a fidelity criterion. IEEE Transactions on Information The- 
ory, 21(5):511-523, September 1975. 
[a] P. A. Chou. Code clustering for weighted universal VQ and other applications. In 
Proceedings of the IEEE International Symposium on Information Theory, page 
253, Budapest, Hungary, June 1991. 
309 
[3] P. A. Chou and M. Effros. Rate and distortion redundancies for source cod- 
ing with respect to a fidelity criterion. In IEEE International Symposium on 
Information Theory, San Antonio, Texas, January 1993. 
[4] P. A. Chou, T. Lookabaugh, and R. M. Gray. Entropy-constrained vector quanti- 
zation. IEEE Transactions on Acoustics Speech and Signal Processing, 37( 1):31- 
42, January 1989. 


A mean-removed variation of weighted universal vector quantization for image coding

Code clustering for weighted universal VQ and other applications.

Entropy-constrained vector quantization.

Fixed rate universal block source coding with a fidelity criterion.

Rate and distortion redundancies for source coding with respect to a fidelity criterion.

https://authors.library.caltech.edu/2955/1/ANDdcc93.pdf

A mean-removed variation of weighted universal vector quantization for image coding

Abstract

Similar works

Full text

Available Versions

Caltech Authors - Main

Caltech Authors - Main