Correlated sources X and Y are drawn i.i.d. according to probability mass function (pmf) p(x,y). In the side information source code (SISC) configuration: p(x,y) is known a priori to both the encoder and the decoder; the encoder knows X but not Y; the decoder knows Y but not X; the encoder encodes X and transmits the description of X to the decoder; the decoder reconstructs X using the source description and side information Y. The universal linked side information source code (ULSISC) configuration modifies the SISC configuration by assuming that p(x,y) is unknown a priori and that a asymptotically negligible amount of communication is allowed from the decoder to the encoder. We combine SISC design with ULSISC theory to build the codes for applications where the source statistics are unknown at design time. Experimental results compare ULSISC and SISC performance

Effros, Michelle

Jaggi, Sidharth

Zhao, Qiang

English

Zhao, Qian

Explore Bristol Research

Side information source coding: Low complexity design and source independence

Caltech Authors - Main

Side Information Source Coding: 
Low Complexity Design and Source Independence * 
Qian Zhao Sidharth Jaggi Michelle Effros 
{qianz, jaggi, effros}@z.caltech.edu 
Mectrical Engineering Department, 
California Institute of Technology, 
Pasadena, CA 91125 
Abstract 
Correlated sources X and Y are dmwn i.i.d. ac- 
cording to probability mass function (pmf) p(z, y). In 
the side information source code (SlSC) configuration: 
p ( x , y )  is known a priori to both the encoder and the 
decoder; the encoder knows X but not Y ;  the decoder 
knows Y but not X ;  the encoder encodes X and tmns- 
mits the description of X to the decoder; the decoder 
reconstrzlcts X using the source description and side 
information Y. The universal linked side information 
source code (ULSISC) configuration modifies the SISC 
configuration by assuming that p(z,y) is unknown a 
priori and that an asymptotically negligible amount of 
communication is allowed from the decoder to the en- 
coder. We combine SISC design with ULSISC theory 
to build codes for applications where the source statis- 
tics are unknown at design time. Experimental results 
compare ULSISC and S E C  performance. 
1 Introduction 
A side information source code (SISC) [l] is a data 
compression algorithm designed for a network with 
a single transmitter and a single receiver where the 
receiver is assumed to have access to  side informa- 
tion that is unavailable to the encoder. In particu- 
lar, consider finite alphabets X and Y ,  and assume 
that (XI, Y I ) ,  (X2, Yz), . . . are drawn i.i.d. according 
to joint pmf p(z,y) on alphabet X x Y .  An SISC 
comprises an encoder and a decoder. The encoder 
maps a sequence of source samples XI,. . . , X,, to a 
binary description, and the decoder reconstructs the 
'This material is based upon work supported by NSF under 
Award No. CCR-0220039, the Caltech Lee Center for Advanced 
Networking, and the Intel Technology for Education 2000 p r o  
gram. 
0-7803-7576-9/02$17.00 D 2002 IEEE 
source sequence using both the encoder's binary de- 
scription and the side information Y I ,  .. . , Y,,. Like 
traditional compression systems where no side infor- 
mation is available, SISCs are source dependent. That 
is, the optimal SISC designed for pmf p(z, y) is not the 
same as the optimal SISC designed for a distinct pmf 
q(z,y) ,  in general. 
When the pmfp(z,y) is unknown a t  design time, 
we require more sophisticated techniques to achieve 
good performance. A uniuersal linked side informa- 
tion sourte code (ULSISC) [2, 31 is a modified SISC 
that achieves asymptotically optimal performance for 
any pmf p(z,y) on fixed alphabet X x Y .  The modi- 
fication involves allowing an asymptotically negligible 
amount of communication from the decoder back to 
the encoder. This modification is critical to achieving 
universality in the SISC framework [Z, 31. 
The proof of the existence of ULSISCs given in [2,3] 
is constructive. The encoder describes some fraction of 
the incoming data sequence to the decoder using a sim- 
ple source code. The decoder then estimates p ( ~ ,  y) 
and describes its estimate t o  the encoder. Finally, the 
encoder describes X" using an SISC matched to the 
pmf estimate. A careful balance between the fraction 
of the data sequence used in the pmf estimate and 
the resulting estimation accuracy allows for asymp- 
totically optimal performance for ULSISC. 
This approach is difficult to implement in practice. 
Using an SISC matched t o  the estimate of p(z,y) re- 
quires that both the encoder and the decoder either 
have access to a family of pre-designed SECS or are 
able to design SISCs to  match the estimated pmf. 
However, optimal SISC design is NP-hard [4]. In this 
paper we investigate the performance associated with 
implementing the construction of [2, 31 using the fast 
but suboptimal SISC design techniques from [5 ,  61. 
The remainder of the paper is organized as follows. 
257 
Section 2 gives a brief review of fast SISC design. Sec- 
tion 3 applies fast SISC design to ULSISC design. Sec- 
tion 4 contains experimental results that compare UL- 
SISC and SISC performance, thereby quantifying the 
performance penalty associated with not knowing the 
source statistics. Section 5 gives asummary of the key 
contributions of the paper. 
2 Fast SISC Design 
Since optimal SISC design is NP-hard [4]: we use 
a fast, suboptimal SISC design algorithm. The algo- 
rithm in [5,6]  gives an iterative descent technique for 
searching the space of possible codes for a solution that 
minimizes the rate subject to a constraint on the error 
probability. We here employ an unconstrained mini- 
mization of the form R + U,, where R is the rate, 
P, is the error probability, and X is a non-negative 
L.agrangian constant. The output of the design is an 
instantaneous SISC with a non-zero error probability. 
Constraining the search to a number of operations that 
grows as O(lXi6)  yields good performance in the ex- 
periments performed to date and makes the algorithm 
feasible for on-line design. Some details follow. 
In designing SISCs with non-zero error probabili- 
ties, we allow only errors that do not propagate. Thus 
the code must decode the right number of bits wen 
xhen an error occurs. Given this constraint, any sym- 
bols I,Z' E X can have identical binary descriptions 
r.x(z) = rx(z'), but only symbols z,z' E X such 
that p(z,y)p(z',y) = 0 for all y E Y can have de- 
scriptions yx (I) and yx (I') that are proper prefixes 
of each other. 
In low complexity SISC design, we order alphabet 
X as 0 = {z1,12,. . . , z ~ } ,  where N = /XI and i < j 
implies zi precedes zj in the chosen ordering. The 
order-constrained SISC for ordering 0 requires that 
symbols with identical description hold adjacent posi- 
tions in 0, and groups of symbols whose descriptions 
are prefixes or siblings of each other hold adjacent po- 
sitions in 0. 
For a given ordering we use a dynamic pro- 
gramming approach to design the optimal order- 
constrained SISC such that J = R + XP, is mini- 
mized. The complexity of this approach is O ( N 4 ) .  
Finding the ordering that gives the globally optimal 
SISC is NP-hard. We therefore employ an iterative de- 
scent technique to search for a suboptimal ordering in 
polynomial time. Our experiments show that stopping 
the iterative descent technique after 0(M2) orderings 
usually gives satisfactory performance. Thus the total 
complexity of the approach is O(jv6). 
3 ULSISC Design 
Recall that in the ULSISC configuration, the se- 
quence pair ( X n , Y n )  is distributed i.i.d. according to  
p(z,y). The encoder sees X" but not Y". The de- 
coder sees Y" but not X". Since p(s,y) is unknown, 
we allow the decoder to  send some feed-hack to the 
encoder. The encodmg/decoding process for (X", Y") 
follows. 
The encoder uses a traditional universal code to  
transmit the first n(n )  symbols Xm(") = { X , ) z : )  
of X" to the decoder; this description requires 
L,(Xm(")) bits. The decoder then uses the recov- 
ered Xm(") and known Ym(n) to  calculate the empir- 
ical pmf Pm(z,y) = ~2:) ~ ( ( X , , X )  = ( ~ , ~ ) ) / m ( n ) ;  
the decoder describes this pmf to  the encoder us- 
ing S(m(n)) = IX(lyllog(m(n) + 1) bits. The en- 
coder and decoder then calculate a new pmf q(z, y) 
as a function of the estimate p,(z,y), and then in- 
dependently design identical codes to  match q(z;y). 
The encoder encodes the remaining n - m(n) sym- 
this description requires LS,~(X&,)+,) bits. The de- 
coder reconstructs X;(")+* with a number of errors 
eu,,(X%(n,+l). The total number of bits transmit- 
ted in the above process is Lv(X")  = Lm(Xm(n)) + 
6(m(n)) + L S , ~ ( X ; ( ~ ) + J .  The total number of errors 
In [2], it is shown that there exist codes such that 
bols X;(,,)+l - {X,};=m(,,)+l of X" using this SEC;  
is 4 X n )  = edx;(, ,)+,) .  
for any true underlying pmf p(z, y), we can achieve 
lim E,h(X") /n  = lim EDLs.q(XE(,)+l)/n 
lim E,eo(X")/n = lim Epeo.,(XEcn,+,)/n 
n-m n + m  
= ww), 
n+m n+m 
= 0. 
This performance is achieved using an ensemble of ran- 
dom codes. To achieve similar performance in practice, we 
choose the estimate q(2,y) as follows. 
We wish to use an estimate q(z,y) such that for the 
sequence (X~,,)+,,Y&)+,), the SISC matched to  q(z, y) 
has the rate and error probability performance "closen to 
the rate and error probability performance achieved by 
the SISC matched to the true pmfp(z.y). The empirical 
pmf pm(z, y) is a good candidate, since am($, y) is a good 
estimate of p(z,y) based on (Xm(nl,Ym(n)). 
However, for some (21,yl) E X x Y such that 
p(zi,yi) # 0, it is possible that pm(zl,yl) = 0 (e.g., 
if p(si,y~) is much smaller than l/m(n), then it is very 
likely that the pair (51, yl) will not appear in the sequence 
(Xmcn),Y*("))). Therefore for two symbols z,z' E X, 
even ifp(z,y)p(z', y) > 0 for some y E Y ,  an SISC matched 
to pmf p,(z, y) may still give z, 2' descriptions rx(z) and 
258 
yx(z’) that are proper prefixes of each other, and this will 
result in error propagation and catastrophic decoding fail- 
ure. 
To remedy this problem, we choose the estimate q ( z , y )  
as follows. Let Z be the number of zeros in the empirical 
pmfp,(z,v). Then 
C m  ifp,(s,y)=O 
q(z’y)={  p,(z:y)(l-emZ) otherwise 
where t, is as small as possible, e.g., we use cm = lo-‘ in 
our experiments. 
4 Experimental Results 
The following experimental results show the perfor- 
mance of a code of the type described in Section 3 on two 
classes of randomly generated sources with 1x1 = IYl = 8. 
The first class of random sources assumes a uniform 
probability density function p on the family of memory- 
less pmfs on X x Y .  A pmf p(z, y) is generated by fust 
independently choosing a random value from [O, 1) for each 
entry of p(z, g) then normalizing. 
The second class of random sources assumes a fixed per- 
centage of zero entries in p(z,y) and is generated by as- 
signing zeros to a number of randomly chosen entries then 
choosing the remaining entries at random and normalizing. 
In our experiments, we generate four groups of pmfs 
with 20 pmfs each. The first group comes from the first 
class of pmfs. The second, third and fourth groups are from 
the second class of pmfs and containing 25%, 50%, and 75% 
percent zero elements, respectively. The average entropies 
for these four groups of p(z, y)s are given in Table 1. 
For each p(z,y), we randomly generate 50 pairs of se- 
quences ( X ’ , Y M ) ( M  = 65536) i.i.d. according to p ( q g ) .  
For each (z’, y’) pair, we design a sequence of indepen- 
dent ULSISCs with block length n 5 M. Thus for every 
length-n subsequence pair (z”, y”), we design a ULSISC 
for (z”,y”) as in Section 3. The performances are then 
averaged over the 50 pairs and are further averaged over 
the 20 pmfs in the same group. By varying m(n), n, and X 
(used in SISC design) we obtain curves of rates and error 
probabilities. 
We compare these cum- with the performance of an 
SISCdesignedfor theti-nepmfp(z,y) andthesame Xused 
in ULSISC design. Again we average over the 50 pairs and 
over the 20 pmfs in the s a m e  group. We denote the average 
performances of ULSISC and SISC by (Ru, P e p ,  Ju) and 
(Rs,P,,s, J s )  respectively. 
For the low complexity SISC design, we use the mul- 
tiple descent neighbor algorithm 161 allowing a total of C 
orderings to be searched. We use the same pseudo-random 
number generator at the encoder and the decoder to en- 
sure that the decoder can independently perform the SISC 
design performed at the encoder. 
We test on X = 2‘,0 5 i _< 7, X = 0, and X = 1OM)O. 
As X increases, rate R increases and error probability P. 
Table 1: Average statistics. 
fw) 2.9684 2.9321 2.8668 2.7021 
H ( X ( Y )  2.7321 2.3462 1.9404 1.1699 
Huffman Rate 2.9959 2.9663 2.9069 2.7518 
L 
decreases. By varying X from zero to infinity, we trace 
out the curves for R against Pe. For the SISC curve, we 
can achieve Pe,s = 0 by setting X = m. For the ULSISC 
curves, P.,u = 0 is not always achieiable. We focus on 
P. E l0,O.l)  in the following analysis. 
Figure 1 shows the performance far m(n) E 
{nf2, no-’, Jiilogn, J;;,logn}. As expected, for m(n) 
large (e.g., m(n) = n/Z or no-’), the estimate g(z,y) is 
more accurate resulting in a smaller Pe,u hut mare rate 
spent on sending these m(n) symbols to the decoder. Sim- 
ilarly, for m(n) small (e.g., m(n) = logn), P,,u is larger 
but &,U is smaller. We observe that m(n) = f i logn  
and m(n) = Jii achieve performance very close to that of 
the SISC in these experiments. (Note that n ( n )  = logn 
results in large P, and does not appear in the figure.) 
Figure 2 compares the curves for m(n) = f i l o g n  as 
n increases. As expected, as n increases, the ULSISC 
performance approaches the SISC performance. When 
n = A4 = 65536: the observed performances are very dose 
(but not identical). 
We next investigate the trade-off between SISC design 
complexity and ULSlSC performance. As shown in Fig- 
ure 3 by increasing the design complexity of SISC from 
C = N to C = N 3 ,  the performance of ULSISC improi’es. 
Finally, we compare the performance for the four gaups 
of pmfs. For the fist group, p ( z ,  y) contains almost no 
zero d u e  elements and H ( X I Y )  is very close to H ( S )  
(see Table 1). Thus the hene6ts of ULSISC is not obvious, 
since the theoretically achievable rate H ( X I Y )  of ULSlSC 
does not differ much from the theoretically achievable rate 
H(X) of traditional universal codes. As the number of 
zeros in p(z,y) increases, the difference between H(XIY)  
and H(X) also increases, thus the benefit of ULSISC over 
traditional universal code amplifies. As shown in figure 2, 
when Pc E [O,O.l], the achievable rate of ULSISC ranges 
in [I, 2.251 for p(z,y)’s with 75% zeros and increases to 
[2.25,3.25] for p(z,y)’s wit.h 25% zeros. (Here, a rat,e 
greater than log 1x1 is caused hy the nonneglible number 
of bits 6(m(n)) that is needed by the decoder to transmit 
the estimate q(z,y)  to the encoder when the block length 
n is not large enough.) 
259 
g 01 
. . . . . . .  .......... ...  
. . . . .  . ...... ...... -. 
, \ .  ' .'..? ,  . 'L. . + .  \ \ 
.+:>.:, .... .. , . . - ?~.* 
0 ;  1 2  I .  I. z 1  
R 
Figure 1: Observe P,,u - Rv curves for various m(n) Figure 2: Observe Pe,v - Ro curves for various n 
(n = 65536, C = 512). (m(n) = Jii1ogn.C = 512). 
260 
5 Summary 
In this paper, we present a low complexity ULSISC de- 
sign algorithm. Our experimental results give guidelines 
for choosing appropriate parameters used in these algo- 
rithms. 
References 
[l] D.Slepian and J.K.Wolf. Noiseless coding of correlated 
information sources. IEEE Pansactions on Informa- 
tion Theory, IT-19(4):471-480, July 1973. 
[Z] S. Jaggi and M. EEros. Universal linked multiple 
access source code. In Proe. of the IEEE Intemn- 
tional Symposium on Information Theory, Lausanne, 
Switzerland, June 2002. 
[3] S. Jaggi and M. Efios. Universal linked multiple ac- 
cess source codes. IEEE Dansactions on Information 
Theory. Submitted 2002. In review. 
[4] P. Koulgi, E. Tuncel, S. Regunathan, and K. Rase. 
Minimum redundancy zero-error source coding with 
side information. In Proc. of the IEEE International 
Symposium on Information T h w y ,  Washington DC, 
USA, June 2001. IEEE. 
151 Q. Zhao and M. Efios. Lossless and near lossless source 
coding for multiple access networks. IEEE 7kansac- 
tions on Information Theory, January 2003. To Ap- 
pear. 
161 Q. Zhao and M. E&os. Low complexity code design for 
lossless and near lossless side information source codes. 
Submitted to Data Compression Conference 2003. 
Figure 3: Observe Pep  - Ru curves for various C 
(m(n) = filogn). 
261 


Side information source coding: low complexity design and source independence

Correlated sources X and Y are dmwn i.i.d. ac-cording to probability mass function (pmf) p(z, y). In the side information source code (SlSC) configuration: p ( x, y)  is known a priori to both the encoder and the decoder; the encoder knows X but not Y;  the decoder knows Y but not X;  the encoder encodes X and tmns-mits the description of X to the decoder; the decoder reconstrzlcts X using the source description and side information Y. The universal linked side information source code (ULSISC) configuration modifies the SISC configuration by assuming that p(z,y) is unknown a priori and that an asymptotically negligible amount of communication is allowed from the decoder to the en-coder. We combine SISC design with ULSISC theory to build codes for applications where the source statis-tics are unknown at design time. Experimental results compare ULSISC and S E C performance. 

Side information source coding: low complexity design and source independence

Abstract

Similar works

Full text

Available Versions

Explore Bristol Research

Caltech Authors - Main

CiteSeerX