The theoretical relationship among signal power, distortion, and bandwidth for several source and channel models is presented. The work is intended as a reference for the evaluation of the performance of specific data compression algorithms

Dolinar, S. J.

Pollara, F.

English

NASA Technical Reports Server

TDA Progress Report 42-102
N91-11980
A _ August15,1990
The Theoretical Limits of Source and Channel Coding
S. J. Dolinar and F. Pollara
Communications Systems Research Section
This article presents the theoretical relationship among signal power, distortion,
and bandwidth for several source and channel models. It is intended as a reference
for the evaluation of the performance of specific data compression algorithms.
I. Introduction
The theoretical limits on the perfornaance of source and
channel coding are well known for several source and chan-
nel models [1,3]. In this article these limits are calculated
for the Gaussian channel, used as a model of the deep-
space channel, and for some simple sources, potentially
useful as models of planetary images. The formulas un-
derlying these calculations are well known; the aim of this
article is to collect and graphically display the results. The
performance of specific data compression algorithms can
be compared to tile ultimate limits shown ill these graphs.
Similar results were presented in [2] for a binary symmetric
source with probability of error distortion criterion.
The results show the basic tradeoffs that must be made
among signal power, distortion, and bandwidth, assuming
that the system designer is free to design encoders and
decoders but has no control over the source, the channel,
or the user. These results also suggest that improvements
in information transmission in future missions should be
sought primarily through better source encoding rather
than by pushing channel coding gain closer to its limit.
II. The Communication System
Consider a communication system as shown in Fig. 1.
The source produces source symbols x with average en-
ergy Cr at the rate of R_ source symbols per second. The
source encoder outputs bits b with average energy Eb at the
rate of Rb bits/sec. The channel encoder produces channel
symbols y with average energy £u at tile rate of R u channel
symbols per second. The channel produces noisy outputs
_; the channel decoder and source decoder produce out-
puts b and &, attempting to reproduce tile values of b and
x, respectively.
A. Channel Models
Channel capacity is defined as
C(.) = max I(y; y) (1)
where I(y;,)) is the average nmtual information provided
by the output y about tile input y. The maximization is
performed over all probability densities q(y) satisfying tile
desired cost constraint E[y 2] < £_. Tim capacity is written
as a function of an unspecified argument; it depends on the
energy constraint £;_ and the transition statistics P([I]Y)
that characterize the channel.
Case C1. For a discrete-time additive white Gaussian
noise channel with continuous amplitude input and out-
put, and with noise distribution N(0, E,_), Eq. (1) yields
the capacity function in bits per channel symbol [3]:
1 log. (1 + p)
c(p) = -5 -
62
https://ntrs.nasa.gov/search.jsp?R=19910002667 2020-03-19T20:08:56+00:00Z
wherep = _?_/_ = 2gv/No, and No 2 is the two-sided
spectral density of tile noise.
As a comparison consider two additional channel mod-
els obtained by constraining the input and/or the output
of the additive white Gaussian noise channel. One chan-
nel, Case C3, is obtained by restricting the channel in-
put to binary symbols, while leaving the channel output
unconstrained. The other case, Case C2, is obtained by
constraining the channel output as well as its input to be
binary, that is, by hard-quantizing the Gaussian channel
output to two levels.
Case C2. A hard-quantized Gaussian channel with bi-
nary signaling and signal levels 4- x/t'_-v is a binary symmet-
ric channel (BSC) with crossover probability e = Q(v/'fi),
1 f oowhere Q(x) = _ x e-U_/2du" The capacity for such a
channel in bits per channel symbol is [3]
C(p) = 1 - 7-/[Q(x/-fi)]
where "H(¢) = -clog 2 e - (1 - e)1og2(1 - e) is the binary
entropy function.
Case C3. For the binary input Gaussian channel, the
capacity can be written [2]
c(p) = 1 - &[log2(1 + ¢-2")]
where u is a random variable with distribution N(p,p),
and E_, represents expectation over u.
B. Source Models
A discrete-time, continuous-amplitude stationary
source is considered. Its rate distortion function is defined
as
R(.) = rain I(x; _)
The minimization is over all conditional probability den-
sities p(klx) satisfying a distortion constraint E[d(x, _)] <
D, where d(.,.) is a distortion measure. As with channel
capacity, R(-) is written as a function of an unspecified
argument; it depends on the distortion constraint D and
on the source statistics p(x).
Case S1. For a Gaussian memoryless source with mean
square error (MSE) distortion constraint E[z 2] < ,f.¢,
where z = _ -x, the rate distortion function can be
expressed in bits per source symbol as [1]
1
/7(6) = 7 I°g2(1/5)' 0 < 6 < 1 (2)
where 6 = f z/E,: is the normalized MSE distortion.
Case $2. As a comparison consider a memoryless binary
symmetric source (BSS) under the probability of error dis-
tortion measure, Prob(d" # x) < Pc. This source has the
rate distortion fimction [1]
R(P,) = 1 - _(P,), 0 < t'_ < 1/2
Case S3. A simple example of a source with melnory is a
stationary Gaussian source with MSE distortion constraint
as in Case $1. In this case only a parametric form for R(a)
is known [1]:
/1 min[O, ¢P0(w)l&o6(0) =
1 max[0, log 2 ]dwR(0) = G .
oo _ ' ¢(k)
where (I)0(w) = _-_k=-oo _o)e -3k'_, and ,-'('63 is the nor-
malized autocorrelation flmction of the Gaussian process.
Of particular interest is the case of a first-order Markov
source with autocorrelation ¢(k) = Sx-r Ikl, 0 < 7 < 1,
which provides a good approximation to the autocorrela-
tion function of actual images, as shown in the example of
Fig. 2. The rate distortion function for a Gauss-Markov
source is shown in Fig. 3 for several values of 7- Note
that, at any given distortion level, the rate R varies in-
significantly with 7 for moderate values of the corrclation
coefficient (7 < 0.5) but is reduced dramatically as 7 ap-
proaches 1.
III. The Source-Channel Coding Theorem
Shannon's channel coding theorem and source coding
theorem can be merged into the source-channel coding
theorem [3], which is the central result of information the-
ory. The source-channel coding theorem answers the fol-
lowing fundamental question: Given a source with rate
distortion function R(.) and a channel with capacity func-
tion C(-), under what conditions is it possible, with suffi-
cient coding, to achieve source distortion and channel cost
that do not exceed the constraints used in the definitions
of R(.) and C(-), respectively? According to the theorem,
this is always possible if the capacity C(.) is greater than
the rate distortion R(.) (where the two fimctions must be
measured in consistent units), and never possible if the
rate distortion is greater than the capacity.
63
In general, the source symbol rate Rx need not corre-
spond to the channel symbol rate R,. Therefore the re-
quirement on the information transmission system is nor-
malized as
>
or
c(.) > (3)
where r = R_:/R u is the number of source symbols per
channel symbol. A large r corresponds to a small channel
bandwidth requirement relative to the bandwidth of the
source. To get an idea of the practical range of interest of
r, consider the following examples: A system with an 8-
bit/symbol source and no source compression on a channel
encoded with code rate 1/6 yields r __ 0.02; at another
extreme, the same source with a source compression ratio
of 8 : 1 and no channel coding yields r = 1; the Voyager
concatenated system with a source compression of 2 : 1
yields r _ 0.11.
This requirement can now be applied to all the combi-
nations of source and channel models introduced in Sec-
tion II. For all three channels considered in Section II,
the capacity C(.) is a function of a single variable p,
which is a signal-to-noise ratio involving the energy per
encoded channel symbol. Usually it is more meaning-
ful to measure the channel cost in terms of a signal-to-
noise ratio, Pb = fb/Cn = 2fb/No, involving the energy
per channel encoder input bit, or a signal-to-noise ratio,
p:: = _:/_,a = 2_Cx/N0, involving the energy per source
symbol. The variables p, pb, and px are related by
Rx Rb
rpx = -A-Px = P ----"W-Pb = rcpb -_ rrspb
r_u 1%
where rc = Rb/R u is the rate of the channel code, and
r, = Rb/Rx is tile rate of the source code. Therefore tile
capacity constraint, Eq. (3), can be equivalently written
in terms of px as
or, ill terms of Pb as
c( ,,0pb) >
With re taken to equal its minimum allowable value, R(.),
the latter requirement is
>
These formulas give the required relationships among r, P6
or Px, and the unspecified distortion variable in R(.).
Case S1.C1. h Gaussian channel and a Gaussian memo-
ryless source under the MSE distortion measure gives the
requirement
p > 6 -r - 1 (4)
For a given source and channel the designer seeks a
realization with small 6 and p, and large r. Of course
these are conflicting requirements, limited by Eq. (4), and
illustrated in Figs. 4(a), 4(b), and 4(c), where the feasi-
ble region is above each curve with a given parameter r.
When the source and channel have the same clock rate
(r = 1) and p + 1 = 1/6, the source and channel are per-
fectly matched, and ideal performance C = R is obtained
without any need for source or channel coding, assuming
proper scaling of the signal levels. If the channel clock rate
is much higher than the source clock rate, the required
£b/No approaches the well known infinite bandwidth ex-
pansion limit of -1.59 dB.
Case $2.C1. For a BSS and an unconstrained Gaussian
channel the following requirement, shown in Figs. 5(a),
5(5), and 5(c), is obtained:
p > 22r[1-_t(P')] - 1
Ia this case the distortion Pe is extremely sensitive to small
changes in fb/No or r. Conversely, extremely low error
probabilities can be obtained as long as Eb/No is greater
than the channel capacity limit for any fixed channel code
rate, rc = r.
Case S1.C2. A Gaussian memoryless source and a BSC
gives the requirement
2[x-_(_)] > 6-_/2
which relates the channel cost e = Q(x/_) to the normal-
ized MSE distortion 6, as in Figs. 6(a), 6(b), and 6(c).
The asymptotic behavior of these graphs for large signal-
to-noise ratio is due to the hard quantization introduced by
the BSC. The quantization loss can be seen by comparing
Fig. 6(a) to Fig. 4(a), Fig. 6(b) to Fig. 4(b), or Fig. 6(c)
to Fig. 4(c). The required Eb/No at infinite bandwidth
expansion is 0.37 dB, or 1.96 dB higher than the corre-
sponding requirement for the unquantized channel.
64
Case S2.C2. For a BSS and BSC under the Pe distortion
measure, the requirement is
1 - > r[1 -
as illustrated in Figs. 7(a), 7(b), and 7(c). When r = 1 and
e = Pc, another case of perfect matching between source
and channel is obtained, which yields ideal performance
C = R without any necd for source or channel coding.
The loss due to the hard quantization of the channel can
be seen by comparing Fig. 7(a) to Fig. 5(a), or Fig. 7(b)
to Fig. 5(b), or Fig. 7(c) to Fig. 5(c).
Case $2.C3. The cases involving the binary input Gaus-
sian channel do not yield a simple analytical expression.
The results are shown in Figs. 8(a), 8(b), and 8(e), for
this channel combined with the BSS. This source-channel
combination is the same ease considered in [2], and
Fig. 8(c) is directly comparable to Fig. 1 of [2]. (Note
that Eb/No in [2] refers to the source symbol signal-to-
noise ratio denoted E_/No in this article.)
Case S3.C1. Finally, the case of a Gauss-Markov source
and Gaussian channel is shown in Figs. 9, 10, and 11, for
three selected values of the rate r and several values of
the correlation coefficient 7. It is apparent that a Gauss
Markov source with 7 = 0.99, which is a good model for
some highly correlated planetary images, offers large sav-
ings in signal power due to tile high redundancy of the
source.
IV. Conclusion
The results presented provide a reference against which
the performance of specific data compression algorithms
can be measured. Choosing an appropriate model from
Figs. 4-11, and a desired ratio r of source symbol rate
to channel symbol rate, one can determine the minimum
channel symbol signal-to-noise ratio Ev/No, bit signal-to-
noise ratio ,fs/No, or source symbol signal-to-noise ratio
S;_:/No required to produce a normalized source distortion
that is less than 6 or Pc-
References
[1] T. Berger, Rate Distortion Theory, Englewood Cliffs, New Jersey: Prentice Ilall,
1971.
[2] S. A. Butman and R. J. MeEliece, "The Ultimate Limits of Binary Coding for a
Wideband Gaussian Channel," DSN Progress Report 42-22, vol. May-June 1974,
Jet Propulsion Laboratory, Pasadena, California, pp. 78-80, August 15, 1974.
[3] R. J. McEliece, The Theory of Information and Coding, Reading, Massachusetts:
Addison-Wesley, 1977.
65
SOURCE
USER
X
SOURCE
SYMBOLS
[SOURCE b I 1ENCODER BITS _ CHANNEL ENCODER
^
s°° CEiI IDECODER _1 CHANNEL DECODER _,
Flg. 1. A communlcatlon system model.
CHANNEL
SYMBOLS
CHANNEL I
v
lOOO
90O
80O
700
60O
500_
400-
3O0
-12
I ' I ' I ' [ ' I
A s ACTUI':': 
I _ I _ l , I , (
-8 -4 0 4 8
k
12
Fig. 2. Autocorrelation functions of a first-order Markov model
and of an image of the moon.
10-1
10-2
10-3
0
I 1 1 I
0,99
1,=0
1 2 3 4 5
R (6)
Fig. 3. Rate distortion function for a Gauss-Markov source with
correlation '7.
66
10-1
10-2
10-3
-20
1 (b)l
10-1_
10-2 _
1
10-3
-2
-15
I
\
\
I/8
-10 -5
Ey/N O, dB
I ]
r=2
1/4
o 5 lO
I I i I
r=2
-1 0 1 2 3 4
Eb/N O, dB
I I _ _ I
(c)
r=2
1,'2
I I I L i \t \
5 1 2 3 4 5 6
._x/No, dB
Fig. 4. Source-channel coding limits for a Gausslan source and a Gausslan channel: distortion versus (a) channel symbol signal-to-
noise ratio; (b) source-encoded bit signal-to-noise ratio; end (c) source symbol signal-to-noise ratio.
67

I t
=2
10-
10-2
10-3 I
-20 -15 -10 -5 0 5 10
Ey/No, dB
10-
10-2
T r 1 T--
r=2
1
10-3 __L__.L L
0 1 2 3 4 5 6
Eb/N O,dB
1 I I I 1 ] I
(c)
r=2
10-1 _
10-2
10-3 l I I I I I
7 0 1 2 3 4 5 6 7
Ex/No,a8
Fig. 6. Source-channel coding limits for a Gaussian source and a hard-quantlzed Gaussian channel: distortion versus (a) channel
symbol signal-to-noise ratio; (b) source-encoded bit signal-to-noise ratio; and (c) source symbol signal-to-noise ratio.
69
10-2 _ , /4
r=2
1
10-3 --
- 1/32
1/16
I/8
10-4 --
lO-S I
-20 -10
10 -1
10-2
1 _(b) T ]
EylNo, dB
0 10
r ]
10-3
10-4
II 11/81 '
IU_-116
t/3211 I I
lO-5 _ ___
0 1 2 3 4 5 6 7
Eb/N O, dB
r T T r T
r=2
10-
10-:
10-:
10 -_
10-!
N
1
1/2
0 1 2 3 4 5 6 7
Ex/,vo,_B
Fig. 7. Source-channel coding limits for a binary symmetric source (BSS) and a hard-quantlzed Gausslan channel: distortion versus
(a) channel symbol signal-to-noise ratio; (b} source-encoded bit signat.to-noise ratio; and (c) source symbol signal-to-noise ratio.
7O
I_ i I I
10-1 I _"_
10-2
//'32
10 -5 , [ I i I
-20 -15 -10 -5 0 5
EyIN O, dB
I --T l f T f
(b)
\10-1
10-2
10-3
10-4
11#
10
r=2
1/8
1/16
I1_ ,//f/_- 1/32
10 -5 = L_LJ.__L __.L__.L J L__
-2 -1 0 1 2 3 4
_b/No, dB
r=2
_ 1
1/2
1/8
J J_ l i_
-2 -1 0 1 2 3 4
._x/NO, dB
Fig. 8. Source-channel coding limits for s binary symmetric source (BSS) and a binary input Gausslan channel: distortion versus
(a) channel symbol signal-to-noise ratio; (b) source-encoded bit signal-to-noise ratio; and (c) source symbol signal-to-noise ratio.
71
10-
10-2
I I I I
r=l
7=0
0.99
10-3 I
-2 -1 0 1 2 3 4 5
Eb/No, dB
Fig. 9. Source-channel coding limits for a Gauss-Markov source
and a Gausslan channel (r = 1): distortion versus source-
encoded blt signal-to-noise ratio.
10-
10-2
10.3 I
-2
I I 1 I
r= 1/4
=0
0.99
I
-1 0 1 2 3 4 5
Eb#vo,dB
Fig. 10. Source-channel coding limits for a Gauss-Markov source
and a Gaussian channel (r ----1/4): distortion versus source-
encoded bit signal-to-noise ratio.
1
io-1 _
10-2 -
10-3
-2
I I I I I I
r = 1/16
y=O
0.5
0.9
0.99
-1 0 1 2 3 4 5
Eb/No, dB
Fig. 11. Source-channel coding limits for a Gauss-Markov source
and a Gausslan channel (r _-1/16): distortion versus source-
encoded bit signal-to-noise ratio.
72


The theoretical limits of source and channel coding

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server