This correspondence considers low-resolution scalar quantization for a memoryless Gaussian source with respect to absolute error distortion. It shows that slope of the operational rate-distortion function of scalar quantization is infinite at the point Dmax where the rate becomes zero. Thus, unlike the situation for squared error distortion, or for Laplacian and exponential sources with squared or absolute error distortion, for a Gaussian source and absolute error, scalar quantization at low rates is far from the Shannon rate-distortion function, i.e., far from the performance of the best lossy coding technique

Marco, Daniel

Neuhoff, David L.

English

Caltech Authors - Main

IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 53, NO. 3, MARCH 2007 1177
Low-Resolution Scalar Quantization for Gaussian Sources
and Absolute Error
Daniel Marco, Member, IEEE, and David L. Neuhoff, Fellow, IEEE
Abstract—This correspondence considers low-resolution scalar quanti-
zation for a memoryless Gaussian source with respect to absolute error
distortion. It shows that slope of the operational rate-distortion function
of scalar quantization is infinite at the point D where the rate becomes
zero. Thus, unlike the situation for squared error distortion, or for Lapla-
cian and exponential sources with squared or absolute error distortion, for
a Gaussian source and absolute error, scalar quantization at low rates is far
from the Shannon rate-distortion function, i.e., far from the performance
of the best lossy coding technique.
Index Terms—Absolute error, entropy constrained quantization,
Gaussian, low rate, low resolution, scalar quantization.
I. INTRODUCTION
This correspondence considers the asymptotic low-resolution, i.e.,
low-rate, performance of scalar quantization for a Gaussian source and
absolute error distortion measure. One motivation is to increase un-
derstanding of the performance of scalar quantization. While there are
well-known asymptotically accurate formulas for the rate-distortion
performance of scalar quantizers for the high-resolution region, only
a few results are known for the low-resolution region, see [1] and the
references therein. Another motivation comes from transform coding
where a sizable fraction of coefficients might need to be coded at low
rates, and one would like to know how well scalar quantization per-
forms relative to ideal lossy source coding.
The correspondence follows a somewhat similar path to that taken
in [1], where a Gaussian source and squarederror distortion measure
is considered. Specifically, we find the slope of the operational rate-
distortion function of scalar quantization, R(D), at D = Dmax, where
Dmax is the minimum distortion attainable with zero rate. This slope
determines the speed with which R(D)! 0 as D ! Dmax.
Following [1], we write
R(D) = s 1 
D
Dmax
1 + o
where oD!D is a quantity that tends to zero asD goes toDmax, and
s is the magnitude of the slope with respect to normalized distortion.
The values of s in the case of exponential and Laplacian sources with
both absolute and squared error distortion measures have been given in
[2]; the values in the case of a uniform source and both distortion mea-
sures can be deduced from [3]; the value of s for Gaussian source and
squared error was provided in [1]; finally, the value of s for a Gaussian
source and absolute error is given in this correspondence. Table I sum-
marizes these values.
Manuscript received December 30, 2005; revised November 27, 2006. The
work of D. Marco was supported by the Center for the Mathematics of Informa-
tion at the California Institute of Technology. The work of D. L. Neuhoff was
supported by the National Science Foundation under Grant CCR-0329715. The
material in this correspondence was presented at the IEEE International Sym-
posium on Information Theory, Seattle, WA, July 2006.
D. Marco was with the Department of Electrical Engineering, California In-
stitute of Technology, Pasadena, CA 91125 USA (e-mail: idaniel@ist.caltech.
edu).
D. L. Neuhoff is with the Department of Electrical Engineering and Com-
puter Science, University of Michigan, Ann Arbor, MI 48109 USA (e-mail:
neuhoff@eecs.umich.edu).
Communicated by W. Szpankowski, Associate Editor for Source Coding.
Digital Object Identifier 10.1109/TIT.2006.890714
TABLE I
MAGNITUDE OF THE SLOPE OF THE OPERATIONAL RATE-DISTORTION
FUNCTION R(D) AT D = D
We observe from Table I that since the slopes ofR(D) atD = Dmax
equal 0 for exponential and Laplacian sources with squared error, they
must equal the slopes of the corresponding Shannon rate-distortion
functions (because the latter’s magnitudes could be no larger). Further,
for a Gaussian source with squared error, and for Laplacian and
exponential sources with absolute error, the Shannon rate-distortion
functions are known [4], [5],1 and their slopes match the corresponding
slopes of R(D). Thus, in low resolution, scalar quantization for these
sources and distortion measures is asymptotically optimal, i.e., as
good as any quantization technique—vector or otherwise. For a
uniform source with both distortion measures, and for a Gaussian
source with absolute error, the slopes of R(D) at D = Dmax are
negatively infinite, whereas the slopes of the corresponding Shannon
rate-distortion functions cannot be negatively infinite, i.e., they must
be finite (because these functions are convex). Thus, for these sources
and distortion measures, low-resolution scalar quantization is far
from optimal. Furthermore, the negatively infinite slopes of R(D) at
D = Dmax for the mentioned sources and distortion measures imply
that the corresponding operational rate-distortion functions are not
convex (a fact that is already known in the case of uniform sources
[3]).
The remainder of this correspondence is organized as follows. Sec-
tion II provides background and introduces notation. The main result
is given in Section III. Section IV offers concluding remarks. Finally,
one lemma proof is left to the Appendix.
II. BACKGROUND
The assumption throughout the correspondence is that the source to
be quantized is stationary, memoryless, and Gaussian with zero mean
and variance 2. We denote this source by N (0; 2).
A scalar quantizer q is a partition of the real line into cells Sk , each
of which contains a reconstruction level rk such that when the input
lies in Sk , the output of the quantizer is rk . The number of cells may
be finite or infinite. Pk is the probability of the input lying in Sk . The
(output) entropy of quantizer q is given by H(q) =  
k
Pk logPk ,
where all logarithms in this correspondence have base 2. The mean
absolute error induced by the quantizer is
d(q) =
1
 1
jx  q(x)jf(x)dx =
k
S
jx   rkjf(x)dx
where f is the Gaussian density of the source. It is well known that
reconstruction levels at cell medians minimize mean absolute error,
for a given partition. Specifically, for a contiguous cell Sk , i.e., Sk =
[ak; bk), where it matters not if the interval is open or closed on either
side, the median rk satisfies
r
a
f(x)dx =
b
r
f(x)dx: (1)
1Reference [5] makes an error in applying its Theorem 2 to compute the
rate-distortion function, with respect to absolute error, of an exponential source.
Specifically, for f(x) = e ,  > 0, a correct application of this theorem
yieldsR(D) =   ln (2(1  e )), rather than the formula given in [5, eq.
(24)].
0018-9448/$25.00 © 2007 IEEE
1178 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 53, NO. 3, MARCH 2007
The operational rate-distortion function of scalar quantization for a
Gaussian source with variance 2 and absolute error distortion measure
is defined as follows:
R (D) = inf
d(q)D
H(q)
which specifies the least entropy of any scalar quantizer with distortion
D or less.
Let Dmax denote the minimum distortion attainable when the rate
is zero. Specifically, for a Gaussian source with variance 2 we have
Dmax =
2

.
Following the notation in [1], let the entropy function be defined as
H(. . . ; z 1; z0; z1; . . .) =  
1
k= 1
zk log zk
where 0 < zk  1 for all k, are a finite or countably infinite set
of numbers that need not sum to one. Let ox;y denote a quantity that
converges to zero when both x ! 1 and y ! 1. If this quantity
depends on parameters other than x and y, then its convergence to zero
is uniform in such parameters. Finally, G(x) = 1p
2
e  denotes the
Gaussian density with zero mean and unit variance.
III. MAIN RESULT
The following lemma is used to show Theorem 2 later.
Lemma 1: Consider a scalar quantizer applied to aN (0; 2) source.
If the cell containing the origin has boundaries  a and b, has recon-
struction level at the median, and contributes Do to the mean absolute
error of the quantizer, then
Dmax  Do =  G
a

+G
b

[1 + oa;b]:
Proof: Let ro denote the median of the cell ( a; b), and let f de-
note the Gaussian density with zero mean and variance 2. We evaluate
Do as follows:
Do =
r
 a
(ro   x)f(x)dx+
b
r
(x  ro)f(x)dx
= ro
r
 a
f(x)dx 
b
r
f(x)dx
 
r
 a
xf(x)dx+
b
r
xf(x)dx
(a)
=
b
r
xf(x)dx 
r
 a
xf(x)dx
(b)
=  2G
ro

 G
a

 G
b

;
where (a) is due to the fact that ro is the median of ( a; b), and (b)
follows since 1
x
tG(t)dt = G(x). Next, we observe that
Dmax =
1
 1
jxjf (x)dx = 2G(0):
Therefore,
Dmax Do =  G
a

+G
b

+2 G(0) G
ro

: (2)
Finally, we have
2 G(0) G
r0

=2
0
xG(x)dx
 2G(0)
r0

2
= G
a

+G
b

oa;b (3)
where the last equality is due to having
r0

2
= G
a

+G
b

oa;b
as shown by Lemma A1 of the Appendix. The lemma now follows from
(2) and (3).
The following theorem is the principal result of this correspondence.
Theorem 2: For aN (0; 2) source and absolute error distortion, the
operational rate-distortion function of scalar quantization satisfies
lim
D!D
R (D)
Dmax  D
=1 :
Proof: It suffices to consider only scalar quantizers with con-
tiguous cells, as follows from [6]. By definition of R (D), for any
D 2 (0;Dmax) there exists a quantizer q such that
H(q )  R (D) + (D) and d(q )  D (4)
where (D) is some function of D such that (D) > 0 and
limD!D
(D)
D  D = 0. (The choices of q and (D) are not
unique, but any fixed choices will do.) Let So;D = ( AD; BD)
denote the cell of q containing the origin (it is immaterial if the cell is
open or closed on either side). As D ! Dmax, AD; BD ! 1. Note
that either AD or BD (but not both simultaneously) might be infinite.
Let Do;D be the contribution to distortion of cell So;D, where the
reconstruction level of So;D lies at the median of the cell. It follows
from Lemma 1 that
Dmax  Do;D =  G
AD

+G
BD

[1 + o ]: (5)
Next, applying Lemma 5 from [1], which shows that H (Q(x)) =
log e
2
xG(x) [1 + ox], where Q(x) = 1x G(t)dt is the usual “Q func-
tion,” we obtain
H Q
AD

+H Q
BD

=
log e
2
AD

G
AD

+
BD

G
BD

[1 + o ]: (6)
Finally, we have that
lim inf
D!D
R (D)
Dmax  D
(a)
 lim inf
D!D
H(q )  (D)
Dmax   d(q )
(b)
 lim inf
D!D
H Q(A

) +H Q(B

)
Dmax  Do;D
(c)
=1
where (a) follows from (4), (b) is due to an elementary property of
entropy and from having Do;D  d(q ), and (c) derives from (5) and
(6). Thus,
lim
D!D
R (D)
Dmax  D
= lim inf
D!D
R (D)
Dmax  D
=1
as needed to show.
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 53, NO. 3, MARCH 2007 1179
IV. CONCLUSION
This correspondence considered the asymptotic low-resolution per-
formance of scalar quantizers for a Gaussian source with absolute error
distortion measure. This performance is determined by the slope of the
operational rate-distortion function of such quantizers at D = Dmax.
It was shown that the slope of the operational rate-distortion function of
scalar quantization is infinite, and hence does not match the slope of the
Shannon rate-distortion function, which is finite. Consequently, scalar
quantization is not an optimal coding technique, in asymptotically low
rate, for the given source and distortion measure. This is somewhat
surprising since, as noted earlier, scalar quantization is optimal for a
Gaussian source and squared error distortion measure, and for Lapla-
cian source and both squared and absolute error distortion measures.
APPENDIX
Lemma A1: Let a and b be the boundaries of the cell containing
the origin for a quantizer applied to a N (0; 2) source. Let ro be the
median of ( a; b). Then
r0

2
= G
a

+G
b

oa;b:
Proof: From (1) we obtain that
Q(
r0

) =
Q  a

+Q b

2
=
1 Q a

+Q b

2
: (A1)
Next, let v 2 be arbitrary. If v  0, then
Q(v) =
1
v
G(x)dx =
1
2
 
v
0
G(x)dx 
1
2
  vG(v)
from which it follows that 0  v   Q(v)
G(v)
. Similarly, if v < 0, then
Q(v) = 1 Q(jvj) = 1 
1
2
 
jvj
0
G(x)dx
=
1
2
+
jvj
0
G(x)dx 
1
2
+ jvjG(v) =
1
2
  vG(v)
from which it follows that  Q(v)
G(v)
 v < 0. These two bounds to v
imply that v2   Q(v)
G(v)
2
. This is now used as follows:
r0

2

1
2
 Q r

G r

2
(a)
=
1
2
  1
2
+
Q( )
2
 
Q( )
2
G r

2
(b)
=
Q a

 Q b

2p
2
[1 + oa;b]
2
(c)
= Q
a

 Q
b

oa;b
(d)
=
G a

a=
[1 + oa] 
G b

b=
[1 + ob] oa;b
= G
a

[1 + oa]oa  G
b

[1 + ob]ob oa;b
= G
a

+G
b

oa;b;
where (a) follows from (A1), (b) is obtained from the fact that r

! 0
as both a and b tend to infinity, (c) follows since Q a

 Q b

! 0
as both a and b tend to infinity, and (d) derives from having Q(x) =
1
x
G(x) [1 + ox] for x > 0, which is obtained from the fact that for any
x > 0,
1
x
1 
1
x2
G(x) < Q(x) <
1
x
G(x)
as shown in [7, pp. 82–83].
REFERENCES
[1] D. Marco and D. L. Neuhoff, “Low resolution scalar quantization for
Gaussian sources and squared error,” IEEE Trans. Inf. Theory, vol. 52,
no. 4, pp. 1689–1697, Apr. 2006.
[2] G. J. Sullivan, “Efficient scalar quantization of exponential and Lapla-
cian random variables,” IEEE Trans. Inf. Theory, vol. 42, no. 5, pp.
1365–1374, Sep. 1996.
[3] A. György and T. Linder, “Optimal entropy-constrained scalar quan-
tization of a uniform source,” IEEE Trans. Inf. Theory, vol. 46, no. 7,
pp. 2704–2711, Nov. 2000.
[4] T. Berger, Rate Distortion Theory. Englewood Cliffs, NJ: Prentice-
Hall, 1971.
[5] K. Yao and H. H. Tan, “Absolute error rate-distortion functions for
sources with constrained magnitudes,” IEEE Trans. Inf. Theory, vol.
IT-24, no. 4, pp. 499–503, Jul. 1978.
[6] A. György and T. Linder, “On the structure of optimal entropy-con-
strained scalar quantizers,” IEEE Trans. Inf. Theory, vol. 48, no. 2, pp.
416–427, Feb. 2002.
[7] J. M. Wozencraft and I. M. Jacobs, Principles of Communication En-
gineering. New York: Wiley, 1967.
On Divergence-Power Inequalities
Jacob Binia, Member, IEEE
Abstract—Expressions for (entropy-power inequality (EPI) Shannon
type) divergence-power inequalities (DPIs) in two cases (time-discrete and
time-continuous) of stationary random processes are given. The new ex-
pressions connect the divergence rate of the sum of independent processes,
the individual divergence rate of each process, and their power spectral
densities. All divergences are between a process and a Gaussian process
with same second-order statistics, and are assumed to be finite. A new
proof of the Shannon EPI, based on the relationship between divergence
and causal minimum mean-square error (CMMSE) in Gaussian channels
with large signal-to-noise ratio, is also shown.
Index Terms—Causal minimum mean-square error (CMMSE), diver-
gence rate, divergence-power inequality, entropy-power inequality.
I. INTRODUCTION
The Shannon entropy-power inequality (EPI) is expressed with dif-
ferential entropies of random variables or vectors. The aim of this cor-
respondence is twofold. First, we give results for the (EPI Shannon
type) divergence-power inequality (DPI) in two cases: time-discrete
and time-continuous stationary random processes. Simple expressions
connect the divergence rate of the sum of independent processes, the
individual divergence rates of each process, and their power spectral
densities. All divergences are between a process and a Gaussian process
with same second-order statistics, and are assumed to be finite.
Second, we show in the Appendix a new proof of the EPI based on
the relationship between divergence and causal minimum mean-square
error (CMMSE) in Gaussian channels with large signal-to-noise ratio.
The proof is similar to the new and simple one that was based on the
relationship between mutual information and minimum mean-square
error (MMSE) in Gaussian channels ([1], [2]).
Manuscript received September 7, 2006.
The author is with the New Elective–Engineering Services Ltd, 34970 Haifa,
Israel (e-mail: biniaja@netvision.net.il).
Communicated by K. Kobayashi, Associate Editor for Shannon Theory.
Digital Object Identifier 10.1109/TIT.2006.890715
0018-9448/$25.00 © 2007 IEEE


Low-Resolution Scalar Quantization for Gaussian Sources and Absolute Error

IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 53, NO. 3, MARCH 2007 1177
Low-Resolution Scalar Quantization for Gaussian Sources
and Absolute Error
Daniel Marco, Member, IEEE, and David L. Neuhoff, Fellow, IEEE
Abstract—This correspondence considers low-resolution scalar quanti-
zation for a memoryless Gaussian source with respect to absolute error
distortion. It shows that slope of the operational rate-distortion function
of scalar quantization is infinite at the point D where the rate becomes
zero. Thus, unlike the situation for squared error distortion, or for Lapla-
cian and exponential sources with squared or absolute error distortion, for
a Gaussian source and absolute error, scalar quantization at low rates is far
from the Shannon rate-distortion function, i.e., far from the performance
of the best lossy coding technique.
Index Terms—Absolute error, entropy constrained quantization,
Gaussian, low rate, low resolution, scalar quantization.
I. INTRODUCTION
This correspondence considers the asymptotic low-resolution, i.e.,
low-rate, performance of scalar quantization for a Gaussian source and
absolute error distortion measure. One motivation is to increase un-
derstanding of the performance of scalar quantization. While there are
well-known asymptotically accurate formulas for the rate-distortion
performance of scalar quantizers for the high-resolution region, only
a few results are known for the low-resolution region, see [1] and the
references therein. Another motivation comes from transform coding
where a sizable fraction of coefficients might need to be coded at low
rates, and one would like to know how well scalar quantization per-
forms relative to ideal lossy source coding.
The correspondence follows a somewhat similar path to that taken
in [1], where a Gaussian source and squarederror distortion measure
is considered. Specifically, we find the slope of the operational rate-
distortion function of scalar quantization, R(D), at D = Dmax, where
Dmax is the minimum distortion attainable with zero rate. This slope
determines the speed with which R(D)! 0 as D ! Dmax.
Following [1], we write
R(D) = s 1 
D
Dmax
1 + o
where oD!D is a quantity that tends to zero asD goes toDmax, and
s is the magnitude of the slope with respect to normalized distortion.
The values of s in the case of exponential and Laplacian sources with
both absolute and squared error distortion measures have been given in
[2]; the values in the case of a uniform source and both distortion mea-
sures can be deduced from [3]; the value of s for Gaussian source and
squared error was provided in [1]; finally, the value of s for a Gaussian
source and absolute error is given in this correspondence. Table I sum-
marizes these values.
Manuscript received December 30, 2005; revised November 27, 2006. The
work of D. Marco was supported by the Center for the Mathematics of Informa-
tion at the California Institute of Technology. The work of D. L. Neuhoff was
supported by the National Science Foundation under Grant CCR-0329715. The
material in this correspondence was presented at the IEEE International Sym-
posium on Information Theory, Seattle, WA, July 2006.
D. Marco was with the Department of Electrical Engineering, California In-
stitute of Technology, Pasadena, CA 91125 USA (e-mail: idaniel@ist.caltech.
edu).
D. L. Neuhoff is with the Department of Electrical Engineering and Com-
puter Science, University of Michigan, Ann Arbor, MI 48109 USA (e-mail:
neuhoff@eecs.umich.edu).
Communicated by W. Szpankowski, Associate Editor for Source Coding.
Digital Object Identifier 10.1109/TIT.2006.890714
TABLE I
MAGNITUDE OF THE SLOPE OF THE OPERATIONAL RATE-DISTORTION
FUNCTION R(D) AT D = D
We observe from Table I that since the slopes ofR(D) atD = Dmax
equal 0 for exponential and Laplacian sources with squared error, they
must equal the slopes of the corresponding Shannon rate-distortion
functions (because the latter’s magnitudes could be no larger). Further,
for a Gaussian source with squared error, and for Laplacian and
exponential sources with absolute error, the Shannon rate-distortion
functions are known [4], [5],1 and their slopes match the corresponding
slopes of R(D). Thus, in low resolution, scalar quantization for these
sources and distortion measures is asymptotically optimal, i.e., as
good as any quantization technique—vector or otherwise. For a
uniform source with both distortion measures, and for a Gaussian
source with absolute error, the slopes of R(D) at D = Dmax are
negatively infinite, whereas the slopes of the corresponding Shannon
rate-distortion functions cannot be negatively infinite, i.e., they must
be finite (because these functions are convex). Thus, for these sources
and distortion measures, low-resolution scalar quantization is far
from optimal. Furthermore, the negatively infinite slopes of R(D) at
D = Dmax for the mentioned sources and distortion measures imply
that the corresponding operational rate-distortion functions are not
convex (a fact that is already known in the case of uniform sources
[3]).
The remainder of this correspondence is organized as follows. Sec-
tion II provides background and introduces notation. The main result
is given in Section III. Section IV offers concluding remarks. Finally,
one lemma proof is left to the Appendix.
II. BACKGROUND
The assumption throughout the correspondence is that the source to
be quantized is stationary, memoryless, and Gaussian with zero mean
and variance 2. We denote this source by N (0; 2).
A scalar quantizer q is a partition of the real line into cells Sk , each
of which contains a reconstruction level rk such that when the input
lies in Sk , the output of the quantizer is rk . The number of cells may
be finite or infinite. Pk is the probability of the input lying in Sk . The
(output) entropy of quantizer q is given by H(q) =  
k
Pk logPk ,
where all logarithms in this correspondence have base 2. The mean
absolute error induced by the quantizer is
d(q) =
1
 1
jx  q(x)jf(x)dx =
k
S
jx   rkjf(x)dx
where f is the Gaussian density of the source. It is well known that
reconstruction levels at cell medians minimize mean absolute error,
for a given partition. Specifically, for a contiguous cell Sk , i.e., Sk =
[ak; bk), where it matters not if the interval is open or closed on either
side, the median rk satisfies
r
a
f(x)dx =
b
r
f(x)dx: (1)
1Reference [5] makes an error in applying its Theorem 2 to compute the
rate-distortion function, with respect to absolute error, of an exponential source.
Specifically, for f(x) = e ,  > 0, a correct application of this theorem
yieldsR(D) =   ln (2(1  e )), rather than the formula given in [5, eq.
(24)].
0018-9448/$25.00 © 2007 IEEE
1178 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 53, NO. 3, MARCH 2007
The operational rate-distortion function of scalar quantization for a
Gaussian source with variance 2 and absolute error distortion measure
is defined as follows:
R (D) = inf
d(q)D
H(q)
which specifies the least entropy of any scalar quantizer with distortion
D or less.
Let Dmax denote the minimum distortion attainable when the rate
is zero. Specifically, for a Gaussian source with variance 2 we have
Dmax =
2

.
Following the notation in [1], let the entropy function be defined as
H(. . . ; z 1; z0; z1; . . .) =  
1
k= 1
zk log zk
where 0 < zk  1 for all k, are a finite or countably infinite set
of numbers that need not sum to one. Let ox;y denote a quantity that
converges to zero when both x ! 1 and y ! 1. If this quantity
depends on parameters other than x and y, then its convergence to zero
is uniform in such parameters. Finally, G(x) = 1p
2
e  denotes the
Gaussian density with zero mean and unit variance.
III. MAIN RESULT
The following lemma is used to show Theorem 2 later.
Lemma 1: Consider a scalar quantizer applied to aN (0; 2) source.
If the cell containing the origin has boundaries  a and b, has recon-
struction level at the median, and contributes Do to the mean absolute
error of the quantizer, then
Dmax  Do =  G
a

+G
b

[1 + oa;b]:
Proof: Let ro denote the median of the cell ( a; b), and let f de-
note the Gaussian density with zero mean and variance 2. We evaluate
Do as follows:
Do =
r
 a
(ro   x)f(x)dx+
b
r
(x  ro)f(x)dx
= ro
r
 a
f(x)dx 
b
r
f(x)dx
 
r
 a
xf(x)dx+
b
r
xf(x)dx
(a)
=
b
r
xf(x)dx 
r
 a
xf(x)dx
(b)
=  2G
ro

 G
a

 G
b

;
where (a) is due to the fact that ro is the median of ( a; b), and (b)
follows since 1
x
tG(t)dt = G(x). Next, we observe that
Dmax =
1
 1
jxjf (x)dx = 2G(0):
Therefore,
Dmax Do =  G
a

+G
b

+2 G(0) G
ro

: (2)
Finally, we have
2 G(0) G
r0

=2
0
xG(x)dx
 2G(0)
r0

2
= G
a

+G
b

oa;b (3)
where the last equality is due to having
r0

2
= G
a

+G
b

oa;b
as shown by Lemma A1 of the Appendix. The lemma now follows from
(2) and (3).
The following theorem is the principal result of this correspondence.
Theorem 2: For aN (0; 2) source and absolute error distortion, the
operational rate-distortion function of scalar quantization satisfies
lim
D!D
R (D)
Dmax  D
=1 :
Proof: It suffices to consider only scalar quantizers with con-
tiguous cells, as follows from [6]. By definition of R (D), for any
D 2 (0;Dmax) there exists a quantizer q such that
H(q )  R (D) + (D) and d(q )  D (4)
where (D) is some function of D such that (D) > 0 and
limD!D
(D)
D  D = 0. (The choices of q and (D) are not
unique, but any fixed choices will do.) Let So;D = ( AD; BD)
denote the cell of q containing the origin (it is immaterial if the cell is
open or closed on either side). As D ! Dmax, AD; BD ! 1. Note
that either AD or BD (but not both simultaneously) might be infinite.
Let Do;D be the contribution to distortion of cell So;D, where the
reconstruction level of So;D lies at the median of the cell. It follows
from Lemma 1 that
Dmax  Do;D =  G
AD

+G
BD

[1 + o ]: (5)
Next, applying Lemma 5 from [1], which shows that H (Q(x)) =
log e
2
xG(x) [1 + ox], where Q(x) =
1
x
G(t)dt is the usual “Q func-
tion,” we obtain
H Q
AD

+H Q
BD

=
log e
2
AD

G
AD

+
BD

G
BD

[1 + o ]: (6)
Finally, we have that
lim inf
D!D
R (D)
Dmax  D
(a)
 lim inf
D!D
H(q )  (D)
Dmax   d(q )
(b)
 lim inf
D!D
H Q(A

) +H Q(B

)
Dmax  Do;D
(c)
=1
where (a) follows from (4), (b) is due to an elementary property of
entropy and from having Do;D  d(q ), and (c) derives from (5) and
(6). Thus,
lim
D!D
R (D)
Dmax  D
= lim inf
D!D
R (D)
Dmax  D
=1
as needed to show.
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 53, NO. 3, MARCH 2007 1179
IV. CONCLUSION
This correspondence considered the asymptotic low-resolution per-
formance of scalar quantizers for a Gaussian source with absolute error
distortion measure. This performance is determined by the slope of the
operational rate-distortion function of such quantizers at D = Dmax.
It was shown that the slope of the operational rate-distortion function of
scalar quantization is infinite, and hence does not match the slope of the
Shannon rate-distortion function, which is finite. Consequently, scalar
quantization is not an optimal coding technique, in asymptotically low
rate, for the given source and distortion measure. This is somewhat
surprising since, as noted earlier, scalar quantization is optimal for a
Gaussian source and squared error distortion measure, and for Lapla-
cian source and both squared and absolute error distortion measures.
APPENDIX
Lemma A1: Let a and b be the boundaries of the cell containing
the origin for a quantizer applied to a N (0; 2) source. Let ro be the
median of ( a; b). Then
r0

2
= G
a

+G
b

oa;b:
Proof: From (1) we obtain that
Q(
r0

) =
Q  a

+Q b

2
=
1 Q a

+Q b

2
: (A1)
Next, let v 2 be arbitrary. If v  0, then
Q(v) =
1
v
G(x)dx =
1
2
 
v
0
G(x)dx 
1
2
  vG(v)
from which it follows that 0  v 
 Q(v)
G(v)
. Similarly, if v < 0, then
Q(v) = 1 Q(jvj) = 1 
1
2
 
jvj
0
G(x)dx
=
1
2
+
jvj
0
G(x)dx 
1
2
+ jvjG(v) =
1
2
  vG(v)
from which it follows that
 Q(v)
G(v)
 v < 0. These two bounds to v
imply that v2 
 Q(v)
G(v)
2
. This is now used as follows:
r0

2

1
2
 Q r

G r

2
(a)
=
1
2
  1
2
+
Q( )
2
 
Q( )
2
G r

2
(b)
=
Q a

 Q b

2p
2
[1 + oa;b]
2
(c)
= Q
a

 Q
b

oa;b
(d)
=
G a

a=
[1 + oa] 
G b

b=
[1 + ob] oa;b
= G
a

[1 + oa]oa  G
b

[1 + ob]ob oa;b
= G
a

+G
b

oa;b;
where (a) follows from (A1), (b) is obtained from the fact that r

! 0
as both a and b tend to infinity, (c) follows since Q a

 Q b

! 0
as both a and b tend to infinity, and (d) derives from having Q(x) =
1
x
G(x) [1 + ox] for x > 0, which is obtained from the fact that for any
x > 0,
1
x
1 
1
x2
G(x) < Q(x) <
1
x
G(x)
as shown in [7, pp. 82–83].
REFERENCES
[1] D. Marco and D. L. Neuhoff, “Low resolution scalar quantization for
Gaussian sources and squared error,” IEEE Trans. Inf. Theory, vol. 52,
no. 4, pp. 1689–1697, Apr. 2006.
[2] G. J. Sullivan, “Efficient scalar quantization of exponential and Lapla-
cian random variables,” IEEE Trans. Inf. Theory, vol. 42, no. 5, pp.
1365–1374, Sep. 1996.
[3] A. György and T. Linder, “Optimal entropy-constrained scalar quan-
tization of a uniform source,” IEEE Trans. Inf. Theory, vol. 46, no. 7,
pp. 2704–2711, Nov. 2000.
[4] T. Berger, Rate Distortion Theory. Englewood Cliffs, NJ: Prentice-
Hall, 1971.
[5] K. Yao and H. H. Tan, “Absolute error rate-distortion functions for
sources with constrained magnitudes,” IEEE Trans. Inf. Theory, vol.
IT-24, no. 4, pp. 499–503, Jul. 1978.
[6] A. György and T. Linder, “On the structure of optimal entropy-con-
strained scalar quantizers,” IEEE Trans. Inf. Theory, vol. 48, no. 2, pp.
416–427, Feb. 2002.
[7] J. M. Wozencraft and I. M. Jacobs, Principles of Communication En-
gineering. New York: Wiley, 1967.
On Divergence-Power Inequalities
Jacob Binia, Member, IEEE
Abstract—Expressions for (entropy-power inequality (EPI) Shannon
type) divergence-power inequalities (DPIs) in two cases (time-discrete and
time-continuous) of stationary random processes are given. The new ex-
pressions connect the divergence rate of the sum of independent processes,
the individual divergence rate of each process, and their power spectral
densities. All divergences are between a process and a Gaussian process
with same second-order statistics, and are assumed to be finite. A new
proof of the Shannon EPI, based on the relationship between divergence
and causal minimum mean-square error (CMMSE) in Gaussian channels
with large signal-to-noise ratio, is also shown.
Index Terms—Causal minimum mean-square error (CMMSE), diver-
gence rate, divergence-power inequality, entropy-power inequality.
I. INTRODUCTION
The Shannon entropy-power inequality (EPI) is expressed with dif-
ferential entropies of random variables or vectors. The aim of this cor-
respondence is twofold. First, we give results for the (EPI Shannon
type) divergence-power inequality (DPI) in two cases: time-discrete
and time-continuous stationary random processes. Simple expressions
connect the divergence rate of the sum of independent processes, the
individual divergence rates of each process, and their power spectral
densities. All divergences are between a process and a Gaussian process
with same second-order statistics, and are assumed to be finite.
Second, we show in the Appendix a new proof of the EPI based on
the relationship between divergence and causal minimum mean-square
error (CMMSE) in Gaussian channels with large signal-to-noise ratio.
The proof is similar to the new and simple one that was based on the
relationship between mutual information and minimum mean-square
error (MMSE) in Gaussian channels ([1], [2]).
Manuscript received September 7, 2006.
The author is with the New Elective–Engineering Services Ltd, 34970 Haifa,
Israel (e-mail: biniaja@netvision.net.il).
Communicated by K. Kobayashi, Associate Editor for Shannon Theory.
Digital Object Identifier 10.1109/TIT.2006.890715
0018-9448/$25.00 © 2007 IEEE


Low-Resolution Scalar Quantization for Gaussian Sources and Absolute Error

Abstract

Similar works

Full text

Available Versions

Caltech Authors - Main

Caltech Authors - Main