The following question about Huffman coding, which is an important technique for compressing data from a discrete source, is considered. If p is the smallest source probability, how long, in terms of p, can the longest Huffman codeword be? It is shown that if p is in the range 0 less than p less than or equal to 1/2, and if K is the unique index such that 1/F(sub K+3) less than p less than or equal to 1/F(sub K+2), where F(sub K) denotes the Kth Fibonacci number, then the longest Huffman codeword for a source whose least probability is p is at most K, and no better bound is possible. Asymptotically, this implies the surprising fact that for small values of p, a Huffman code's longest codeword can be as much as 44 percent larger than that of the corresponding Shannon code

Abu-Mostafa, Y. S.

Mceliece, R. J.

English

NASA Technical Reports Server

TDA ProgressReport42-110 August15, 1992
/(
Maximal Codeword Lengths in Huffman Codes
Y. S. Abu-Mostafa
CaliforniaInstitute of Technology, ElectricalEngineeringDepartment
R. J. McEliece
Communications SystemsResearch Section
In this article, the authors consider the following question about Huffman coding,
which is an important technique for compressing data from a discrete source. If p
is the smallest source probability, how long, in terms of p, can the longest Huffman
codeword be? It is shown that if p is in the range 0 < p _< I/2, and if K is the
unique index such that I/FI<+3 < p _< 1/FK+2, where FK denotes the Kth Fibonacci
number, then the Iongest tluffman codeword for a source whose least probability is
p is at most K, and no better bound is possible. Asymptotically, this implies the
surprising fact that for small values of p, a Huffman code's longest codeword can
be as much as 44 percent larger than that of the corresponding Shannon code.
I. Introduction and Summary
Huffman coding is optimal (in the sense of minimiz-
ing average codword length) for any discrete memoryless
source, and Huffman codes are used widely in data com-
pression applications. In many situations it would be use-
ful to have an easy way to estimate the longest Huffman
eodeword length for a given source, without having to go
through Huffman's algorithm, but since there is no known
closed-form expression for the Huffman codeword lengths,
no such estimate immediately suggests itself. However,
since the longest codeword will always be associated with
the least-probable source symbol, one way to address this
problem is to ask the following question: Ifp is the smallest
source probability, how long, in terms of p, can the longest
Huffman codeword be? It turns out that this quantity, de-
noted by L(p), is easy to calculate, and so L(p) provides an
"easy estimate" of the longest Huffman codeword length.
The formula for L(p) involves the famous Fibonacci
numbers (Fn),>0, which are defined recursively, as follows:
Fo = O, FI = I, and F,, = Fn_l + Fn-2 for n > 2 (1)
Thus, F2 = 1, F3 = 2, F4 = 3, F5 = 5, F_ = 8, etc. The
Fibonacci numbers and their properties are discussed in
detail in [1, Section 1.2.8]. Here is the main result of this
article. (Note that since the definition of L(p) assumes p
to be the smallest probability in a source, p must lie in the
range 0 < p _< 1/2.)
188
https://ntrs.nasa.gov/search.jsp?R=19930010238 2020-03-17T08:14:57+00:00Z
Theorem 1. Let p be a probability in the range 0 <
p < 1/2, and let K be the unique index such that
1 1
< p < (2)
Then L(p) - K. Thus p e (1/3, 1/2] implies L(p) = 1,
p e (1/5, 1/3] impies L(p) -- 2, p fi (1/8, 1/5] implies
L(p) = 3, etc.
It is easy to prove by induction that the Fibonacci num-
bers satisfy the following inequalities:
¢,-2 < F, < ¢,-I for n > 3 (3)
where ¢ = (1 +v/5)12 = 1.618... is the "golden ratio." By
combining inequality (3) with Theorem 1, one sees that
logs ! _ 2 < L(p) < log,, (4)P
which, in turn, implies that
lim L(p)
p-.0 = 1 (5)
Since log s z - (log 2 x)/(log 2 _b) = 1.4404 log 2 x, Eq. (5)
implies the surprising fact that for small values of p, a Huff-
man code's longest ¢odeword can be as much as 44 percent
larger than that of the corresponding (in general, subopti-
mal) Shannon code [2, Chapter 5], which assigns a symbol
with probability p a codeword of length [log2 bl.
Theorem 1 is closely related to a result of Katona and
Nemetz [4], which identifies the length of the longest pos-
sible Huffman codeword for a source symbol of probability
p (whether or not p is the smallest source probability).
Denoting this quantity by L*(p), their result is as follows:
Theorem 2. (Katona and Nemetz [41) Let p be a prob-
ability in the range 0 < p < 1, and let K be the unique
index such that
1 1
< p < (6)
Then L*(p) = K. Thus, p E [1/2, 1) implies L*(p) = 1,
p e [1/3,1/2) implies L'(p) = 2, p e [1/5,1/3) implies
L*(p) = 3, etc.
By comparing Theorems 1 and 2, one sees that L*(p) =
L(p) + 1 unless p is the reciprocal of a Fibonacci number,
in which case L*(p) = L(p))
II. Proof of Theorem 1
The proof of Theorem 1 is in two parts. First, it will be
shown that ifp > 1/FK+3, then in any Huffman code for
a source whose smallest probability is p, the longest code-
word length is at most K. In fact, a considerably stronger
result will be proved. The class of efficient prefix codes
will be defined, and it will be shown that any Huffman
code, and in fact any optimal code for a given source, is
efficient. Then it will be shown that ifp > 1/FK+3, in any
efficient code for a source whose smallest probability is p,
the longest codeword length is at most K. In the second
half of the proof, it will be shown that ifp <: 1/FK+_, there
exists a source whose smallest probability is p, which has
at least one Huffman code whose longest word has length
K. As an extension, it will be seen that if p < 1/FK+2,
there exists a source whose smallest probability is p, and
for which every optimal code has the longest word of length
K. (If p = 1/FK+2, however, there is no such source.)
Now comes the definition of efficient prefix codes, which
is best stated in terms of the associated binary code tree
(see Fig. 1). Each source symbol and its corresponding
codeword is associated with a unique terminal node on
the tree. Also, each node in the tree is assigned a proba-
bility. The probability of a terminal node is defined to
be the probability of the corresponding source symbol,
and the probability of any other node of the code tree
is defined to be the sum of the probabilities of its two
"children." The level of the root node is defined to be
zero, and the level of every other node is defined to be
one more than the level of its parent. Two nodes de-
scended from the same parent node are called siblings.
Figure 1 shows two different code trees for the source
[3/20, 3/20,3/20, 3/20,8/20]. The tree in Fig. l(a) cor-
responds to the prefix code {000,001,01, 10, 11}, and the
tree in Fig. l(b) corresponds to {000,001,010,011, 1}.
Definition. A prefix code for a source S is efficient if
every node except the root in the code tree has a sibling,
and if level(v) < level(v') implies p(v) > p(v').
1 In fact, however, if one were to make a subtle change in the deft-
rdtion of L(p), this special case would disappear, The change re-
quired is to define L(p) as the minimum maximum Huffman code-
word length over all Huffman codes for a source with p as the least
probability, where the outer minimum is over all Huffman codes
for a given source.
189
Gallager [3] noted that every Huffman tree is efficient,
but in fact it is easy to see more generally that every op-
timal tree is efficient. This is because in an ine_cient
tree, with nodes v and v' such that level(v) < level(v')
but p(v) < p(v_), by interchanging the subtrees rooted
at v and v _, one arrives at a new code tree for the same
source, whose average length has been reduced by ex-
actly (level(v') - level(v))(p(v') - p(v)). However, it is
not true that every efficient code is optimal. Indeed,
Fig. 1 shows two different efficient code trees for the source
[3/20, 3/20, 3/20, 3/20, 8/20]. The code in Fig. l(b) is op-
timal, but the one in Fig. l(a) is not.
Theorem 3. Ifp > 1/FK+s, then in any efficient prefix
code for a source whose least probability is p, the longest
codeword length is at most K.
Proof: The contrapositive will be proved, i.e., if p is the
least probability in a source that has an efficient prefix
code whose longest word has length >_ K + 1, then p <
1/FK+3.
Thus, suppose that S is a source whose least proba-
bility is p and that there is an efficient prefix code for S
whose longest word is of length >__K + 1. In the code
tree for this code, there must be a path of length K + 1
starting from the terminal node, which corresponds to the
longest word and moves upward toward the root. This
path is shown in Fig. 2 as the path whose probabilities axe
PO,Pl,... ,pK+:. Since the code is assumed to be efficient,
each of the vertices in this path (except possibly the top
vertex) has a sibling; these siblings are shown in Fig. 2 as
having probabilities qo, ql,..., qK. Now one can prove the
following:
pi>_Fi+2p fori=O, 1,...,K+l (7)
The proof of (7) is by induction. For i = 0, (7) merely says
that P0 > P, which is true since P0 = P, by definition. Also,
note that q0 _> P since p is the least source probability.
Thus, Pl = P0 + q0 > P + P = 2p = F3p, which proves
(7) for i = 1. For i > 2, one has Pi = pi-1 + qi-1. But
pi-1 >_ /_+lp by induction, and qi-1 > Pi-2 since the
code is efficient (qi-1 is a higher level node than pi-2).
Thus, one has qi-1 >_ Pi-2 >-- rip by induction, and so
Pi = Pi-1 "{-qi-1 > (Fi+I -{-Fi)p = Fi+2p, which completes
the proof of (7).
Now consider the probability PK+I. On one hand,
PK+: < 1; but on the other hand, PK+I >_ FK+3P, by
(7). Thus, p <_ 1/FK+3, which completes the proof.
Theorem 4. If p < 1/FK+2, there exists a source
whose smallest probability is p and which has a Huffman
code whose longest word has length K. If p < 1/FK+2,
there exists such a source for which every optimal code has
a longest word of length K.
Proof: Consider the following set of K + 1 source proba-
bilities:
p F: F_ Fh'-I FK+I p]
"FK+_' Fg+2'"" FK+2' F/¢+2 (8)
Note that p is the minimal probability for this source, since
p < 1/FK+2 -" Fz/FK+2. Now, consider the code tree for
this source depicted in Fig. 3, which a_igns the source
probability p a word of length K. This tree is in fact a
Huffman tree for these probabilities, i.e., a code tree that
arises when Huffman's algorithm is applied to the source
of (8). To see this, one first proves that the internal vertex
probabilities pl in Fig. 3 are given by the following formula:
Pi = FI+_/FK+2 - h, for i = 0, 1,..., K - 1 (9)
p/< -- 1 (10)
where h = 1/FK+2 -p.
To prove (9), one uses induction. For i = 0, by def-
inition, p0 = p = I/FK+2--h = F2/FK+z-h. For
i > 1, one then has Pi = Pi-1 + Fi/FK+_ = (Fi+z/FK+I -
h) + Fi/FK+2 = Fi+2/FK+2 - h. To prove (10), note
that PK = PK-Z + (FK + 1)/FK+2 -- p. But from (9),
PK-1 = (FK+I/FK+2 -- h), so that PK = (FK+I/FK+2 --
h) + (FK/FK+2 + h) = FK+2/FK+2 = 1. Thus the prob-
abilities in (8) sum to one.
It now follows that the tree in Fig. 3 is a Huffman tree,
for from (9) one sees that at the ith stage (i -- 0,..., K-l),
the "collapsed" source consists of the probabilities
[&+_/FK+2 - h, &+I/FK+_, F_+2/FK+_,...,
FK-_/FK+2, FK/FK+2 + h] (II)
Plainly the two leftmost probabilities in (11), namely
Fi+2/FN+2 - h and FI+I/FK+2, are two of the smallest
probabilities, and so the tree of Fig. 3 is a Huffman tree,
as asserted.
Finally, note that if h > 0, i.e., if p < 1/FK+2, that
the leftmost two probabilities in (11) are uniquely the two
190
smallest probabilities in the list, so that the Huffman tree
in Fig. 3 is the unique Huffman tree for the source of
Eq. (8). And since the set of codeword lengths in any
optimal code is the same as the set of lengths in some
Huffman code, the last statement in Theorem 4 follows.
D
By combining Theorems 3 and 4, one obtains a result
that is stronger than Theorem 1.
Example 1: Letp= 2 -s. Then l/F14 = 1/377 < p <
1/F13 -- 1/233, and so by Theorem 1, L(2 -s) = 11, More
concretely, Theorem 3 shows that no Huffman code for a
source whose smallest probability is 2 -s can have a code-
word whose length is longer than 11. By Theorem 4, on
the other hand, every optimal code for the source
2_ s 1 1 2 3 5 8 13 21
'2--_' _'3' 233' 2"_' 233' 233' 233' 233'
34 55 90 ]233'233' 233 2-s (12)
has a longest word of length 11. D
III. Extension of the Katona-Nemetz
Theorem
In this section, two theorems are stated without proof.
When taken together, they yield a result that is slightly
stronger than Katona and Nemetz's Ttleorem 2. The
proofs are entirely similar to the proofs of Theorems 3
and 4.
Theorem 5. Let S be a source containing a symbol a
whose probability is p. Ifp :> 1/FK+_, then in any efficient
prefix code for S, the length of the codeword assigned to
the symbol a is at most K.
Theorem 6. Let p < 1/FK+I. Then there exists a
source S containing a symbol a whose probability is p, and
such that every optimal code for S assigns a a codeword
of length K. Explicitly, one sucli source is given by
1 /'1 F2_ FK-_ ]S = Fr_+_ P- e'P' F_+_' Fs,-+,"'" _+--'--_+ e
(13)
where e is any real number such that 0 < e < 1/FK+2 -- p.
Example 2: Let p= 2 -s. Then 1/F14 = 1/377 < p <
1/F13 = 1/233, and so by Theorem 2, L*(2 -s) = 12. In-
deed, by Theorem 6, every optimal code for the source
1 _2_ s 1 1 2 3 5 8- e, 2 -s, 23"---3'23---3'233' 233' 233' 233'
13 21 34 55 89 1
233' 233' 233' 233' 233 + eJ (14)
where 0 < e < 1/233 - 1/256, assigns the symbol with
probability 2 -s a codeword of length 12.
Acknowledgment
The authors are grateful to Douglas Whiting of STAC, Inc., for suggesting this
problem.
191
References
r
[1] D. E. Knuth, The Art of Computer Programming, vol. 1: Fundamental Algo-
rithms, 2nd ed., Reading, Massachusetts: Addison-Wesley, 1973.
[2] T. M. Cover and J. A. Thomas, Elements of Information Theory, New York:
Wiley & Sons, 1991.
[3] R. V. Gallager, "Variations on a theme by Huffman," IEEE Trans. Inform.
Theory, vol. IT-24, pp. 668-674, November 1978.
[4] G. O. H. Katona and T. O. It. Nemetz, "I-Iuffman Codes and Self-Information,"
IEEE Trans. b_form. Theory, vol. IT-22, pp. 337-340, May 1976.
192
(a) (b)1 1
3/20 3/20 3/20 3/20 3/20 3/20
Fig. 1. Two code trees for the source [3/20, 3/20, 3120, 3/20, 3/20]:
(a) a tree that Is efflclent but not optlmal (average length = 2.3) and
(b) a tree that Is optlmal (average length = 2.2).
LEAS%
PROBABLE
SOURCE
SYMBOL
PK+I
._ \qK-,
' q2
P_
PO qo
Fig. 2. A portion of an efficient code tree, In which the longest
codeword has length _ K+ 1..Do Is the least source probability.
PK
/.// _(FK+ 1)/FK+2-P
_K__/_K+_
P_ F2/FK+2
Fig. 3. A Huffman code tree for the source In (8). Its smallest
probability Is p, where p _ I/FK÷ 2, and Its longest codeword
length Is K.
193


Maximal codeword lengths in Huffman codes

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server