The capacity of channels with block memory is investigated. It is shown that, when the problem is modeled as a game-theoretic problem, the optimum coding and noise distributions when block memory is permitted are independent from symbol to symbol within a block. Optimal jamming strategies are also independent from symbol to symbol within a block

McEliece, Robert J.

Stark, Wayne E.

English

Caltech Authors - Main

322 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 34, NO. 2, MARCH 1988 
On the Capacity of Channels with Block Memory 
WAYNE E. STARK, MEMBER, IEEE, AND 
ROBERT J. MCELIECE, FELLOW, IEEE 
Abstract -The capacity of channels with block memory is investigated. 
It is  shown that, when modeled as a game theoretic problem, the optimum 
coding and noise distributions when block memory is permitted are inde- 
pendent from symbol to symbol within a block. 
I. INTRODUCTION 
We consider communication channels in which the memory of 
the channel lasts for a fixed finite time duration; that is, the 
channel is a block memoryless channel in the sense that, if we 
consider blocks of fixed length as single symbols in some much 
larger alphabet, then the channel is memoryless. One motivation 
for considering such channels is a spread-spectrum frequency- 
hopped (SSFH) communication system in which the transmitted 
signal occupies a certain frequency band for a fixed time dura- 
tion during which a certain number of symbols are transmitted, 
whereupon the carrier frequency changes. Interference during 
one carrier dwell usually is assumed to be independent of the 
interference during any other. The types of interference for which 
this is true are partial-band jamming (tone or noise), fading 
(self-jamming), and certain kinds of multiple-access interference. 
Another channel which exhibits similar behavior is that of a 
sequence of computer memory chips each of which stores a finite 
number of bits. Bits within a single chip may have highly corre- 
lated errors (i.e., row errors, column errors, or even whole chips 
being in error); however, bit errors from one chip to the next are 
independent. 
There are several problems concerning channels with block 
interference. One of these is to determine optimal coding strate- 
gies and the largest rate that information can be reliably trans- 
mitted. In a spread-spectrum system subject to jamming, the 
worst-case jamming strategy subject to constraints on the first- 
order distribution of the jammer only is of considerable impor- 
tance. We formulate these as game theory problems in which 
both the coder and jammer have constraints placed on their 
first-order distributions. The payoff function is the mutual infor- 
mation between the input and output of the channel. The main 
result is that, provided the interference (jammer) is independent 
of the transmitted signal (repeat-back jamming is not allowed), 
the optimal distribution of symbols within a block for the coder 
and jammer are independent from symbol-to-symbol. In Section 
I1 we formulate and solve the game theory problem. In Section 
I11 we give several examples. We conclude in Section IV with 
some discussion about the results and other applications. 
11. GAME THEORY FORMULATION 
The communication channel we consider has input alphabet A 
and output alphabet B. Player I, called the coder, wishes to 
communicate information through the channel reliably with the 
largest possible rate. Player 11, called the jammer, wants to 
minimize the rate at which information can be transmitted 
through the channel. The channel is described by specifying two 
random variables, X and Y. The random variable X is the input 
to the channel from the coder and the random variable Y is the 
Manuscript received June 12, 1984; revised February 10, 1987. This work 
was supported in part by the National Science Foundation under Grant 
ECS-8307150 and in part by the Air Force Office of Scientific Research under 
Grant AFOSR-83-0296. 
W.  E. Stark is with the Department of Electrical Engineering and Computer 
Science at the University of Michigan, Ann Arbor, MI 48109. 
R. J. McEliece is with the Department of Electrical Engineering at the 
California Institute of Technology, Pasadena, CA 91125. 
IEEE Log Number 8819894. 
output of the channel. The coder's strategies are distributions F, 
on the random variable X, while the jammer's strategies are the 
distributions C,(yla) on the output of the channel when X- a 
is the input. The jammer thus chooses the conditional probabili- 
ties of the output given the input while the coder chooses the 
distribution of the input. We restrict the set of distributions the 
players can have as follows. The allowable distributions (strate- 
gies) for the coder are given by a set S (Fx E S ) .  The collection 
{ G,(yla): a E A}, which we denote by C y ,  is required to be in a 
set T of allowable channels. The payoff function +(F,, G,) for 
this game is taken to be the mutual information I( 2, Y) between 
the input to the channel X and the output of the channel Y. The 
objective of player I is to choose F, E S to make $(F,, G,) as 
large as possible. Player I1 chooses G ,  E T to minimize 
+(Fx, C y ) .  Thus associated with the game are two programs. 
Program I (Coder's Program): 
Program II  (Jammer's Program): 
C"= inf sup + ( F x , G y ) .  
GY E T F, E S 
A strategy F,* E S such that inf { +( F,*, CY) :  Gy E T }  = C' 
is called an optimum strategy for the coder. S d a r l y ,  if 
sup{+(F,,G;C.): F , € S } = C "  then G;  is called an optimum 
strategy for the jammer. 
It is clear from the above programs that C' s C", and it is easy 
to give examples where C'< C". However, since $ is convex n 
(concave) in F, and convex U (convex) in G ,  [l, Theorems 1.6, 
1.71 if S and T are compact convex sets, then C'= C" [2]. This 
equality is equivalent to the existence of a saddlepoint, i.e., a pair 
of strategies F,* E S,  Gjl: E T such that 
+ ( F x , G ; )  s + ( F x , G Y )  s + ( F , + , G y ) ,  F x € S , G y ~ T .  
(1) 
If (1) holds, then F,* and C,* are optimal strategies for the coder 
and jammer, respectively. This game theory formulation was 
considered by Dobrushin [3] and Blachman [4]. While we do not 
have any physical justification for assuming the set of possible 
strategies is compact, time-sharing between two strategies is a 
physical justification for the set of possible strategies to be 
convex. 
We generalize this game theory formulation by allowing the 
players to adopt m-dimensional strategies (i.e., nonmemoryless 
strategies). We extend the definition of admissible strategies to 
higher dimensions by using the notion of the mixture of a set of 
distribution functions. Let the m-dimensional distribution 
Fim)(x) ,  X = ( X , ;  . -, Xm), z = (x,, x2; . e ,  x,) have marginal 
distribution Fx(x )=F~m) (oo ,oo , . . . ,oo ,x ,oo  -.*,GO) with the 
ith component'being x. We say Fim' E S(') if the uniform 
mixture of the marginals is in S: 
1 m  
F:"' E s'"' if - C F&( x) E S. (2) 
1 - 1  
The admissible strategies for the jammer are defined similarly. 
We say GL"' = { GLm)( yla): a E A m }  E T'"' where &"'( yla) is 
the m-dimensional conditional distribution of the output of the 
channel given the input X = a ,  if the uniform mixture of the 
conditional marginals G,,(yln) is in T: 
cl"' E T'"' if Gy E T (3) 
where 
0018-9448/S8/0300-0322$01.00 01988 IEEE 
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 34, NO. 2, MARCH 1988 323 
is the collection of conditional distributions with uniform mix- 
ture of the marginals CY ( y l a )  of G$"'( yla). We note here that 
we have restricted the sirategies to those with no "intersymbol 
interference;" i.e., previous inputs are not allowed to affect 
current outputs. For these generalized strategies we have the 
following programs. 
Program I ,  (Coder's Program): 
Program II, (Jammer's Program): 
where the payoff function is now (p( F r ' ,  G(ym') = ( l /m)I(X Y). 
We have the following result concemg C; and C,,t,'. 
Theorem I: C; = C' and C,,t,' = C" for all m 2 1. 
Pro08 First we prove C; =C'.  Let FX be an admissible 
strategy; i.e., Fx ES, and let GLm) ET'") be an admissible 
strategy for the jammer. Then if X = (XI,. . ., X,) is a random 
vector consisting of m independent identically distributed (i.i.d.) 
copies of X, we have 
+(F;"',G&"') 2; + ( F x , G y , )  [l,Theorem1.8] 
2 + ( F , , G Y )  [ 1, Theorem 1.71 
1 ,  
I - 1  
l-IyLl F,* (x,)). Thus the interference (which is independent of the 
transmitted signal) that minimizes the mutual information is 
independent from symbol to symbol. 
111. EXAMPLES 
Consider the channel with A = B = (0,1,- . ., M - 1) and let T 
be the set of channels with error probability per symbol less than 
E, 0 I E 11, and S be all distributions on A. Then a result of 
Dobrushin [3] is that 
log M +(1- E)log(l- E) + Clog€ - Clog( M - l ) ,  
E 11 -(l/M) 
0, ~ > l - ( l / M )  
(4) 
C' = C" = 
Here C' and C" are measured in bits per channel use and all 
logarithms have base 2. The optimal distribution F,*(x) is the 
uniform distribution on (0,1,. . ., M - 1) and the optimal chan- 
nel G * ( y l x )  satisfies 
1 
[ c / (M- l ) ,  y f x ,  c s l - -  M 
1 
M 
G * ( y l x )  = 1 - c ,  y = x ,  E < l - - .  (5) 
1 
€21-- Ll M 
where G, is the uniform mixture of the conditional &st&utions 
G,. Since C y  E T  we have 
Here we generalize this game to the m-dimensional case and 
apply the theorem. For the generalized strategies we use the 
channel m times to transmit m symbols. Let c, be the error 
probability of the ith channel. Then T'"' is the set of channels inf +( F:"),G:~') 2 inf + ( F ~ , c ~ ) .  
c;m) E T ( m )  G , € T  with 
Now since the above holds when ( X I , .  . . , X,) are i.i.d. we have 
that 
so that CA 2 C'. Now let Fi") E S'"' and C,(yla> be arbitrary. 
Then if GL"' is the m-dimensional distribution of Y =  
(q , . *  e ,  Y,) given X = a with conditionally independent, then 
1 "  
m 1-1 
+( ~i"', cLrn)) I - +( F ~ ,  , c,) [I, Theorem 1.91 
S + ( F X , G Y )  [ 1, Theorem 1.61 
where Fx is the uniform mixture of Fi"'; i.e., 
i m  
Since the above is true for the case when 3, Y,; . ., Y, are 
conditionally independent we have that 
1 ,  
- EiIE. 
m1-1 
Now since Fx E S we have and 
Also S'"' is the set of distributions on A". By Theorem 1 
C; = C' and C; = C" and is given in (4). The optimal strategies 
are memoryless with marginals F,* ( x )  and GP ( y ( x )  given above. 
A conclusion one can draw is that if the auerage error probability 
is less than E ,  then with memoryless encoding I ( X  Y) 2 C' = C" 
with equality for the optimal strategy given above. Notice that 
the worst possible channel (for fixed average error probability) is 
thus a symmetric channel. This example can be used to model a 
faded channel. The type of fading determines the fist-order 
statistics, i.e., E. For a slowly faded channel the statistics are such 
that 
c, = E ,  i =1,2,..., m .  
However, the lowest mutual information is achieved by fast 
fading, i.e., when the channel fading variable is independent from 
symbol to symbol witbh a block. 
As a second example, consider A = B = R,  the real line. Let 
S =  ( R  F x ( x ) :  / x 2 d F x ( x )  I E ( 6 )  
324 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 34, NO. 2, MARCH 1988 
with F*(x) = @(x/@) and G:(ylx) = @ ( ( y  - x)/fi)  where 
@( u )  is given by 
1 u  
@( u )  = -/ dx. ( 9) 6 - m  
Again we generalize this game to allow for m-dimensional distri- 
butions. The sets S ( m )  and T(m) are given by 
where a‘ denotes the transpose of the vector a, and 
.( y - ~ ) ’ d C b ~ ) (  Y ~ X )  I N ,  x E A” . i 
By Theorem 1, C; = C; = C‘= C“ and the optimal distributions 
are memoryless with marginal distributions being Gaussian. We 
note here that Theorem 1 is the discrete time analog of the result 
for continuous time channels that white Gaussian noise is the 
optimal jamming and coding strategy. 
IV. CONCLUSION 
We have formulated block memoryless channels as a game 
theory problem and have shown that optimum coding strategies 
are independent from symbol to symbol within a block and 
optimal jamming strategies are also independent from symbol to 
symbol within a block. Many important questions still need to be 
answered concerning the optimum jamming distribution for each 
symbol within a block and robust decoding strategies. 
REFERENCES 
[l]  
[2] 
[3] 
[4] 
R. J. McEliece, The TheoT of Information and Coding. Reading, MA: 
Addison-Wesley, 1977. 
J. Stoer and C. Witzgall, Convexity and Optimization in Finite Dimensions 
I .  New York: Springer-Verlag. 1970. 
R. L. Dobrushin, “Optimum information transmission through a channel 
with unknown parameters,” Radio Eng. Electron., vol. 4, no. 12, 1959. 
N. M. Blachman, “Communication as a game,” in Proc. Wescon 1957 
Conf. Rec., Aug. 1957, pp. 61-66. 
Almost Asymptotically Optimal Flag Encoding 
of the Integers 
MUZHONG WANG 
Abstract-A simple prefix-free encoding scheme for the positive in- 
tegers is proposed in which a flag of f zeroes is used to mark the end of a 
codeword, and bit stuffing is used to prevent premature appearance of the 
flag in the conventional binary coding of the integers. It is shown that this 
coding scheme is universal in the sense defined by Mas, and that for large 
f its asymptotic efficiency is virtually 1. 
I. INTRODUCTION 
Elias considered the problem of finding prefix-free encodings 
for the positive integers and introduced the notions of universal- 
ity and asymptotic optimality to characterize codings [l]. A 
D-ary encoding d of the positive integers, N +  = {1,2, . . . }, is a 
function that assigns a different nonempty string d( n )  of letters 
from a given alphabet of D letters to each n in N + .  The 
Manuscript received December 9, 1986; revised May 18, 1987. 
The author is with the Institute of Signal and Information Processing, Swiss 
IEEE Log Number 8820327. 
Federal Institute of Technology, 8092 Zurich, Switzerland. 
encoding B is prefix-free if no codeword B(n)  is the beginning 
of another codeword. Hereafter, we consider only binary encod- 
ing and take {O,l} as the coding alphabet. Note that the conven- 
tional binary coding D of N +  is not prefix-free because, for 
example, 9(2) = 10 is a prefix of D(4) = 100. 
Let d be a given binary prefix-free encoding of N’, and let 
L ( n )  be the length of the codeword B(n) .  Hereafter, let P be 
any probability distribution of N +  (i.e., P(n) 2 0, all n E N + ,  
and P(1) + P(2) + P(3) + . . = 1) that satisfies 
P( n ) 2 P( n + 1) , all n E N’ . (1) 
Let H( P )  be the entropy of P in bits, and let Ep ( L )  denote the 
expected codeword length for I when P is the probability 
distribution. m a s  [l] has defined B to be a universal encoding of 
N +  if a finite number p exists such that 
for all P, and to be asymptotically optimal if it is universal and 
(3) 
where R is some real-valued function such that 
lim R ( x ) = l .  ( 4) 
X ‘ W  
Actually, Elias [l]  defined these notions only for encodings I 
with the “minimal” property that 
L ( n ) s L ( n + l ) ,  n E N + ,  (5) 
which property minimizes Ep( L )  for the codeword set because of 
(l), but the definitions are useful for any 1. 
We now introduce another notion of efficiency for prefix-free 
encodings of N’. We first note that the conventional binary 
coding 9 of the integers gives a codeword of length [log, n 1 + 1 = 
[log, ( n  + 1)1 to n E N’, where 1 . I([  . I )  denotes the largest 
integer not greater than (the smallest integer not less than) the 
enclosed number. It seems natural then to define the asymptotic 
efficiency of a prefix-free encoding of N’ by 
log, n + 1 r =  lim sup- 
n A m  L ( n )  ’ 
Note that 0 I r I 1 and that P is not involved in the definition 
In the next section, we introduce a flag encoding scheme for 
N +  and develop simple but tight upper and lower bounds for 
L( n). The upper bound is used in the following section to show 
that r = 1 when the flag length is large and that the encoding 
then is “almost asymptotically optimal” in a sense that we will 
define there. 
11. THE FLAG ENCODING SCHEME FOR THE INTEGERS 
of r. 
The idea of our flag encoding scheme for N +  is as follows. We 
choose the flag length f to be some positive integer at least 2. 
Note that the codeword a( n) of the conventional binary coding 
of a positive integer always begins with a 1. Reversing D( n), we 
get a string that ends with a 1. All occurrences of the flag Of, a 
string of f zeroes, are removed from this string by stuffing a 1 
after each Occurrence of Of- l .  Finally, we add the flag Of to the 
end of the string to form the codeword &’(TI). 
Assume that integer n to be encoded is represented in its 
binary conventional form D( n). We have the following encoding 
and decoding algorithms: 
Encoding algorithm B (n): 
Step 1: Set d f (n f  equal to (bobl .. . b,), the reverse of 
9 ( n )  = (b, * * * b,b,). (Note that b, =1.) 
0018-9448/88/0300-0324$01.00 01988 IEEE 


On the capacity of channels with block memory

Communication as a game,” in

Convexity and Optimization in Finite Dimensions I .

Elias considered the problem of finding prefix-free encodings for the positive integers and introduced the notions of universality and asymptotic optimality to characterize codings [l]. A D-ary encoding d of the positive integers,

Optimum information transmission through a channel with unknown parameters,”

The TheoT of Information and Coding.

https://authors.library.caltech.edu/6824/1/STAieeetit88.pdf

On the capacity of channels with block memory

Abstract

Similar works

Full text

Available Versions

Caltech Authors - Main

Caltech Authors - Main