A new concept of directed-divergence function of type β is introduced in this paper. This concept is used in obtaining a directed-divergence of type β which generalizes Kullback's directed-divergence and has a relation with RÉnyi's information-gain of order β. This relation can be used to give another characterization of information-gain of order β. A characterization theorem for the directed-divergence of type β is proved with the help of a functional equation

Kannappan, Pl.

Rathie, P.N.

Elsevier - Publisher Connector 

A directed-divergence function of type β 

INFORMATION AND CONTROL 20, 38--45 (!972) 
A Directed-Divergence Function of Type 13* 
P. N. RATHIE AND PL. KANNAPPAN 
Faculty of Mathematics, University of Waterloo, Waterloo, Ontario 
A new concept of directed-divergence function of type fi is introduced in 
this paper• This concept is used in obtaining a directed-divergence of type fl 
which generalizes Kullback's directed-divergence and has a relation with 
R6nyi's information-gain of order ft. This relation can be used to give another 
characterization of information-gain of order ft. A characterization theorem 
for the directed-divergence of type fi is proved with the help of a functional 
equation. 
1. INTRODUCTION 
Let P = (P l ,  P~ ,..., Pn), P/ >~ 0 with ~i~lP i  = 1 and Q = (ql, q~ ,-.., q,), 
q/>~ 0 with ~i=1 qi = 1 be two finite discrete probability distributions. Then 
Kullback's directed-divergence (Kullback, 1959) for the two probability 
distributions P and Q is defined as, 
11 (Pl ..... P") = ~, pilog2(pN71). (1.1) 
ql , '" ,  qn i=1 
Also, R~nyi's information-gain of order/3 (R~nyi, 1961) is given by 
n 
F(P IQ)=(/3-1) -11og2 (~p/~ql-B),  for /3 =A 1. (1.2) 
" /=1"  
In this paper, we introduce another generalization of (1.I) called the 
directed-divergence of type/3: 
In B (Pl  , '", !)n I ~-- (i=~1 Bql-B )(2/3-1 ~ql q#  Pi i - -  1 --  1) -1, for /3 V + 1, (1.3) 
which reduces to (1.1) when/3 -+ 1. The quantity In t~ is characterized by a set 
* This work is partially supported by a Department of University Affairs (Ontario, 
Canada) grant. 
38 
© 1972 by Academic Press, Inc. 
TYPE /3 DIRECTED-DIVERGENCE FUNCTION 39 
of three postulates in the next section. In the process of characterization f I~ e 
a functional equation is obtained which may be regarded as the fundamental 
functional equation defining a new directed-divergence function of type 18 
for (1.3). 
Clearly from (1.2) and (1.3), we get the following relation involving the 
directed-invergence of type fi and the information-gain of order/3: 
" '"  P~] = (2 ~-1 -- 1)-1(2(~-1)I~ (el°) - -  1), InB(q pl ..... q# for /3 =~ 1. (1.4) 
The results of this paper also give new characterization f R6nyi's infor- 
mation of type/3 by means of a functional equation. 
Very recently, (1.1) was characterized (Campbell, 1970) for arbitrary 
probability space by a set of six postulates. Rathie (1970) has recently 
characterized (1.3) by means of a functional inequality for n > 2. 
The case fi = 1 giving new characterization for (1.1) by means of a 
functional equation will be discussed by us elsewhere. 
2. CHARACTERIZATION THEOREM 
We prove the following theorem regarding the directed-divergence of type fi, 
that is (1.3). 
THEOREM. I f  Kn , n = 2, 3,... satisfies the following relations, 
(i) /~, (Pl ,..., P.) 
\ql ,'", qn / 
: Kn_I(Pl @-P2'Pa"'"Pn) 
ql + q~ , % ,..., q~ 
~- (Pl + P2)~(ql -~- q2)l-~K~ \ ql/(qa + q2), q2/(q~ + q2) ] 
with Pl @ P2 , ql + q~ > O; 
(ii) Ka(P"P"P')isasymmetricfunctionofitsvariables IPil, i=l,2,3; 
q~, q2, % qi 
(iii) K,(1½: 0½)= 1, 
4O 
then 
RATHIE AND KANNAPPAN 
\ 
K,  ~ I, fl = \~1 ~*~'~ql"-z~ _ 1)(2 B-1 -- 1) -1, 
Proof. 
for /3 4= 1. (1.3) 
We define 
f (x ,y)  ~-K2(  x' 1 - -x )  x, yE[O, II. (2.1) 
y, 1 - -y  ' 
First we will prove the following lemma giving rise to a functional equation 
related to f given by (2.1). 
LElUMA 1. The function f : I  × I---~R given by (2.1) where I = [0, 1] 
and R, the real numbers, satisfies the functional equation 
f (x ,y)  + (1 -- x)~(1 _y) l -~f  1 x ' l  2 
=f (u ,v )  +(1 - -u )~(1- -v ) l -~f  T -  u ' l  - -v  ' 
for all x, y, u, v ~ [0, 1) with x + u, y + v e I and the boundary conditions 
f(0, 0) =f (1 ,  1) = 0 (2.3) 
and 
/(0, ~) - - f ( i ,  ~) -- 1. 
Proof. Taking n = 3 in (i), (i) becomes 
(2.4) 
K3 (P a ' p~ ' pz] 
ql , q2, qzI 
= 1,:3 (P~ + p~' P~ + (pl + p2l~(ql + q#-~K~ p~ + p2 ' pl + p~ 
ql + q2 , q3 / ql q~ 
ql + q2 ' ql + q~ 
(2.5) 
fo rp l+p2,q l+q2>O.  
TYPE fl DIRECTED-DIVERGENCE FUNCTION 41 
Interchanging Pl and P2 and ql and q~ in (2.5) and using (ii), (2.5) gives 
K2 Pl +P2 ' Pl +P2 = Ks Pl +P2 ' Pl +P2 (2.6) 
ql q2 q2 ql ' 
q l+q~ ' q l+q~ q l+q2 ' qx+q~ 
for P l ,  P~, ql, q2 ~ I with Pl + P~, ql + qz ~ (0, 1]. 
Hence (2.1) and (2.6) yield 
f P l+Pz  ' q l+qz  P l+P2 ' q l - /q~ 
which is equivalent to 
f (x ,  y) = f(1 --  x, 1 --  y) for all x, y ~ I. (2.7) 
Rewriting (ii), we have 
Ka (x, 1 - -x - -u ,u )  (u, l - -  x - -  u, x) (2.8) 
y,  1 - -  y v, = Ka v, 1 y v, y l '  
wherex, y, u, v, l - -  x -- u, l - -  y - -  v e I. 
Now (2.8) with the help of (i) for n = 3 and (2.1) gives 
(x 
f (1  - u, 1 - ~) + (1 - u ) f f l  - ~) l -~f  1 - u '  1 - ~ 
f (1  - x, 1 -y )  + (1 - x)~(1 -y ) l -~f  ~ _ x '  1 - y~ 
for x ,y ,  u, v~ [0, 1) with x + u ,y  + v eL  
In view of (2.7), (2.9) takes the form 
(x 
f (u ,v )  +(1 - -u )~(1- -v ) l - ' f  1 - -u ' l  - -v  
/ U f) 
= f (x ,  y)  + (1 - -  x)~(l - -  y ) l - ' f  [ 1 - x '  l - -  y ) '  
which is precisely (2.2). 
Also (ii) on using (i) gives 
(1,0, 0 
~ ,,/ = 2fO, 1), \1, O, u/  
42 RATHIE AND KANNAPPAN 
and 
giving, 
1'00) ~- f(1, 1) +f (0 ,  0), 
f(1, 1) =f(O, 0). (2.10) 
Putting x = y = ½, u = v = 0 in (2.2), we have, 
f(1, 1) = ½f(O, 0). (2.11) 
Thus, from (2.10) and (2.11), we get 
f(1, 1) = f(0, 0) = 0, 
which is precisely (2.3). 
Furthermore from (iii), (2.1) and (2.7) for x = 0 and y = ½, result 
f(1, ½) =f (0 ,  ½) = 1, 
which is precisely (2.4). 
Thus Lemma 1 is proved. 
DEFINITION 1. A function f : I  × I---~R satisfying the functional 
equation (2.2) and the boundary conditions f(0, 0) = f(1, 1) and (2.4) is 
called a directed-divergence function of type/3. 
Now we will describe all the solutions of the functional equation (2.2) under 
the boundary conditions (2.3) and (2.4). 
LEMMA 2. Every solution f of (2.2) satisfying the additional conditions (2.3) 
and (2.4) is given by 
f(x, y) = [xBy 1-B + (1 -- x)e(1 -- y)l-e _ 1](2~-1 _ 1)-1, (2.12) 
for x, y e I, in which the notation 0 ~ = 0 (~ =/= O) is used. 
Remark 1. In order to obtain the solution of (1.3) in the desired form 
(2.12) for all x,y ~1, the convention 0~ = 0 (~ :/: 0) is used, but nowhere 
in the proof of the theorem, 0~ ~ 0 is used; 
Proof. 
s = 1 - -  y in (2.2), reduce (2.2) to 
f (1  - -  r, 1 - -  s) + rBsl-ef(p, q) 
( l l - - r  l - - s )  (2.13) = f (pr ,  qs) + (1 - -  pr)~(1 - -  qs)l-~f - -  p r '  1 - -  qs ' 
for a l lp ,  q E/ ,  r, se  (0, 1] such that pr  # 1 and qs ~ 1. 
Let r = 1, s = ½ in (2.13), then we have 
f (0 ,  ½) + 2~-l f(p,  q) = f (p ,  q/2) + (1 --p)~(1 - -  q/2)~-~f (0, ~) ,  
(2.14) 
fo rp  e [0, 1), qe[0 ,  1]. 
Taking q = 0 in (2.14) and using (2.4) we get 
f (p ,  0) = [(1 - -  p)~ - -  1](2 ~-1 - -  1) -a, for p e [0, 1). (2.15) 
Put t ingp  = 1, q = 0 in (2.13) we get 
f (1 - -  r, 1 - -  s) + rest-Of(I, O) = f ( r ,  0) + (1 - -  r)Bf(1, 1 - -  s), 
for re (0 ,  1), se (0 ,  1]. 
(2.16) 
Now (2.15) and (2.16) for s - -  1 give 
f (1,  0) = (1 - -  2~-1) -1. (2.17) 
Again taking r = 1, q = 0 in (2.13), we get 
f (O,  1 - -  s) + sl -Bf(p,  O) = f (p ,  0) + (1 - -  p)~ f(O, 1 - -  s), (2.18) 
which on using (2.15) gives us 
f (0 ,  1 - -  s) = (s a-~ - -  1)(2 B-1 - -  1) -1, for s e (0, 1]. (2.19) 
Also fo rp  = 0, q = 1, (2.13) becomes 
f (1  - -  r, 1 - -  s) + r~sa-~f(O, 1) =f (0 ,  s) + (1 - -  s) l -~f(1 - -  r, 1), 
for re (0 ,  1], se (0 ,  1). 
(2.20) 
Now (2.20) with r = 1 and (2.19) give 
f(O, 1 )= (1 2t~-1) -1. (2.21) 
TYPE fl DIRECTED-DIVERGENCE FUNCTION 43 
Substitution of p =u/ (1 - -x ) ,  q =v/ (1 - -y ) ,  r = 1 - -x  and 
44 RATHIE AND KANNAPPAN 
Further (2.16) and (2.20) on using (2.17) and (2.21) give 
f(r, 0) + (1 --  r)~f(1, 1 --  s) = f (0,  s) + (1 --  s)~-Bf(1 --  r, 1), 
for r, se(O, 1). 
(2.22) 
From (2.22) for s = ½-, (2.15) and (2.4), we have 
f (1 - - r ,  1) = [(1 - - r )  ~ --  1](2 ~-~-  1) -a, 
Hence (2.22), (2.23), (2.15) and (2.19) give 
for re (0 ,  1). 
(2.23) 
f (1,  1 - -  s) = [(1 --  s) x-e --  1](2 ~-i - -  1) -1, for s E (0, 1). 
(2.24) 
Thus  (2.15), (2.17), (2.19), (2.21), (2.23) and (2.24) give f(x, 0), f(O, x), 
/ (1 ,  x),f(x, 1) for xe  [0, 1]. 
Also (2.20), (2.21), (2.19) and (2.23) give 
f ( l  - -  r, 1 --  s) =[ r ' s  i-B -~ (1 - -  r)~(1 --  s) i-~ --  1](2 ~-1 --  1) -i, 
for r, se(O, 1). 
(2.25) 
It is easy to see from (2.15), (2.17), (2.19), (2.21), (2.23), (2.24) and (2.25) 
that f  is given by (2.12) with the notation 0 ~ = 0 (~ v ~ 0) and that (2.7) is 
true for all x, y e [0, 1]. This completes the proof of Lemma 2. 
Now, we return to the proof of the theorem. By successive applications 
of (i) and the use of (2.1) give 
Kn ~ In s (Px ,'", Pn) = ~ riSsl_~f ( P~i q~) 
ql ,..., qn i=~ 
(2.26) 
where r i=p i+ ' "+p i ,  s i=q i+ ' "+q~ 
r .  = sn = 1. 
Thus  (2.26) and (2.12) prove the theorem. 
for i=  1,2, . . . ,n with 
DEFINITION 2. I f f  is a directed-divergence function of type/3 (fl va 1) as 
given by Definition 1, then In s given by (2.26) [also (1.3)] is called the 
directed-divergence of type/3 (/3 @ 1). 
Several properties like null- information, non-negativity, symmetry, 
TYPE fl DIRECTED-DIVERGENCE FUNCTION 45 
expansibil ity, etc., can be easily derived for In B. We ment ion below a strong 
non-addit iv ity property for In~. 
I~ ,  (P lPn  , PaPzi ,..., P lPmi  . . . . . . .  , P ,P i ,  , P ,P2  . . . . .  , P ,P~, ]  
\ q iqn ,  qN2i ,..., q iq~l  ,. . . . . .  , q~qi~,  q~q2n ,..., q~q~n / 
. :~ 1-~ I ~ [Pal ,..., P~q ~- i ~ (Px ,..., P , ]  + ,~=a p '  q' ~ (2.27) 
\q l  ,..., q~: = \ql~ ,..., q , , J '  
m d m where ~=1 Pji = 1 an ~=x qJi ----- 1 for all i = 1,..., n. 
An interesting special case of (2.27) is given below. 
I~mn ( p iP1 '  p lP2  ..... P lPm '" . . . . .  , P ,P~ , P ,P2  .... , P ,P ,*  ] 
\q~Q~ , q~Q2 . . . . .  qaQ . . . . . . . . .  qnQ~ , q.Q2 ..... q.Qm: 
ql ..... q,~ QI ..... Q,~! 
+ C ql ,..., q,,: Q1 .... , Q, .  ' 
q74 
where Y,~=l P:  = 1 and ~j=x QJ = 1. 
ReCEWED: January 20, 1971 ; REWSED: August 4, 1971 
REFERENCES 
CAMPBELL, L. L. (1970), Characterization f entropy in arbitrary probability spaces, 
pp. 1-26. Queen's University Preprint No. 32, Kingston. 
KULLBACK, S. (1959), "Information Theory and Statistics," John Wiley and Sons, 
New York. 
RATHIE, P. N. (1970), Some characterization theorems for generalized measures of 
uncertainty and information, Meeting on Information Measures, University of 
Waterloo, Waterloo, April 10-14. 
t{ENYI, A. (1961), On measures of entropy and information, Proc. Fourth Berkeley 
Symposium Math.  Statist. and Probability 1, 547-561. 


file:///data/remote/core/dit/data/elsevier/pdf/3fb/aHR0cDovL2FwaS5lbHNldmllci5jb20vY29udGVudC9hcnRpY2xlL3BpaS9zMDAxOTk5NTg3MjkwMjYwNA==.pdf

A directed-divergence function of type β

Abstract

Similar works

Full text

Available Versions

Elsevier - Publisher Connector

Elsevier - Publisher Connector