Some new inequalities for the relative entropy and applications are given

Dragomir, Sever S

English

Victoria University Eprints Repository

SOME INEQUALITIES FOR THE RELATIVE ENTROPY ANDAPPLICATIONSS.S. DRAGOMIRAbstract. Some new inequalities for the relative entropy and applicationsare given.1. IntroductionThe relative entropy is a measure of the distance between two distributions. Instatistics, it arises as an expected logarithm of the likelihood ratio. The relativeentropy D(p‖q) is a measure of the inefficiency of assuming that the distributionis q when the true distribution is p. For example, if we knew the true distributionof the random variable, then we could construct a code with average descriptionlength H(p). If, instead, we used the code for a distribution q, we would needH(p) +D(p‖q) bits on the average to describe the random variable [1, p. 18].In what follows, unless we will specify, log will denote logb where b is a givenbase greater than 1.Definition 1. The relative entropy or Kullback-Leibler distance between two prob-ability mass functions p(x) and q(x) is defined byD(p‖q) : =∑x∈Xp(x) log(p(x)q(x))(1.1)= Ep log(p(X)q(X)).In the above definition, we use the convention (based on continuity arguments)that 0 log(0q)= 0 and p log(p0)=∞.It is well-known that relative entropy is always non-negative and is zero if andonly if p = q. However, it is not a true distance between distributions since it isnot symmetric and does not satisfy the triangle inequality.The following theorem is of fundamental importance [1, p. 26].Theorem 1. (Information Inequality) Let p(x), q(x) ∈ X , be two probability massfunctions. ThenD(p‖q) ≥ 0(1.2)with equality if and only ifp(x) = q(x) for all x ∈ X .(1.3)Date: January , 1998.1991 Mathematics Subject Classification. Primary 26D15; Secondary 94Xxx.Key words and phrases. Relative Entropy, Shannon Entropy, Mutual Information.12 S.S. DRAGOMIRActually, the inequality (1.2) can be improved as follows (see , [1, p. 300]):Theorem 2. Let p, q be as above. ThenD(p‖q) ≥ 12 ln b‖p− q‖21(1.4)where ‖p− q‖1 =∑x∈X|p(x)− q(x)| is the usual 1-norm of p− q. The equality holdsiff p = q.We remark that the argument of (1.4) is not based on the convexity of the map− log(.).To evaluate the relative entropy D(p‖q) it would be interesting to establish someupper bounds.Before we do this, let us recall some other important concepts in InformationTheory.We introduce mutual information, which is a measure of the amount of informa-tion that one random variable contains about another random variable. It is thereduction in the uncertainty of one random variable due to the knowledge of theother [1, p. 18].Definition 2. Consider two random variables X and Y with a joint probabilitymass function p(x, y) and marginal probability mass function p(x) and q(y). Themutual information is the relative entropy between the joint distribution and theproduct distribution, i.e.I(X;Y ) =∑x∈X∑y∈Yp(x, y) log(p(x, y)p(x)q(y))(1.5)= D (p(x, y) ‖ p(x)q(y))= Ep(x,y) log(p(X,Y )p(X)q(Y )).The following corollary of Theorem 1 holds [1, p. 27].Corollary 1. (Non-negativity of mutual information): For any two random vari-ables, X, Y we haveI(X;Y ) ≥ 0(1.6)with equality if and only if X and Y are independent.An improvement of this result via Theorem 2 is as followsCorollary 2. For any two random variables, X, Y we haveI(X;Y ) ≥ 12 ln b∑x∈X∑y∈Y|p(x, y)− p(x)q(y)|2 ≥ 0(1.7)with equality if and only if X and Y are independent.Now, let u(x) = 1|X | be the uniform probability mass function on X and let p(x)be the probability mass function for X.INEQUALITIES FOR THE RELATIVE ENTROPY 3It is well-known that [1, p. 27]D(p‖u) =∑x∈Xp(x) logp(x)u(x)(1.8)= log |X | −H(X).The following corollary of Theorem 1 is important [1, p. 27].Corollary 3. Let X be a random variable and |X | denotes the number of elementsin the range of X. ThenH(X) ≤ log |X |(1.9)with equality if and only if X has a uniform distribution over X .Using Theorem 2 we also can stateCorollary 4. Let X be as above. Thenlog |X | −H(X) ≥ 12 ln b[∑x∈X∣∣∣∣p(x)− 1|X |∣∣∣∣]2≥ 0.(1.10)The equality holds iff p is uniformly distributed on X .In the recent paper [2], the authors proved between other the following upperbound for the relative entropy and employed it in Coding Theory in connection toNoiseless Coding Theorem:Theorem 3. Under the above assumptions for p (x) and q (x) we have the inequal-ity1ln b[∑x∈Xp2(x)q(x)− 1]≥ D(p‖q)(1.11)with equality if and only if p (x) = q (x) for all x ∈ X .The following upper bound for the mutual information holdsCorollary 5. For any two random variables, X, Y we have1ln b∑x∈X∑y∈Yp2(x, y)p(x)q(y)− 1 ≥ I(X;Y )with equality iff X and Y are independent.Finally, we note that the following upper bound for the difference log |X |−H(X)is validCorollary 6. We have1ln b[|X |∑x∈Xp2(x)− 1]≥ log |X | −H(X)with equality if and only if p is uniformly distributed on X .The main aim of this paper is to point out some refinements of (1.2) and (1.11)and to apply them for the entropy mapping and for mutual information .4 S.S. DRAGOMIR2. An Inequality for Relative EntropyThe following result holdsTheorem 4. Let p(x), q(x), x ∈ X be two probability mass functions. Then1ln b[∑x∈Xp2(x)q(x)− 1]≥ 1ln b{∑x∈X[(p (x)q(x))p(x)− 1]}(2.1)≥ D(p‖q) ≥ 1ln b{∑x∈X[1−(q (x)p(x))p(x)]}≥ 0.The equality holds in all inequalities if and only if p (x) = q (x) for all x ∈ X .Proof. We know that for every differentiable strictly convex mapping f : I → R,we have the double inequalityf ′ (x) (x− y) ≥ f (x)− f (y) ≥ f ′ (y) (x− y)for all x, y ∈ I. The equality holds iff x = y.Now, if we apply this inequality to the strictly convex mapping − ln (·) on theinterval (0,∞) we deduce1y(x− y) ≥ lnx− ln y ≥ 1x(x− y)for all x, y > 0, with equality iff x = y.Now, choose x = aa, y = ba in the above inequality to deduce(ab)a− 1 ≥ a ln a− a ln b ≥ 1−(ba)a(2.2)with equality iff a = b.If in (2.2) we put a = p(x), b = q(x) , then we obtain(p(x)q(x))p(x)− 1 ≥ p(x) ln p(x)− p(x) ln q(x) ≥ 1−(q(x)q(x))p(x),with equality iff p(x) = q(x), x ∈ X .Summing over x ∈ X we get∑x∈X[(p (x)q(x))p(x)− 1]≥ De(p‖q) ≥∑x∈X[1−(q (x)p(x))p(x)]with equality iff p(x) = q(x), x ∈ X .Now, let observe that(q (x)p(x))p(x)= q(x)(1p(x))p(x)( 1q(x))1−p(x), x ∈ X .Using the weighted arithmetic mean-geometric mean inequality, i.e., we recall itatb1−t ≤ ta+ (1− t)b, a, b > 0, t ∈ (0, 1)with equality iff a = b, for the choicesa =1p(x), b =1q(x), t = p(x), x ∈ XINEQUALITIES FOR THE RELATIVE ENTROPY 5we get (1p(x))p(x)( 1q(x))1−p(x)≤ 1 + 1− p(x)q(x)(2.3)with equality iff p(x) = q(x), x ∈ X .If we multiply (2.3) by q(x) ≥ 0 we get(q (x)p(x))p(x)≤ 1 + q(x)− p(x), x ∈ X .(2.4)Summing in (2.4) over x ∈ X , we deduce∑x∈X(q (x)p(x))p(x)≤ |X | −∑x∈Xq(x)−∑x∈Xp(x) = |X |from where we deduce ∑x∈X[1−(q (x)p(x))p(x)]≥ 0with equality iff p(x) = q(x), x ∈ X .Also, we have(p(x)q(x))p(x)= p(x)(1q(x))p(x)( 1p(x))1−p(x), x ∈ X .Using again the weighted A-G-inequality we can state that(1q(x))p(x)( 1p(x))1−p(x)≤ p(x)q(x)+1p(x)− 1.Multiplying this inequality by p(x) ≥ 0 we get(p(x)q(x))p(x)≤ 1 + p(x)2q(x)− p(x), x ∈ Xwhich gives, by summation over x, that∑x∈X(p(x)q(x))p(x)≤ |X |+∑x∈Xp(x)2q(x)−∑x∈Xp(x) = |X |+∑x∈Xp(x)2q(x)− 1,i.e., ∑x∈X[(p(x)q(x))p(x)− 1]≤∑x∈Xp(x)2q(x)− 1with equality iff p(x) = q(x), x ∈ X .As logb(x) =ln xln b , we deduce the desired inequality (2.1) .The following corollaries are importantCorollary 7. Let X be a random variable with the probability distribution p(x),x ∈ X . Then we have the inequality1ln b[|X |∑x∈Xp2(x)− 1]≥ 1ln b∑x∈X[|X |p(x) p(x)p(x) − 1](2.5)6 S.S. DRAGOMIR≥ log |X | −H(X) ≥ 1ln b∑x∈X[1− 1|X |p(x) p(x)p(x)]≥ 0.The equality holds in all inequalities iff p is uniformly distributed on X .The proof follows by the above theorem choosing q = u, the uniform distributionon X .The following corollary for mutual information also holds.Corollary 8. Let X and Y be as in the above theorem. Then1ln b∑x∈X∑y∈Yp2(x, y)p(x)q(y)− 1 ≥ 1ln b∑x∈X∑y∈Y[(p(x, y)p(x)q(y))p(x,y)− 1](2.6)≥ I(X,Y ) ≥ 1ln b∑x∈X∑y∈Y[1−(p(x, y)p(x)q(y))p(x,y)]≥ 0.The equality holds in all inequalities iff X and Y are independent.For recent results in the applications of Theory of Inequalities in InformationTheory and Coding, see the papers listed bellow.References[1] T.M. Cover and J.A. Thomas, Elements of Information Theory, John Wiley & Sons Inc.,1991.[2] S.S. Dragomir and N.M. Dragomir, An inequality for logarithms and its applications in codingtheory, Indian Journal of Mathematics (in press).[3] S. S. Dragomir and C. J. Goh, Some bounds on entropy measures in information theory,Appl. Math. Lett., 10(3) (1997) 23–28.[4] S. S. Dragomir and C. J. Goh, A counterpart of Jensen’s discrete inequality for differentiableconvex mappings and applications in information theory, Math. Comput. Modelling, 24(2)(1996) 1–11.[5] M. Matic´, C. E. M. Pearce and J. Pecˇaric´, Improvements of some bounds on entropy measuresin information theory, Math. Inequal. and Appl., 1(1998) 295–304.[6] M. Matic´, C. E. M. Pearce and J. Pecˇaric´, Refinements of some bounds in information theory,J. Austr. Math. Soc. Ser. B, to appear.[7] S.S. Dragomir, N.M. Dragomir and K. Pranesh, Some Estimations of Kraft number andrelated results, Soochow Journal of Mathematics, 24(4), 291-296.[8] S.S. Dragomir, M. Sholz and J. Sunde, Some upper bounds for relative entropy and applica-tions, submitted.[9] S.S. Dragomir, Some remarks on the noiseless coding theorem, submitted.[10] C. Calude and C. Grozea, Kraft-Chaitin inequality revisited, J. UCS, 5(1996), 306-310.School of Communications and Informatics, Victoria University of TechnologyE-mail address: sever@matilda.vu.edu.auURL: http://matilda.vut.edu.au/~rgmia/

Some Inequalities for the Relative Entropy and Applications

Abstract

Similar works

Full text

Available Versions

Victoria University Eprints Repository

Victoria University Institutional Repository (VUIR)

Name not available