We correct claims about lower bounds on mutual information (MI) between
real-valued random variables made in A. Kraskov {\it et al.}, Phys. Rev. E {\bf
69}, 066138 (2004). We show that non-trivial lower bounds on MI in terms of
linear correlations depend on the marginal (single variable) distributions.
This is so in spite of the invariance of MI under reparametrizations, because
linear correlations are not invariant under them. The simplest bounds are
obtained for Gaussians, but the most interesting ones for practical purposes
are obtained for uniform marginal distributions. The latter can be enforced in
general by using the ranks of the individual variables instead of their actual
values, in which case one obtains bounds on MI in terms of Spearman correlation
coefficients. We show with gene expression data that these bounds are in
general non-trivial, and the degree of their (non-)saturation yields valuable
insight.Comment: 4 page

David V. Foster

M. Li

Peter Grassberger

T. M. Cover

W. H. Press

English

arXiv

We correct claims about lower bounds on mutual information (MI) between real-valued random variables made by Kraskov et al., Phys. Rev. E 69, 066138 (2004). We show that non-trivial lower bounds on MI in terms of linear correlations depend on the marginal (single variable) distributions. This is so in spite of the invariance of MI under reparametrizations, because linear correlations are not invariant under them. The simplest bounds are obtained for Gaussians, but the most interesting ones for practical purposes are obtained for uniform marginal distributions. The latter can be enforced in general by using the ranks of the individual variables instead of their actual values, in which case one obtains bounds on MI in terms of Spearman correlation coefficients. We show with gene expression data that these bounds are in general nontrivial, and the degree of their (non) saturation yields valuable insight

Foster, D.V.

Grassberger, P.

Juelich Shared Electronic Resources

RAPID COMMUNICATIONSPHYSICAL REVIEW E 83, 010101(R) (2011)Lower bounds on mutual informationDavid V. Foster1 and Peter Grassberger2,31Complexity Science Group, University of Calgary, Calgary, Canada T2N 1N42Complexity Science Group, University of Calgary, Calgary, Canada3John von Neumann Institut fu¨r Computing, FZ Ju¨lich, D-52425 Ju¨lich, Germany(Received 6 August 2010; published 20 January 2011)We correct claims about lower bounds on mutual information (MI) between real-valued random variablesmade by Kraskov et al., Phys. Rev. E 69, 066138 (2004). We show that non-trivial lower bounds on MI in termsof linear correlations depend on the marginal (single variable) distributions. This is so in spite of the invarianceof MI under reparametrizations, because linear correlations are not invariant under them. The simplest boundsare obtained for Gaussians, but the most interesting ones for practical purposes are obtained for uniform marginaldistributions. The latter can be enforced in general by using the ranks of the individual variables instead of theiractual values, in which case one obtains bounds on MI in terms of Spearman correlation coefficients. We showwith gene expression data that these bounds are in general nontrivial, and the degree of their (non)saturationyields valuable insight.DOI: 10.1103/PhysRevE.83.010101 PACS number(s): 02.50.Sk, 89.70.−a, 87.10.−e, 87.18.−hMutual information [1] between two objects is the dif-ference between the combined lengths of their individualdescriptions and the length of a joint description, all descrip-tions being “optimal,” i.e. lossless and redundancy-free. Inthe framework of algorithmic information theory [2], this istaken literally, i.e. the “objects” are sequences of letters ofsome alphabet, and “description” means a compression of thesequence on some specified but otherwise arbitrary universalTuring machine. In the framework of Shannon theory, incontrast, we deal with random variables, and “descriptionlength” is to be understood as the minimal average informationneeded to specify their realizations, given the probabilitydistributions.In the following we shall only use the Shannon framework,but we shall not forget entirely about individual objects.When confronted with them, we make some (explicit orimplicit) estimate about the probability distribution (assumingthat the observed objects are in some sense “typical”);computing their MI is actually a problem of statisticalinference.More precisely, consider two random variables X and Ywith realizations x and y and probability densities pX(x) andpY (y). For simplicity we shall assume that x and y are bothscalars taken either from a finite interval or from the interval[−∞,∞]. In both cases pX and py are normalized to 1. Thejoint distribution is p(x,y). The MI is then defined asI (X : Y ) =∫dxdyp(x,y) log p(x,y)pX(x)pY (y), (1)where the base of the logarithm specifies the units inwhich information is measured. Bits correspond to logarithmbase 2.From this one sees that I is symmetrical, I (X : Y ) = I (Y :X), and positive definite: I (X : Y ) = 0 if and only if X and Yare strictly independent. Thus I (X : Y ) is a universal measureof dependency, being nonzero whenever X and Y have any-thing in common. This can also be seen in the following way:the (differential) entropy H (X) = − ∫ dxpX(x) log pX(x) isthe (negative) average log-likelihood of x, andI (X : Y ) = H (X) − H (X|Y ) (2)is the logarithm of the ratio between the unconditionedlikelihood of x and the posterior likelihood conditioned onthe value y of Y .For the differential entropy, there is a well-known upperbound in terms of the variance: H (X) is maximal for aGaussian with the same variance as the data [1]. Indeed, thisis true also for multivariate distributions. In the Appendixof [3], a formal proof based on Lagrangian multipliers wasgiven that analogous bounds hold also for the MI. Accordingto [3], a given covariance matrix implies a lower bound on theMI. Unfortunately, this proof is wrong, and the claim madein [3] is incorrect (see also the Erratum [4]). The error in [3]was subtle: The unique solution of the Lagrangian variationalproblem was given correctly, but the fact was missed that thissolution is in general a saddle point, the correct bound beingan infimum which is not reached by any actual distribution (atleast not by a distribution in the class admitted in the variationalproblem).Indeed, it is easily seen that the MI can be arbitrarilysmall for any value of the correlation. Assume that the jointdistribution is a sum of a δ peak with weight 1 −  centeredat (x,y) = (1,1) and a two-dimensional (2D) Gaussian withweight  centered at the origin,p(x,y) = (1 − )δ(x − 1)δ(y − 1) + 2πσ 2e− x2+y22σ2 . (3)Then the correlation between X and Y varies between zeroand one as the width σ shrinks to zero, for any fixed  > 0.But the MI is bounded for all σ by I (X : Y )  − log  −(1 − ) log(1 − ), which tends to zero as  → 0. Thus the MIcan be arbitrarily close to zero, even when the correlation isarbitrarily close to 1—although this is unlikely to appear inreal applications, except for outliers.It is the purpose of the present paper to present correctbounds replacing those given in [3]. As we shall see, to obtainnontrivial bounds for the MI, one needs both the covariancematrix and the marginal distributions. But the latter can be010101-11539-3755/2011/83(1)/010101(4) ©2011 American Physical SocietyRAPID COMMUNICATIONSDAVID V. FOSTER AND PETER GRASSBERGER PHYSICAL REVIEW E 83, 010101(R) (2011)chosen arbitrarily to a large extent, since I (X : Y ) as definedin Eq. (1) is invariant under homeomorphism. Let φ(x) bea continuous and monotonic function, such that its inverseφ−1(x) is also continuous and monotonic, and let X′ be arandom variable with realization x ′ = φ(x) if X has realizationx. ThenpX(x) =∣∣∣∣dφ(x)dx∣∣∣∣pX′(x ′), (4)and I (X : Y ) = I (X′ : Y ). By symmetry, the same holds forhomeomorphisms of Y .This leads to the following strategy for obtaining boundson I (X : Y ): One first transforms X and Y independentlyso that they have a given distribution, e.g., a Gaussian or auniform distribution. Notice that the first and second momentsin general will change during such a transformation. After thatis done, one applies the bound suitable for the chosen marginaldistributions.The case of Gaussian marginal distributions is the simplestto treat theoretically. In that case the arguments given in theAppendix of [3] apply, and the MI is bounded from below bythe MI of a joint Gaussian with the observed first and secondmoments. But this is not the most practical choice, becauseit is nontrivial to transform any empirical distribution into aGaussian.For practical purposes much more suitable is transformationto uniform distributions over finite intervals, say x ′ ∈ [−1,1]and y ′ ∈ [−1,1]. This transformation, which also leads usuallyto improved MI estimates, is de facto achieved by using forx ′ and y ′ their normalized ranks. Assume that the empiricaldata consist of N pairs (xi,yi), i = 1, . . . ,N . Then the rank riof xi is defined as the number of values xj which are less thanor equal to xi (here we assume that all xi are different, as is truewith probability 1 if X is drawn from a continuous distribution;if there are degeneracies due, e.g., to discretization, weremove them by adding small random fluctuations to xi).Finally,x ′i = 2ri/N − 1. (5)and analogously for y. Notice that this does not, strictlyspeaking, define X′, as it defines the homeomorphism φ onlyat the discrete values xi , but this does not pose a practicalproblem. Furthermore, in the limit N → ∞ the “empiricalφ(x)” tends with probability 1 toward a true homeomorphism.The linear correlation between the ranks of x and y is bydefinition the Spearman coefficient S = CX′Y ′ [5].To obtain a bound on the MI for given marginal distributionsand given first and second moments, we use the Lagrangianmethod. Without loss of generality we assume that the data arecentered, i.e., 〈X〉 = 〈Y 〉 = 0. We use p(x,y) as independentvariables, andpX(x) =∫dyp(x,y), pY (y) =∫dxp(x,y);CXY =∫∫dxdyxyp(x,y)/[σXσY ] (6)as constraints. The Lagrangian function isL =∫∫dx dy p(x,y) log p(x,y)pX(x)pY (y)+∫dxνX(x)[pX(x) −∫dy p(x,y)]+∫dyνY (y)[pY (y) −∫dx p(x,y)]+ λ[σXσYCXY −∫∫dx dy xyp(x,y)], (7)where νX(x), νY (y), and λ are Lagrangian parameters. Thevariational equations areδLδp(x,y) = logp(x,y)pX(x)pY (y)+ 1 − νX(x) − νY (y) − λxy = 0,(8)which can also be written asp(x,y) = fX(x)fY (y)e−λ(x−y)2 (9)with unknown functions fX,fY and unknown λ, all ofwhich are determined by the constraints. The Kolmogorovconsistency condition for pX(x), in particular, givespX(x)fX(x)=∫dyfY (y)e−λ(x−y)2 . (10)In the following we shall discuss only the two cases ofGaussian and uniform marginals. For Gaussian marginals, onefinds that p(x,y) is also Gaussian, and thus the results of [3]are obtained,I (X : Y )  I−Gauss(CXY ) ≡ − 12 log(1 − C2XY). (11)For uniform marginals, we indeed do not solve the problem offinding a bound I−unif on the MI for given S, but we solve theeasier implicit problem of finding both I−unif and S for given λ.We do this recursively, starting with the zeroth approximationf(0)X (x) = f (0)Y (y) = 1/2. (12)From the kth approximation of fX and fY we obtain the(k + 1)st approximations by means of1f(k+1)X (x)= 2∫ 1−1dyf(k)Y (y)e−λ(x−y)2, (13)1f(k+1)Y (y)= 2∫ 1−1dxf(k)X (x)e−λ(x−y)2. (14)When doing this, we observe that f (k)X and f(k)Y are evenfunctions for each k, and that both indeed are equal. We canthus drop the subscripts and write the recursion asf (k+1)(x) =[2∫ 1−1dyf (k)(y)e−λ(x−y)2]−1. (15)010101-2RAPID COMMUNICATIONSLOWER BOUNDS ON MUTUAL INFORMATION PHYSICAL REVIEW E 83, 010101(R) (2011)TABLE I. Spearman coefficient and lower bound on the MI (innatural units).λ S I−unif0.00 0.0000 0.00000.25 0.0829 0.00340.50 0.1633 0.01350.75 0.2390 0.02921.00 0.3086 0.04951.25 0.3713 0.07291.50 0.4270 0.09842.00 0.5189 0.15172.50 0.5897 0.20403.00 0.6428 0.25314.00 0.7177 0.33965.00 0.7666 0.41236.00 0.8007 0.47467.00 0.8260 0.52928.00 0.8455 0.57779.00 0.8610 0.621510.00 0.8736 0.661411.50 0.8887 0.715613.00 0.9005 0.763615.00 0.9128 0.820817.00 0.9224 0.871720.00 0.9333 0.938923.00 0.9415 0.997527.00 0.9498 1.065732.00 0.9572 1.139340.00 0.9654 1.236650.00 0.9721 1.3357After convergence, the joint density is obtained asp(x,y) ∝ limk→∞f (k)(x)f (k)(y)e−λ(x−y)2 . (16)Here we have left the normalization open, in order to allowfor errors in the numerical integration which might have ac-cumulated during the recursion. The proportionality constantis thus fixed by the normalization condition∫p = 1. Finally,S and the lower bound I−unif(S) on I (X : Y ) are obtained byusing Eq. (1) andS = 3∫∫ 1−1dx dy xy p(x,y)e−λ(x−y)2 . (17)Numerical results for several values of λ, obtained byusing Gaussian quadrature for the integrals, are given inTable I. Except for values of S close to ±1, I−unif(S) is wellapproximated byI−unif(S) ≈ − 12 (1 − 0.122S2 + 0.053S12) log(1 − S2). (18)The two bounds for Gaussians [Eq. (11)] and for uniformdistributions [Eq. (18)] are shown in Fig. 1.As an application we show in Fig. 2 gene expressiondata obtained from human B lymphocyte cells [6]. In thatexperiment, the expressions of 12 600 different gene loci were 0 0.2 0.4 0.6 0.8 1 1.2 1.4-1 -0.8 -0.6 -0.4 -0.2  0  0.2  0.4  0.6  0.8  1bounds   I-   (natural units)C  resp.  SI-unif(S)   [Eq. (18)]I-Gauss(C)   [Eq. (11)]FIG. 1. (Color online) Lower bounds of MI in terms of theSpearman correlation coefficient (continuous line, red) and in terms ofthe Pearson correlation coefficient in the case of Gaussian marginals(dashed, green). For both curves, the MI is measured in natural units.measured in 336 different conditions, with special interest intumor cells. For each pair of genes the data can thus be rep-resented as 336 points in a two-dimensional plane. Spearmancoefficients were obtained by ranking both coordinates (afterdisambiguating degeneracies by adding low-level noise asexplained above). Mutual informations were estimated usingthe k-nearest-neighbor method of [3] with k = 40. Althoughthis was done for all 12 600 × 12 599/2 pairs, only resultsfor the 12 599 pairs involving the important cancer geneBCL6 are shown in Fig. 2. We can make the followingobservations:(a) The bound is respected by most pairs, and it formsroughly a lower envelope for the distribution.(b) There are several pairs for which the bound is violated,mostly for small values of S. This reflects the fact that theMI estimator is not perfect. Indeed, no MI estimator canbe perfect. Most estimators are chosen such that they never 0 0.1 0.2 0.3 0.4 0.5-0.8 -0.6 -0.4 -0.2  0  0.2  0.4  0.6  0.8Mutual Info., BCL6 versus all othersSpearman coeff.k = 40 neighborslower boundFIG. 2. (Color online) Mutual informations (in nats) betweengene BCL6 and all the other 12 599 genes as measured in themicroarray gene expression experiment of [6]. Values of the MI wereestimated by means of k nearest neighbors with k = 40. The greenline is the lower bound discussed in this paper.010101-3RAPID COMMUNICATIONSDAVID V. FOSTER AND PETER GRASSBERGER PHYSICAL REVIEW E 83, 010101(R) (2011) 0 500 1000 1500 2000 2500 3000 0  500  1000  1500  2000  2500  3000expression  of  gene  AA978353BCL6  expression 0 500 1000 1500 2000 2500 0  500  1000  1500  2000  2500  3000expression  of  gene  AL079277BCL6  expressionFIG. 3. (Color online) Each panel shows the gene expressionintensities (arbitrary units) of two genes, one of which is BCL6(x axis). The other gene (y axis) was chosen such as to have very largeMI with BCL6, but very small Spearman coefficient (the uppermosttwo points in Fig. 2 with |S| < 0.1). The color coding (green for BCL6expression <200 and AL079277 expression <500, red otherwise) issuch that the same cell conditions have in both panels the same color.It suggests that the observed nonlinear correlations are related to theexistence of two cell populations with very different properties. Thetwo genes correspond to accession numbers AA978353 (top) andAL079277 (bottom).produce negative MI, which is achieved by tolerating a positivebias. The estimator of [3] was constructed such that the biasis minimized, at the cost of obtaining occasionally negativevalues due to statistical fluctuations.(c) For most pairs the bound is not saturated, showingthat there are important nonlinear dependencies between thesepairs. As an illustration for the latter we take the two pointswith |S| < 0.1 and I > 0.3 and plot their gene expressionvectors in Fig. 3. They show the coexpression of BCL6 withthe genes with GenBank accession numbers AA978353 (top)and AL079277 (bottom). In both panels of Fig. 3 we see verystrong dependencies which cannot be approximated by linearcorrelations. Neither of these two genes is known to be relatedto BCL6, maybe because such relations were overlookedbecause of the small linear correlations. The data suggest thepresence of (at least) two different subpopulations of cells,marked in Fig. 3 by different colors. In the subpopulation inwhich BCL6 is strongly expressed (red points in Fig. 3) thereare also significant linear correlations.In summary, we have derived lower bounds on the MIbetween real-valued variables in terms of linear correlationcoefficients. We have seen that such bounds are not indepen-dent of the marginal distribution, in contrast to the claims madein the Appendix of [3]. But one can use the homeomorphisminvariance of the MI to transform the variables to new variableswith uniform distribution, in which case the linear correlationcoefficient becomes equal to the Spearman coefficient S.At least in one specific and scientifically relevant example,the resulting bound of the MI in terms of S was found tobe numerically nontrivial. In particular, large discrepanciesbetween the bound and the actual values gave hints at specificstructures in the data which then could be investigated in moredetail. The bound can also be useful in testing MI estimators.Usually, an estimator is deemed unacceptable if it violates thebound I (X : Y )  0. But it will be equally unacceptable if itviolates the stronger bound I (X : Y )  I−.Finally, our results also answer the question of how linearcorrelations change under reparametrizations. There is noreason to expect a universal exact answer, but approximatelythey should change such that the numerical values of thebounds I− stay the same.We thank Andrea Califano for providing us the data ofRef. [6], and Alexander Kraskov and Maya Paczuski fordiscussions.[1] T. M. Cover and J. A. Thomas, Elements of Information Theory,2nd edition (John Wiley & Sons, New York, 2006).[2] M. Li and P. M. B. Vita´nyi, An Introduction to KolmogorovComplexity and Its Applications, 3rd ed. (Springer, Berlin,2008).[3] A. Kraskov, H. Sto¨gbauer, and P. Grassberger, Phys. Rev. E 69,066138 (2004).[4] A. Kraskov, H. Sto¨gbauer, and P. Grassberger, Phys. Rev. E83, 019903(E) (2011).[5] W. H. Press, B. P. Flannery, S. A. Teukolsky, and W. T. Vetterling,Numerical Recipes: The Art of Scientific Computing, 3rd ed.(Cambridge University Press, Cambridge, 2007).[6] K. Basso, A. A. Margolin, G. Stolovitzky, U. Klein, R. Dalla-Favera, and A. Califano, Nat. Genet. 37, 382 (2005).010101-4

Lower bounds on mutual information

Crossref

Physical Review E

http://juser.fz-juelich.de/record/19793/files/PhysRevE.83.010101.pdf

Lower Bounds on Mutual Information

Abstract

Similar works

Full text

Available Versions

Juelich Shared Electronic Resources

Crossref