We present a derivation of the Kullback Leibler (KL)-Divergence (also known
as Relative Entropy) for the von Mises Fisher (VMF) Distribution in
$d$-dimensions.Comment: 8 pages 1 figur

Diethe, Tom

English

arXiv

We present a derivation of the Kullback Leibler (KL)-Divergence (also known as Relative Entropy) for the von Mises Fisher (VMF) Distribution in d-dimensions

Explore Bristol Research

                          Diethe, T. (2015). A Note on the Kullback-Leibler Divergence for the vonMises-Fisher distribution. arXiv.Peer reviewed versionLink to publication record in Explore Bristol ResearchPDF-documentUniversity of Bristol - Explore Bristol ResearchGeneral rightsThis document is made available in accordance with publisher policies. Please cite only the publishedversion using the reference above. Full terms of use are available:http://www.bristol.ac.uk/pure/about/ebr-termsA Note on the Kullback-Leibler Divergence forthe von Mises-Fisher distributionT.R. DietheDepartment of Electrical and Electronic EngineeringUniversity of BristolFebruary 26, 2015AbstractWe present a derivation of the Kullback Leibler (KL)-Divergence (alsoknown as Relative Entropy) for the von Mises Fisher (VMF) Distributionin d−dimensions.1 IntroductionThe von Mises Fisher (VMF) Distribution (also known as the Langevin Distri-bution [8]) is a probability distribution on the (d− 1)-dimensional hypersphereSd−1 in Rd [3]. If d = 2 the distribution reduces to the von Mises distributionon the circle, and if d = 3 it reduces to the Fisher distribution on a sphere.It was introduced by [3] and has been studied extensively by [6, 7]. The firstBayesian analysis was in [5] and recently it has been used for clustering on ahypersphere by [2].Figure 1: Three sets of 1000 points sampled from three VMF distributions onthe 3D sphere with κ = 1 (blue), κ = 10 (green) and κ = 100 (red), respectively.The mean directions are indicated with arrows.1arXiv:1502.07104v1  [stat.ML]  25 Feb 20152 Preliminaries2.1 DefinitionsWe will use log(z) to denote the natural logarithm of z throughout this article.Before continuing it will be useful to define the Gamma function Γ(z),Γ(z) =∫ ∞0tz−1e−tdt, z ∈ C, Re(z) > 0 (1)Γ(z) = (z − 1)!, z ∈ Z+ (2)and its relation, the incomplete Gamma function Γ(z, s),Γ(z, s) = (s− 1)!e−xs−1∑m=0zmm!, z ∈ Z+ (3)and the Modified Bessel Function of the First Kind Iα(z),Iα(z) =∞∑m=0(z/2)2m+αm!Γ(m+ α+ 1), (4)which also has the following integral representations [1],Iα(z) =(z/2)α√piΓ(α+ 1/2)∫ pi0e±z cos θ sin2d θ dθ, (α ∈ R) (5)=(z/2)α√piΓ(α+ 1/2)∫ 1−1(1− t2)(α−1/2)e±zt dt. (α ∈ R, α > −0.5) (6)Also of interest is the logarithm of this quantity (using the second integraldefinition (6)),log (Iα(z)) = log[( z2 )α√piΓ(α+ 1/2)∫ 1−1(1− t2)(α−1/2)e±zt dt]= log( z2 )α√piΓ(α+ 1/2)+ log[∫ 1−1(1− t2)(α−1/2)e±zt dt]= log(z2)α− log√piΓ(α+ 1/2) + log[∫ 1−1(1− t2)(α−1/2)e±zt dt].(7)Note that the second term does not depend on z.The Exponential Integral function En(z) is given by,Eα(z) =∫ ∞1e−zttαdt,= zα−1Γ(1− α, z). (8)An identity that will be useful is,∫ 1−1(1− t)detκ = −2d−1E−d(2κ)eκ. d > 0 (9)22.2 The von Mises Fisher (VMF) distributionThe probability density function (PDF) of the VMF distribution for a randomd-dimensional unit vector x(‖x‖2 = 1) is given by:Md(µ, κ) = cd(κ)eκµ′x, x ∈ Sd−1, (10)where the normalisation constant cd(κ) is given by,cd(κ) =κd/2−1(2pi)d/2Id/2−1(κ). (11)The (non-symmetric) Kullback Leibler (KL)-Divergence from one probabilitydistributions q(x) to another probability distribution p(x) is defined as,KL(q(x)||p(x)) =∫xq(x) logq(x)p(x)dx, (12)= Ex[logq(x)p(x)]. (13)Although this is general to any two distributions, we will assume that p(x) isthe “prior” distribution and q(x) is the “posterior” distribution as commonlyused in Bayesian analysis.3 KL-Divergence for the VMF Distribution3.1 General CaseWe will assume that we have prior and posterior distributions defined overvectors x ∈ Rd, ‖x‖2 = 1 as follows,p(x) ∼Md(µp, κp),q(x) ∼Md(µq, κq). (14)We will now derive the KL-Divergence for two VMF distributions. The mainproblem in doing so will be the the normalisation constants cd(κp) and cd(κq).Theorem 3.1 For prior and posterior distributions as defined above over vec-tors x ∈ Rd, ‖x‖2 = 1, d <∞, d odd1, we haveKL(q(x)||p(x)) ≤ κq − κpµ′pµq + d• log(κq) +d∑m=1κmqm!−(d2 − 2d+ 14)log(κp) + d(d + 1) log d − d2 + 1(15)Proof From (12), letting d? = d2 − 1, d = d−32 , and d• = d−12 , we have,KL(q(x)||p(x)) =∫xq(x) logq(x)p(x)dx,1For even d we can simply add a “null” dimension3=∫xq(x)[log cd(κq)eκqµ′qx − log cd(κp)eκpµ′px]dx,=∫xq(x)[log cd(κq)− log cd(κp) + κqµ′qx− κpµ′px]dx,=∫xq(x) [d? log(κq)− (d/2) log(2pi)− log Id?(κq)−d? log(κp) + (d/2) log(2pi) + log Id?(κp) + κqµ′qx− κpµ′px]dx,=∫xq(x)[d? log(κqκp)− log Id?(κq) + log Id?(κp) + κqµ′qx− κpµ′px]dx=∫xq(x)[d? log(κqκp)+ κqµ′qx− κpµ′px− log(κq2)d?+ log√piΓ(d? +12)− log∫ 1−1(1− t2)(d?−1/2)e±κqt dt+ log(κp2)d?− log√piΓ(d? +12)+ log∫ 1−1(1− t2)(d?−1/2)e±κpt dt]dx(Using (7)) (16)=∫xq(x)[d? log(κqκp)+ κqµ′qx− κpµ′px− d? log(κq2)+ d? log(κp2)− log∫ 1−1(1− t2)de±κqt dt+ log∫ 1−1(1− t2)de±κpt dt]dx=∫xq(x)[κqµ′qx− κpµ′px− log∫ 1−1(1− t2)de±κqt dt+ log∫ 1−1(1− t2)de±κpt dt]dx=∫xq(x)[κqµ′qx− κpµ′px− log[−2 d−32 E−d(2κq)eκq]+ log[−2 d−32 E−d(2κp)eκp]]dx(Using (9)) (17)=∫xq(x)[κqµ′qx− κpµ′px− κq + κp − log [E−d(2κq)] + log [E−d(2κp)]]dx=∫xq(x)[κq(µ′qx− 1)− κp(µ′px− 1)− log(2κ−d•q Γ (d•, 2κq))+ log(2κ−d•p Γ (d•, 2κp))]dx(Using the definition of the Exponential Integral function (8)) (18)=∫xq(x)[κq(µ′qx− 1)− κp(µ′px− 1) + d• log(2κq)− d• log(2κq)− log (Γ (d•, 2κq)) + log (Γ (d•, 2κp))] dx=∫xq(x)[κq(µ′qx− 1)− κp(µ′px− 1) + d• log(2κq)− d• log(2κq)− log(d!e−κqd∑m=0κmqm!)+ log(d!e−κpd∑m=0κmpm!)]dx4(Using (3) and that d• − 1 = d) (19)=∫xq(x)[κq(µ′qx− 1)− κp(µ′px− 1) + d• log(κq)− d• log(κp) + κq − κp− log(d∑m=0κmqm!)+ log(d∑m=0κmpm!)]dx=∫xq(x)[κqµ′qx− κpµ′px+ d• log(κq)− d• log(κp)− log(d∑m=0κmqm!)+ log(d∑m=0κmpm!)]dxFurther simplifications: (20)≤∫xq(x)[κqµ′qx− κpµ′px+ d• log(κq)− d• log(κp)− log(d∑m=0κmqm!)+(d∑m=0logκmpm!)]dx(by Jensen’s inequality) (21)=∫xq(x)[κqµ′qx− κpµ′px+ d• log(κq)− d• log(κp)+ log(d∑m=0κmqm!)−d∑m=0(m log(κp)− logm!)]dx≤∫xq(x)[κqµ′qx− κpµ′px+ d• log(κq)− d• log(κp)+ log(d∑m=0κmqm!)−d∑m=1(m log(κp)−m logm+m− 1)]dx(using n log ne + 1 ≤ log n! ≤ (n+ 1) log n+1e + 1) (22)=∫xq(x)[κqµ′qx− κpµ′px+ d• log(κq)− d• log(κp) + log(d∑m=0κmqm!)−d∑m=1(m log(κp)−m logm)− d(d + 1) + (d + 1)]dx=∫xq(x)[κqµ′qx− κpµ′px+ d• log(κq)− d• log(κp) + log(d∑m=0κmqm!)−d∑m=1(m log(κp)−m logm)− d2 + 1]dx=∫xq(x)[κqµ′qx− κpµ′px+ d• log(κq)− d• log(κp) + log(d∑m=0κmqm!)−d∑m=1(m log(κp)) + d(d + 1) log d − d2 + 1]dx5=∫xq(x)[κqµ′qx− κpµ′px+ d• log(κq)− d• log(κp) + log(d∑m=0κmqm!)−d(d + 1) log(κp) + d(d + 1) log d − d2 + 1]dx=∫xq(x)[κqµ′qx− κpµ′px+ d• log(κq)− d• log(κp) + log(d∑m=0κmqm!)−((d− 3)24+d− 32)log(κp) + d(d + 1) log d − d2 + 1]dx=∫xq(x)[κqµ′qx− κpµ′px+ d• log(κq) + log(d∑m=0κmqm!)−(d2 − 2d+ 14)log(κp) + d(d + 1) log d − d2 + 1]dx=∫xq(x)[κqµ′qx− κpµ′px+ d• log(κq) + log(1 +d∑m=1κmqm!)−(d2 − 2d+ 14)log(κp) + d(d + 1) log d − d2 + 1]dx≤∫xq(x)[κqµ′qx− κpµ′px+ d• log(κq) +d∑m=1κmqm!−(d2 − 2d+ 14)log(κp) + d(d + 1) log d − d2 + 1]dx(using n ≥ log(1 + n) ≥ n1+n , (n > −1)) (23)=∫xq(x)[κqµ′qx− κpµ′px+ d• log(κq) +d∑m=1κmqm!−(d2 − 2d+ 14)log(κp) + d(d + 1) log d − d2 + 1]dx= κq − κpµ′pµq + d• log(κq) +d∑m=1κmqm!−(d2 − 2d+ 14)log(κp) + d(d + 1) log d − d2 + 1(as∫xq(x) = 1, and E[x] = µq, and µ′qµq = 1) (24)The term µ′qµp can be seen as the cosine distance between the prior andpostieror mean vectors. For 0 < κq < 1, the term∑dm=1κmqm! ≥ κq. However forlarge κq and large d this term can grow very large.Special case: uniform priorSince the VMF distribution is defined on the Sd−1, hypersphere, which is actu-ally a specific case of a Stiefel manifold where r = 1 is the radius. The Stiefel6manifold has finite area,τ(d, r) =2rpipr2pir(r−1)4∏rj=1 Γ(p−j+12) , (25)and so,τ(d, 1) =2pip2Γ(p2) , (26)For the special case of the uniform prior (more precisely limκp→0), the priorPDF reduces to,Md(µ, κ) = cd(0)e0=Γ(d2)2pid2, (27)which is simply one over the area on the manifold. This leads to a simpler formfor the KL-divergence.Corollary 3.2 For prior and posterior distributions as defined above over vec-tors x ∈ Rd, ‖x‖2 = 1, d <∞, we haveKL(q(x)||p(x)) = κq − d? log 2(28)ProofKL(q(x)||p(x)) =∫xq(x) logq(x)p(x)dx,=∫xq(x)[log cd(κq)eκqµ′qx − log cd(0)]dx,=∫xq(x)[κqµ′qx+ log cd(κq)− log Γ(d2)+ log(2pid2)]dx,=∫xq(x)[κqµ′qx+ log cd(κq)− log(d?)! + (d/2) log (2pi)]dx,=∫xq(x)[κqµ′qx+ d? log(κq)− (d/2) log(2pi)− log Id?(κq)− log(d?)! + (d/2) log (2pi)] dx,=∫xq(x)[κqµ′qx+ d? log(κq)− log Id?(κq)− log(d?)!]dx,=∫xq(x)[κqµ′qx+ d? log(κq)− log(κq2)d?+ log Γ(d2)− log(d?)!]dx,=∫xq(x)[κqµ′qx+ d? log(κq)− d? log(κq2)]dx,=∫xq(x)[κqµ′qx− d? log 2]dx,= κq − d? log 2,(29)7For this special case, it can be seen that the dependence on the dimensionis much more benign. This could prove useful for further computation (e.g. ifthe KL-divergence were to be used in a probably approximately correct (PAC)-Bayes bound [4]).4 ConclusionsWe have presented a derivation of the Kullback Leibler (KL)-divergence for thevon Mises Fisher (VMF)-distribution, including the special case of a uniformprior over the hypersphere.References[1] Milton Abramowitz, Irene A Stegun, et al. Handbook of mathematical func-tions, volume 1. Dover New York, 1972.[2] Arindam Banerjee, Inderjit S Dhillon, Joydeep Ghosh, and Suvrit Sra. Clus-tering on the unit hypersphere using von mises-fisher distributions. In Jour-nal of Machine Learning Research, pages 1345–1382, 2005.[3] Ronald Fisher. Dispersion on a sphere. In Proceedings of the Royal Societyof London A: Mathematical, Physical and Engineering Sciences, volume 217,pages 295–305. The Royal Society, 1953.[4] John Langford. Tutorial on practical prediction theory for classification. InJournal of machine learning research, pages 273–306, 2005.[5] Kanti V Mardia and SAM El-Atoum. Bayesian inference for the von mises-fisher distribution. Biometrika, 63(1):203–206, 1976.[6] Kantilal Varichand Mardia. Statistics of directional data. Academic Press,2014.[7] KV Mardia and PJ Zemroch. Algorithm as 86: The von mises distributionfunction. Applied Statistics, pages 268–272, 1975.[8] Yoko Watamori et al. Statistical inference of langevin distribution for direc-tional data. Hiroshima Mathematical Journal, 26(1):25–74, 1996.8

A Note on the Kullback-Leibler Divergence for the von Mises-Fisher distribution

Abstract

Similar works

Full text

Available Versions

Explore Bristol Research