Stein unbiased risk estimation is generalized twice, from the Gaussian shift
model to nonparametric families of smooth densities, and from the quadratic
risk to more general divergence type distances. The development relies on a
connection with local proper scoring rules.Comment: This is the author's version of a work that was accepted for
  publication in Comptes rendus Mathematiqu

Ehm, Werner

English

arXiv

Werner Ehm

Crossref

Comptes Rendus Mathematique

Unbiased risk estimation and scoring rules

Comptes Rendus Mathématique

C. R. Acad. Sci. Paris, Ser. I 349 (2011) 699–702Contents lists available at ScienceDirectC. R. Acad. Sci. Paris, Ser. Iwww.sciencedirect.comStatisticsUnbiased risk estimation and scoring rulesEstimation de risque non biaisée et règles d’évaluationWerner EhmInstitute for Frontier Areas of Psychology and Mental Health, Wilhelmstr. 3a, 79098 Freiburg, Germanya r t i c l e i n f o a b s t r a c tArticle history:Received 15 April 2011Accepted 27 April 2011Available online 16 June 2011Presented by Paul DeheuvelsStein unbiased risk estimation is generalized twice, from the Gaussian shift model tononparametric families of smooth densities, and from the quadratic risk to more generaldivergence type distances. The development relies on a connection with local properscoring rules.© 2011 Académie des sciences. Published by Elsevier Masson SAS. All rights reserved.r é s u m éLa méthode d’estimation du risque Steinien est doublement généralisée, d’une part dumodèle de translation Gaussien à des familles non paramétriques et d’autre part du risquequadratique à des distances du type divergence plus générales. Cette extension repose surune relation avec des règles d’évaluation locales propres.© 2011 Académie des sciences. Published by Elsevier Masson SAS. All rights reserved.1. Introduction: SURE and the Hyvärinen scoreConsider the problem of estimating the parameter θ in the standard Gaussian shift family Pθ = N (θ, Id), θ ∈ Rd , basedon an observation x ∈ Rd . Let T be an estimator of θ of the form T = x + g(x). Using partial integration, Stein [8] showedthat under weak conditions about g , the quadratic risk R(T , θ) = Eθ |T − θ |2 of T can be estimated unbiasedly by theexpression R̂(T ) = 2 div g(x) + |g(x)|2 + d called SURE (Stein unbiased risk estimate), so that Eθ R̂(T ) = R(T , θ) for everyθ ∈ Rd . Here | · | and 〈·,·〉 denote the Euclidean norm and inner product on Rd , respectively, and div g is the divergence of g .If in particular g = ∇ log f for some function f > 0 on Rd , the risk estimate becomesR̂(T ) = 2 log f (x) + ∣∣∇ log f (x)∣∣2 + d, (1)where as usual, ∇ denotes the gradient and  = div∇ the Laplace operator on Rd . This special case occurs if T is the pos-terior mean with respect to a prior distribution π : then T = x +∇ log f (x) where f (x) = ∫ pθ (x)dπ(θ) is the correspondingmixture density, so that g = ∇ log f .The striking similarity between SURE and the Hyvärinen score [5],H(p, x) = 2p(x)p(x)−∣∣∣∣∇p(x)p(x)∣∣∣∣2 = 2 log p(x) + ∣∣∇ log p(x)∣∣2, (2)has been noted in, e.g., [6]. In Eq. (2), p denotes a sufficiently smooth, strictly positive probability density on Rd . Originally,the Hyvärinen score was introduced for score matching, a minimum distance type estimation method. Its formal similarityE-mail address: ehm@igpp.de.1631-073X/$ – see front matter © 2011 Académie des sciences. Published by Elsevier Masson SAS. All rights reserved.doi:10.1016/j.crma.2011.04.015700 W. Ehm / C. R. Acad. Sci. Paris, Ser. I 349 (2011) 699–702to SURE is substantiated on reexpressing the risk of T as a distance between densities. Consider the Hyvärinen divergencedefined for smooth, positive densities p,q on Rd asdH(p,q) =∫ ∣∣∇ log p(y) − ∇ log q(y)∣∣2q(y)dy. (3)If p = f is a mixture density as above and q = pθ is the density of Pθ , we have ∇ log f (x)−∇ log pθ (x) = ∇ log f (x)+x−θ =T − θ, where again T = x + ∇ log f (x) is the corresponding posterior mean. Consequently,R(T , θ) = Eθ |T − θ |2 =∫ ∣∣∇ log f (x) − ∇ log pθ (x)∣∣2 pθ (x)dx = dH( f , pθ ), (4)that is, the risk R(T , θ) of the parameter estimate T = x + ∇ log f (x) equals a distance between densities, dH( f , pθ ). Furthermore,the analogue of SURE in the density scenario is the Hyvärinen score H( f , x), essentially. In fact, Hyvärinen’s idea, reinventingStein’s, was to apply partial integration to (3) which, assuming boundary terms vanish, givesdH(p,q) =∫ (2 log p(y) + ∣∣∇ log p(y)∣∣2)q(y)dy + ∫ ∣∣∇ log q(y)∣∣2q(y)dy; (5)cf. [1,5]. Since∫ |∇ log pθ (x)|2 pθ (x)dx = d (θ ∈ Rd) in the standard normal case, where q = pθ , it follows thatEθ(H( f , x) + d) = Eθ (2 log f (x) + ∣∣∇ log f (x)∣∣2 + d) = dH( f , pθ ). (6)That is, the modified Hyvärinen score H( f , x) + d represents an unbiased estimate of the distance dH( f , pθ ) of f from the unknown“true” density pθ , for any density f > 0 on Rd satisfying suitable regularity conditions.The purpose of this note is to expand on this aspect of unbiased risk estimation by tying it to scoring rules. Local properscoring rules are constructed as gradients of concave functionals [3,4], and then shown to generalize SURE in that theyfurnish unbiased estimates of modified Bregman type distances. The development is related to (parts of) work by Dawidand Lauritzen [1]. See also [2,7].2. Local proper scoring rules and unbiased risk estimationWe restrict the discussion of scoring rules to the setting relevant for this note, and refer to [3] for general information.Let P denote the class of all probability densities with respect to the Lebesgue measure on Rd such that the followingconditions hold for every p ∈ P : (P1) p ∈ C2; (P2) p > 0 everywhere on Rd; (P3) for every m > 0 and i, j ∈ {1, . . . ,d}lim|x|→∞|x|m(p(x) + ∣∣∂xi p(x)∣∣ + ∣∣∂2xi x j p(x)∣∣) = 0;(P4) there exists a = a(p) > 0 such that for i, j ∈ {1, . . . ,d},lim|x|→∞|x|−a(∣∣log p(x)∣∣ + [∂xi p(x)p(x)]2+|∂2xi x j p(x)|p(x))= 0.The class P is quite large, being convex and comprising, e.g., all normal and logistic distributions.A scoring rule is a mapping S : P × Rd → R assigning a numerical score, S(p, x), to the density forecast, p, when theobservation that materializes is x. We write S(p,q) = ∫ S(p, x)q(x)dx = EqS(p,·) for the expected score when the densityforecast is p and the probability measure underlying x is q(x)dx. The scoring rule S is (strictly) proper relative to P ifS(q,q)  S(p,q) for all p,q ∈ P (with equality only if p = q). The scoring rule S is local (of order two, for the class P ) ifthere exists a real function s such thatS(p, x) = s(x, log p(x),∇ log p(x),∇2 log p(x)) (p ∈ P, x ∈ Rd),∇2 f (x) denoting the Hessian matrix of second-order partial derivatives of a function f : Rd → R at x.The classical example of a (strictly) proper local scoring rule is the logarithmic score, S(p, x) = − log p(x). Another exam-ple is the Hyvärinen score (2). The latter can be regarded as being local of order two, in the obvious sense, and the formeras local of order zero. Local scoring rules of any order m  0 were recently investigated in [7], in the case d = 1. Hereafter,“local” always is understood as “local of order two.”The following result lifts the construction of local proper scoring rules in [2] from the one- to the higher-dimensionalcase d  1. Let K denote the class of the kernels k : Rd ×Rd → R satisfying the following conditions: (K1) k ∈ C2; (K2) thereare constants C, r ∈ (0,∞) such that whenever k∗ stands for the function k = k(x, y) or any of its partial derivatives up toorder two, then |k∗(x, y)|  C(1 + |x| + |y|)r (x, y ∈ Rd). With any k ∈ K we associate a functional Φ = Φk : P → R definedbyΦ(p) =∫dk(x,∇ log p(x))p(x)dx (p ∈ P). (7)RW. Ehm / C. R. Acad. Sci. Paris, Ser. I 349 (2011) 699–702 701In view of the growth and decay conditions (K2), (P4), and (P3), the integral in (7) exists and is finite for every p ∈ P . Let∇yk denote the partial gradient referring to the argument y ∈ Rd of k = k(x, y), and recall that div g(x) stands for the traceof the total derivative at x of a function x 	→ g(x) mapping Rd into itself.Theorem 2.1. Let k ∈ K, and suppose that the associated functional Φ is concave on P . ThenS(p, x) = k(x,∇ log p(x)) − 1p(x)div[p(x)∇yk(x,∇ log p(x))] (8)is a local proper scoring rule relative to P . It is strictly proper if Φ is strictly concave. Furthermore, if y 	→ k(x, y) is concave on Rd forevery x ∈ Rd, then the functional Φ is concave on P .Proof. The proof follows similar lines as in the case d = 1, see [2, Sections 4.1, 5.1]. We only indicate that the tangentconstruction in [2, Section 4.1] yields the scoring rule (8). To compute the (weak) gradient of Φ at q ∈ P , let pt = (1−t)q+tpwhere p ∈ P , t ∈ [0,1]. Formal differentiation ignoring all technicalities givesddt[Φ(pt)] = ∫ ∂∂t[Kpt pt]dx =∫[Kpt ](p − q)dx +∫ [∂∂tKpt]pt dx, (9)wherein we put Kpt (x) = k(x,∇ log pt(x)) and omitted the argument x of the integrands. For the last integral in (9) we getby the divergence theorem, assuming the boundary integral vanishes,∫ 〈∇yk(·,∇ log pt),∇(p − qpt)〉pt dx = −∫div[pt∇yk(·,∇ log pt)] p − qptdx. (10)Setting t = 0 in (9) and (10) and noting that p0 = q we find thatddt[Φ(pt)]∣∣∣∣t=0=∫ {k(·,∇ log q) − 1qdiv[q∇yk(·,∇ log q)]}(p − q)dx. (11)Thus, the gradient of Φ at q is given by the expression in curly brackets in (11). The scoring rule resulting from the tangentconstruction, S(q,·), differs from this gradient only by a correction term which can be shown to vanish. The negligibilityof the boundary integral in (10), and all the technicalities (existence of integrals, exchangeability of differentiation andintegration, etc.) can be settled similarly as in [2, Section 4.1], using the assumptions made about the classes P and K. Any convex combination of a scoring rule S as in Theorem 2.1 with the logarithmic score yields a local proper scoringrule. In the case d = 1, scoring rules of this form exhaust the class of all local proper scoring rules [2,7]. The completecharacterization in the case d > 1 remains open.Examples. Let k ∈ K be a kernel of the form k(x, y) = k(y) = ψ(|y|), where ψ is a concave C2-function on [0,∞) withψ(0) = ψ ′(0) = 0. Then y 	→ k(y) is concave on Rd , and the corresponding scoring rule (8) is proper. Explicitly we haveS(p,·) = ψ(|σ |) − ψ ′(|σ |)|σ | (|σ |2 +  log p) −[ψ ′′(|σ |) − ψ ′(|σ |)|σ |]〈σ|σ | ,(∇2 log p) σ|σ |〉where σ = ∇ log p. For ψ(t) = −t2 we obtain the Hyvärinen score (2); putting ψ(t) = − log cosh t yields another interestingexample parallel to [2, Example 5.3].A local scoring rule S that is proper relative to P gives rise to a Bregman type divergence measure dS(p,q) = S(p,q) −S(q,q) on P × P . The following representation of dS is closely related to [7, Eq. (53)].Theorem 2.2. Suppose that S is of the form (8) for some kernel k ∈ K such that y 	→ k(x, y) is concave on Rd for every x ∈ Rd. Thenthe divergence dS admits the representationdS(p,q) = Eq{k(·,∇ log p) − k(·,∇ log q) +〈∇qq− ∇pp,∇yk(·,∇ log p)〉}(p,q ∈ P). (12)Proof. Let p,q ∈ P . By the assumptions on P and K, the divergence theorem applied to the scalar function u(x) = q(x)/p(x)and the vector field v(x) = p(x)∇yk(x,∇ log p(x)) giveslimr→∞−∫qpdiv[p ∇yk(·,∇ log p)]dx = limr→∞∫ 〈∇qq− ∇pp,∇yk(·,∇ log p)〉q dx. (13)|x|r |x|r702 W. Ehm / C. R. Acad. Sci. Paris, Ser. I 349 (2011) 699–702The relation (12) follows on writing dS(p,q) = Eq{S(p,·) − S(q,·)}, substituting (8) and using (13), and observing that∫q−1 div(q∇yk(·,∇ log q))q dx = 0. Note that the expression in curly brackets in (12) is nonnegative because for a concave function f on Rd one hasf (y1) − f (y2)  〈y1 − y2,∇ f (y1)〉 (y1, y2 ∈ Rd). For the Hyvärinen score, where k(x, y) = −|y|2, that expression becomes|∇ log p − ∇ log q|2, and dS becomes the Hyvärinen divergence (3).To clarify the connection with SURE we note that the partial integration in (13) was used conversely by Stein andHyvärinen, to pass from the risk representation (12) to an expression of the form Eq{S(p,·) − S(q,·)}. In the latter, thescoring rule S(p,·) may serve as an unbiased estimate of EqS(p,·), while the term EqS(q,·) is the same for all candidates p,hence can be ignored if the focus is on risk comparison. In nonparametric density estimation, e.g., risk comparison ofcompeting estimates is applied for bandwidth selection, using cross-validation. Briefly, if p̂n = p̂n(·|x1, . . . , xn) is an estimateof the unknown density q ∈ P underlying the i.i.d. observations x1, . . . , xn that is symmetric in the xi , the cross-validatedexpression R̂n(p̂n−1) = n−1 ∑ni=1 S(p̂n,−i, xi) is an unbiased estimate of Rn−1(p̂n−1,q), where Rn(p̂n,q) = EqS(p̂n,q) denotesthe modified risk ignoring the term EqS(q, ·) = S(q,q), which depends only on q.The possibility of risk estimation is of course not confined to the local scoring rules considered here, as any properscoring rule S, whether local or not, gives rise to a divergence measure dS. Therefore, cross-validatory estimation of the(modified) risk generally is feasible, although exact unbiasedness as with the local scoring rules may not be achievablewhen global terms are involved. For example, unbiased estimation of the term∫p(x)2 dx entering the quadratic score [3]does not seem possible.The particular interest of the scoring rules of the form (8) ensues from the fact that they do not require the knowledge ofthe normalizing constants of the probability densities, which may be unknown or hard to obtain in complex settings [5,7].This advantage can be combined with other desirable features such as improved robustness by working, for instance, withthe log cosh scoring rule mentioned above.AcknowledgementsThe author thanks Tilmann Gneiting, Steffen Lauritzen, and Matthew Parry for helpful comments on earlier drafts of thepaper.References[1] A.P. Dawid, S.L. Lauritzen, The geometry of decision theory, in: Proc. 2nd Int. Symp. Inf. Geom. Appl., Univ. Tokyo, 2005, pp. 22–28.[2] W. Ehm, T. Gneiting, Local proper scoring rules of order two, Preprint, arXiv:1102.5031v1, 2011.[3] T. Gneiting, A.E. Raftery, Strictly proper scoring rules, prediction, and estimation, J. Amer. Statist. Assoc. 102 (2007) 359–378.[4] A.D. Hendrickson, R.J. Buehler, Proper scores for probability forecasters, Ann. Math. Statist. 42 (1971) 1916–1921.[5] A. Hyvärinen, Estimation of non-normalized statistical models using score matching, J. Mach. Learn. Res. 6 (2005) 695–709.[6] A. Hyvärinen, Optimal approximation of signal priors, Neural Computation 20 (2008) 3087–3110.[7] M. Parry, A.P. Dawid, S. Lauritzen, Proper local scoring rules, Preprint, arXiv:1101.5011v1, 2011.[8] C.M. Stein, Estimation of the mean of a multivariate normal distribution, Ann. Statist. 9 (1981) 1135–1151.

Numérisation de Documents Anciens Mathématiques

http://www.numdam.org/item/10.1016/j.crma.2011.04.015.pdf

Unbiased risk estimation and scoring rules

Abstract

Similar works

Full text

Available Versions

Crossref

Comptes Rendus Mathématique

Numérisation de Documents Anciens Mathématiques