I give a pedagogical derivation of the Cramer-Rao Bound, which
          gives a lower bound on the variance of estimators used in statistical
          point estimation, commonly used to give numerical estimates of the
          systematic uncertainties in a measurement

Reece, Ryan

English

PhilPapers

Derivation of the Crame´r-Rao BoundRyan D. ReeceOctober 1, 2009AbstractI give a pedagogical derivation of the Crame´r-Rao Bound, whichgives a lower bound on the variance of estimators used in statisticalpoint estimation, commonly used to give numerical estimates of thesystematic uncertainties in a measurement.1 DerivationFor estimators θˆi of parameters θi in a given model with liklihood functionL, the bias of these estimators bi is defined asbi ≡ E[θˆi(x)− θi] ≡∫dx L(x, θ) (θˆi(x)− θi)Note that the estimators θˆi depend on the observable x, and the likelihoodfunction L depends on x and the paramters of the model θ. For ease ofnotation these dependencies will now be left implicit. The observable couldbe a tuple of many measurements x = {x1, x2, . . . xN}, in which case theintegral over x actual denotes integrals over each independent measurement.∫dx =∫dx1∫dx2 · · ·∫dxNIn that case, the likelihood function is the joint likelihood function, which isa product of the likelihood functions for each measurement.L =N∏i=1Li1Taking the derivative of the bias with respect to its corresponding pa-rameter gives∂bi∂θi=∫dx (θˆi − θi) ∂L∂θi︸︷︷︸L ∂ lnL∂θi−∫dx L︸ ︷︷ ︸11 +∂bi∂θi=∫dx (θˆi − θi) L ∂ lnL∂θi(1 +∂bi∂θi)(1 +∂bj∂θj)=[∫dx (θˆi − θi) L ∂ lnL∂θi] [∫dx (θˆj − θj) L ∂ lnL∂θj]Then we use the Cauchy-Schwarz inequality:∣∣∣∣∫ dx f(x) g(x)∣∣∣∣2 ≤ ∫ dx |f(x)|2 · ∫ dx |g(x)|2(1 +∂bi∂θi)(1 +∂bj∂θj)≤[∫dx L (θˆi − θi)(θˆj − θj)] [∫dx L∂ lnL∂θi∂ lnL∂θj]The first integral is the covariance of the estimators.Vij ≡ Cov[θˆi, θˆj] ≡∫dx L (θˆi − θi)(θˆj − θj)The second integral is defined as the Fisher information matrix.Iij ≡ E[∂ lnL∂θi∂ lnL∂θj]≡∫dx L∂ lnL∂θi∂ lnL∂θjA more convenient and equivalent expression for Iij can be derived as follows.2Consider the following.E[∂2 lnL∂θi ∂θj]= E[∂∂θi(1L∂L∂θj)]= E[− 1L2∂L∂θi∂L∂θj+1L∂∂θi∂L∂θj]= −E[∂ lnL∂θi∂ lnL∂θj]+ E[1L∂∂θi(L∂ lnL∂θj)]= −Iij +∫dx L1L∂∂θi(L∂ lnL∂θj)= −Iij + ∂∂θi∫dx L∂ lnL∂θj= −Iij + ∂∂θi∫dx L1L∂L∂θj= −Iij + ∂∂θi∂∂θj∫dx L= −Iij + ∂∂θi∂∂θj(1) = −IijThereforeIij ≡ E[∂ lnL∂θi∂ lnL∂θj]≡ E[− ∂2 lnL∂θi ∂θj]One can see that the Fisher information matrix measures the curvature of thelikelihood function in parameters space, averaged over the possible observeddata. Intuitively, this means the larger the value of Iij, the more sharplypronounced the likelihood function is in that region, and the more sensitivethe experiment is to the parameters of the model.Plugging these expressions back into the inequality we derived by us-ing the Cauchy-Schwarz inequality, we have the general expression for theCrame´r-Rao bound.Vij ≥(1 + ∂bi∂θi)(1 +∂bj∂θj)E[− ∂2 lnL∂θi ∂θj] (1)It is often the case that the bias of a well chosen estimator is asymtoticallyzero in the large sample limit. In which case, the bound for the covariance3matrix of unbiased estimators simplifies to the following.Vij ≥(E[− ∂2 lnL∂θi ∂θj])−1The diagonal elements of which give the variance of the estimators.Vii = σ2θˆi≥(E[−∂2 lnL∂θ2i])−1The efficiency of an unbiased estimator is defined asε(θˆi) ≡(E[−∂2 lnL∂θ2i])−1σ2θˆiThe efficiency is therefore less than or equal to one, and equal to one in thecase that the Crame´r-Rao bound becomes an equality.If the estimators are unbiased and efficient (ε = 1), it is evident that thecovariance matrix of the estimators is given by the following.Vij =(E[− ∂2 lnL∂θi ∂θj])−1It can be shown that in the large sample limit, Maximum Likelihood Esti-mators (MLE) are asymptotically unbiased and efficient.Since integrating the second derivatives of the likelihood function over allpossible observables is often prohibitive in practice, one often estimates thisexpectation value by the numerically determined second derivatives evaluatedat the point of maximum likelihood.Vij ≈(− ∂2 lnL∂θi ∂θj∣∣∣∣θ=θˆ)−1Therefore, the variance of the estimators can be estimated byσ2θˆi≈(−∂2 lnL∂θ2i∣∣∣∣θi=θˆi)−1(2)42 ExampleThe purpose of equation 2 is to estimate the variance of an estimator whenan analytic calculation is not practical. In this example, however, we willstudy a case where an analytic calculation of the variance is trivial such thatwe make the validity of equation 2 apparent.Consider an experiment withN repeated measurements that are Gaussiandistributed. The likelihood function is thereforeL =N∏i=11σ√2piexp(−(xi − µ)22σ2)The MLE for the mean, µ, can be found by maximizing the likelihood func-tion, or equivalently, its natural logrhythm.lnL = −N ln(σ√2pi)−N∑i=1(xi − µ)22σ20 =∂ lnL∂µ=N∑i=1(xi − µ)σ2⇒ µˆ = 1NN∑i=1xiTherefore the MLE of µ is just the mean of the sample, as one might expect.Calculating the second derivative of the likelihood gives∂2 lnL∂µ2= −Nσ2Therefore, equation 2 gives the following for the variance of this estimatorσ2µˆ =σ2N(3)which can easily be shown to be the variance of the sample mean of anydistribution as follows.Letx¯ ≡ 1NN∑i=1xi5E [x¯] = E[1NN∑i=1xi]=1NN∑i=1E [xi] =1NN∑i=1µ = µThefore, x¯ is an unbiased estimator of the mean. Now we calculate thevariance.V [x¯] = E[x¯2]− (E [x¯])2= E[(1NN∑i=1xi)(1NN∑j=1xj)]− µ2=1N2(N∑i=1E[x2i]+N∑i 6=jE [xi xj])− µ2For the first sum, note thatV [x] = σ2 = E[x2]− (E [x])2 = E [x2]− µ2⇒ E [x2] = µ2 + σ2For the second sum, recall that the individual measurements xi are madeindependently, thereforeE [x1 x2] = E [x1] E [x2] = µ2ThereforeV [x¯] =1N2(µ2 + σ2 + (N2 −N)µ2)− µ2 = σ2Nin agreement with the variance we calculated using the Crame´r-Rao bound(equation 3), implying that µˆ is indeed an efficient estimator.6

Derivation of the Cramer-Rao Bound

https://philpapers.org/archive/REEDOT-4.pdf

Derivation of the Cramer-Rao Bound

Abstract

Similar works

Full text

Available Versions

PhilPapers