We compare the asymptotic covariance matrix of the ML estimator in a nonlinear measurement error model to the asymptotic covariance matrices of the CS and SQS estimators studied in Kukush et al (2002). For small measurement error variances they are equal up to the order of the measurement error variance and thus nearly equally efficient

Kukush, Alexander

Schneeweiß, Hans

English

Universität München: Elektronischen Publikationen

Relative Efficiency of Maximum Likelihood and Other Estimators in a Nonlinear Regression Model with Small Measurement Errors

Open Access LMU

Kukush, Schneeweiss:Relative Efficiency of Maximum Likelihood and OtherEstimators in a Nonlinear Regression Model with SmallMeasurement ErrorsSonderforschungsbereich 386, Paper 396 (2004)Online unter: http://epub.ub.uni-muenchen.de/ProjektpartnerRelative Efficiency of MaximumLikelihood and Other Estimators in aNonlinear Regression Model withSmall Measurement ErrorsAlexander KukushDepartment of Mathematics,Taras Shevchenko University01601 Kiew, UkraineHans SchneeweissDepartment of Statistics,University of MunichAkademiestr. 1, 80799 Mu¨nchen, GermanyDiscussion Paper 396, 2004AbstractWe compare the asymptotic covariance matrix of the ML estima-tor in a nonlinear measurement error model to the asymptotic covari-ance matrices of the CS and SQS estimators studied in Kukush et al(2002). For small measurement error variances they are equal up tothe order of the measurement error variance and thus nearly equallyefficient.Keywords: Measurement Errors, Maximum Likelihood, Efficiency, Small Er-ror Variance121 IntroductionKukush, Schneeweiss, and Wolf (2002), thereafter KSW, study the relativeefficiency of three estimators in a nonlinear model of the exponential familyf(y|x) = exp{yξ − C(ξ)ϕ+ c(y, ϕ)} (1)where ξ = ξ(x, β) is a known function of the covariate X at the point x withan unknown parameter vector β. The variable X is measured with errors:W = X + σuU, U ∼ N(0, 1), σuU being the measurement error.The naive estimator, which simply substitutes W for X in the originalmodel and then uses Maximum Likelihood, is inconsistent.Two, quite different, consistent estimators can be constructed when σu isknown. The Corrected Score (CS) estimator starts from the likelihood scorefunction ψ(y, x; β, ϕ) of the model and corrects it by constructing a correctedscore function ψc(y, w; β, ϕ) such thatE ψc(Y,W ; β, ϕ|Y,X) = ψ(Y,X; β, ϕ).The estimator β̂CS (together with ϕ̂CS) is then the solution to the equationn∑i=1ψc(yi, wi; β̂CS, ϕ̂CS) = 0,where (yi, wi), i = 1, . . . , n, is a sample of observations.The Structural Quasi Score (SQS) estimator uses the distribution of X,which here is assumed to be a Gaussian distribution. The SQS procedure is3based on the conditional mean and variance functions of Y given W :E(Y |W ) = m(W ; β, ϕ)V (Y |W ) = v(W ; β, ϕ),from which a quasi score function for β and ϕ, is constructed, where theβ-part is of the formψ∗(y, w; β, ϕ) =y −mv∂m∂β.The estimator is then the solution to the equationn∑i=1ψ∗(yi, wi; β̂SQS, ϕ̂SQS) = 0together with a second equation for ϕ̂.Both consistent estimators are asymptotically normally distributed withasymptotic covariance matrices ΣCS and ΣSQS, respectively. It is not knownwhether ΣCS − ΣSQS is positive (semi)definite in general, i.e., whether β̂SQSis more efficient than β̂CS, although this may be expected and has beenproved in the special case of the Poisson model, see Shklyar and Schneeweiss(2002). However, KSW were able to show that for small measurement errorvariances σ2u, both estimators are nearly equally efficient. More precisely:their covariance matrices differ only by a term of order σ4u as σ2u → 0 :ΣCS − ΣSQS = O(σ4u).4In the earlier paper, the ML estimator, being far more complicated thanthe CS and SQS estimators, was completely ignored and was not investi-gated. The purpose of the present paper is to fill this gap and to study theefficiency of ML relative to CS and SQS.Of course, on general grounds, ML is more efficient than CS and SQS.However, it turns out that for small σ2u the ML estimator is approximatelyas efficient as the two other estimators:ΣML = ΣSQS + O(σ4u) = ΣCS + O(σ4u).The proof of this proposition rests on heavy algebra. Here we concentrateon the algebra only and leave aside all questions of a rigorous justification ofthe various algebraic manipulations. Generally speaking, the functions C andξ should be smooth and should be regular in the sense that all expectationsthat arise in the course of the arguments exist and that differentiation andforming of expectations are interchangeable. In KSW, exact conditions forthis to hold were given for the SQS and CS estimators. The correspondingconditions for the ML estimator should be quite similar. But we do not gointo these details.In the next section, we state the model and derive its likelihood. Sec-tion 3 gives an expansion of the model density in terms of powers of themeasurement error variance σ2u, Section 4 does the same for the integrand ofthe information matrix, and Sections 5 and 6 evaluate the various terms ofthis expansion, where it can be seen how the normality assumptions of the5model are utilized. Section 7 has the main result; it presents an asymptoticexpression for small σ2u of the covariance matrix of the ML estimator of βand thereby proves the approximate efficiency of SQS (and CS) for smallσ2u. Section 8 extends this result to the case of unknown nuisance parameters,which have to be estimated along with the parameter vector β of interest.Section 9 has some concluding remarks.2 The model and its likelihoodWe start from model (1) which specifies the conditional density (with respectto a fixed σ-finite measure on the Borel σ-field of R) of a response variableY given the covariate X. The density belongs to an exponential family withcanonical parameter ξ, which depends on X and the parameter vector β tobe estimated. For simplicity, the dispersion parameter ϕ is assumed to beknown. In Section 8 we will analyse the model without assuming ϕ known.The variable X is assumed to be normally distributed:X ∼ N(μ, σ2)with parameters μ and σ2, which, for simplicity, are assumed to be known.(They can easily be estimated from the data wi if σ2u is known.) In the end(Section 8) we will drop this assumption. X is unobservable (latent). Insteadwe observe the variable W , which is X together with an additive measurement6error:W = X + σuU, (2)where U ∼ N(0, 1) and U is independent of (X,Y ). σu is assumed to beknown.The joint density of the observable variables (Y,W ) equalsρ(y, w; β) =12πσ∫exp{yξ∗ − C(ξ∗)ϕ+ c(y, ϕ)− (w − μ− σuu)22σ2− u22} du, (3)where here ξ∗ := ξ(w−σuu, β). This is the likelihood function of β given oneobservation (y, w). The ML estimator of β is found by maximizingn∑i=1log ρ(yi, wi; β) (4)with respect to β, where (yi, wi), i = 1, . . . , n , is an i.i.d. sample. Becauseof the integral in (3) the maximization of (4) may be prohibitively difficult.Therefore other estimation techniques have been proposed, which aresimpler to carry out. Among these the Structural Quasi Score (SQS) methodis most prominent, see KSW. It only uses the conditional mean and varianceof Y given W and not the complete likelihood.Nevertheless, we can still evaluate the asymptotic covariance matrix ofthe ML estimator, at least for small σ2u, and compare it to the asymptoticcovariance of the SQS estimator. This is the purpose of the present paper.The asymptotic covariance matrix ΣML of the ML estimator of β is the7inverse of the information matrix Iβ, and this is given byIβ = Eρβ(Y,W ; β) ρtβ(Y,W ; β)ρ2(Y,W ; β), (5)where ρβ :=∂ρ∂β, and t is the transposition sign.Note on Notation: In the sequel we will generally omit the arguments inthe various functions. Thus ρ stands for ρ(Y,W ; β), and C stands for C(ξ). Asto the function ξ, we let ξ := ξ(W,β), i.e. we replace X by W in the functionξ. In some places, e.g. in (1), we take ξ as a function of X rather than of W .We then write ξ := ξ(X, β) and C = C(ξ). The derivatives of C with respectto ξ are denoted by primes. Partial derivatives of ξ and functions of ξ withrespect to the first or second argument of ξ are denoted by the correspondingsubscripts, x or β, respectively. Note that ξβ is column a vector.As usual, a variable like x will be written in capitals to denote therandom variable X or in small letters to denote a realization of X. Theexpectation sign E is understood to operate on the whole term following thesign, so that brackets will not be necessary, terms being separated by + or –signs.Remember that in an exponential familyC′= E(Y |X), ϕC ′′ = V (Y |X), ϕ2C ′′′ = E[(Y − C ′)3|X] (6)83 Expansion of the model density ρfor small σ2uThe sign ≈ always denotes equality up to terms of order σ2u. We haveξ∗ = ξ(w − σuu, β) ≈ ξ − ξxσuu + 12ξxxσ2uu2 (7)andC(ξ∗) ≈ C + C ′(−ξxσuu + 12ξxxσ2uu2) +C ′′2(ξxσuu)2= C − C ′ξxσuu + (C ′ξxx + C ′′ξ2x)σ2u2u2. (8)Denote the exponent in (3) by M . Then using (7) and (8), we haveM ≈ yϕ−1(ξ − ξxσuu + 12ξxxσ2uu2) + c(y, ϕ)−ϕ−1[C − C ′ξxσuu + (C ′ξxx + C ′′ξ2x)σ2u2u2]−(w − μ)22σ2+(w − μ)σuuσ2− σ2uu22σ2− u22:= A + Bσuu + Dσ2u2u2 − u22withA := ϕ−1(yξ − C) + c(y, ϕ)− (w − μ)22σ2(9)B := −ϕ−1(y − C ′)ξx + V (10)D := ϕ−1[(y − C ′)ξxx − C ′′ξ2x]−1σ2, (11)whereV :=w − μσ2, (12)9and thereforeeM = eA−u22 exp(Bσuu + Dσ2u2u2)≈ eA−u22 [1 + Bσuu + (D + B2)σ2u2 u2].Finally,ρ =12πσ∫eMdu≈ eA√2πσ1√2π∫e−u22 [1 + Bσuu + (D + B2)σ2u2u2]du=eA√2πσ[1 + (D + B2)σ2u2]. (13)4 Expansion of the integrand of the informa-tion matrixFirst note, see (9), thatAβ = ϕ−1(y − C ′)ξβ.Therefore, by (13),ρβ ≈ eA√2πσ[ϕ−1(y − C ′)ξβ{1 + (D + B2)σ2u2}+ (Dβ + 2BBβ)σ2u2]andρβρ≈ϕ−1(y − C ′)ξβ + [ϕ−1(y − C ′)(D + B2)ξβ + Dβ + 2BBβ]σ2u21 + (D + B2)σ2u2≈ ϕ−1(y − C ′)ξβ + (Dβ + 2BBβ)σ2u210Finally, the integrand of the information matrix isρβρtβρ2≈ ϕ−2(y − C ′)2ξβξtβ +σ2u2G (14)withG := ϕ−1(y − C ′)[(Dβ + 2BBβ)ξtβ]s, (15)where the subscript S denotes the symmetrization operator:As := A + At5 Evaluation of GWe first evaluate the derivatives of B and D from (10) and (11):ϕBβ = −(y − C ′)ξxβ + C ′′ξxξβϕDβ = (y − C ′)ξxxβ − C ′′(ξxxξβ + 2ξxξxβ)− C ′′′ξ2xξβConsequently,ϕ(Dβ + 2BBβ) = C′′(2ξxξβV − ξxxξβ − 2ξxξxβ)− C ′′′ξ2xξβ+(y − C ′)(ξxxβ − 2ϕ−1C ′′ξ2xξβ − 2V ξxβ)+2ϕ−1(y − C ′)2ξxξxβ. (16)Substitution (16) in (15) we getϕ2G = 2(y − C ′)[C ′′(2ξxξβξtβV − ξxxξβξtβ − ξx(ξxβξtβ)s)− C ′′′ξ2xξβξtβ]+(y − C ′)2[−2(ξxβξtβ)sV + (ξxxβξtβ)s − 4ϕ−1C ′′ξ2xξβξtβ]+2ϕ−1(y − C ′)3ξx(ξxβξtβ)s. (17)116 Expansion of E(Y − C ′)khAs can be seen from (17), G consists of terms of the form (Y − C ′)kh, k =1, 2, 3, where h is a function of W . The same is true for the other term inthe integrand of the information matrix, see (14). We therefore investigatethe expectation of these terms for alternative values of k and expand themin terms of powers of σ2u. However, it is only for k = 2 that we need anexpansion up to the order of σ2u. For k = 1 and 3 we only need to know thefirst term in the expansion.We first expand ξ:ξ ≈ ξ + ξxσuU + ξxxσ2u2U2,where ξ := ξ(X, β) etc.; see the note on notation at the end of Section 2. Itfollows thatC ′ ≈ C ′ + C ′′(ξxσuU + ξxxσ2u2U2) + C′′′ξ2xσ2u2U2. (18)Now, with k = 1,E(Y − C ′)h ≈ E[Y − C ′ − C ′′(ξxσuU + ξxx σ2u2U2)− C ′′′ξ2x σ2u2U2]∗[h + hxσuU + hxx σ2u2 U2].Due to (6), E(Y − C ′|X) = 0, and as U is independent of X and Y andU ∼ N(0, 1), we thus haveE(Y − C ′)h = O(σ2u). (19)12In a similar way we can now treat the case k = 2. Using (18) and (6),we first getE(Y − C ′)2h ≈ EϕC ′′h + σ2u2E(ϕC′′hxx + 2C′′2ξ2xh). (20)We need to express this result as a function of the observable W rather thanthe latent X. For this remember that because of (2) and the joint normalityof X and U we haveX = W − σ2uσ2 + σ2u(W − μ) + τN,τ 2 =σ2uσ2σ2 + σ2u,where N is a standard Gaussian variable independent of W , see KSW. Upto the order of σ2u we haveσ2uσ2 + σ2u≈ σ2uσ2and consequently, with V from (12),X ≈ W − σ2uV + τN.For any function g (with some regularity properties) we therefore have, be-cause of τ 2 ≈ σ2u,Eg(X) ≈ E[g(W ) + g′(W )(τN − σ2uV ) +12g′′(W )σ2uN2]or, for short, with g := g(X) and g := g(W ),Eg ≈ Eg + E(−2g′V + g′′)σ2u2. (21)13This can be simplified by using the following lemma, which is proved in theappendix, see also KSW, equation (100).LemmaFor any function f(W ) (with some regularity properties) and for V from(12) we haveEf(W )V = (1 +σ2uσ2)Ef ′(W ).Applying this lemma to (21), we getEg ≈ Eg − σ2u2Eg′′. (22)In this way we can transform expectations of functions of X into expectationsof functions of W .We can apply this result to (20) and obtainE(Y − C ′)2h ≈ EϕC ′′h + σ2u2E[−ϕ(C ′′h)xx + ϕC ′′hxx + 2C ′′2ξ2xh].With(C ′′h)xx = C ′′hxx + C ′′′(2ξxhx + ξxxh) + C(4)ξ2xhwe finally haveE(Y − C ′)2h ≈ ϕEC ′′h+σ2u2E[2C ′′2ξ2xh− ϕC ′′′(2ξxhx + ξxxh)− ϕC(4)ξ2xh].(23)14Now for the case k = 3. Here we need only the first term of the expansion.Using (18) an (6), we haveE(Y − C ′)3h = ϕ2EC ′′′h + O(σ2u).This can again be transformed into an expression with observables by using(22):E(Y − C ′)3h = ϕ2EC ′′′h + O(σ2u). (24)7 Expansion of Iβ and ΣMLUsing (19), (23), and (24), we can now expand σ2u2EG from (17) in terms ofpowers of σ2u:σ2u2EG ≈ ϕ−1σ2u2E[C ′′{−2(ξxβξtβ)sV + (ξxxβξtβ)s − 4ϕ−1C ′′ξ2xξβξtβ}+2C ′′′ξx(ξxβξtβ)s].With the help of the lemma this can be simplified toσ2u2EG ≈ −σ2u2ϕ−1E[C ′′{(ξxxβξtβ)s + 4ξxβξtxβ}+ 4ϕ−1C ′′2ξ2xξβξtβ]. (25)In a similar way we also expand the expectation of the first term on theright-hand side of (14):ϕ−2E(Y − C ′)2ξβξtβ ≈ ϕ−1{EC ′′ξβξtβ +σ2u2E[2ϕ−1C ′′2ξ2xξβξtβ−C ′′′{2ξx(ξβξtβ)x + ξxxξβξtβ} − C(4)ξ2xξβξtβ]}.(26)15Finally we find an expansion for the information matrix Iβ, defined in (5).According to (14), this is just the sum of (25) and (26).Iβ ≈ ϕ−1{EC ′′ξβξtβ−σ2u2E[2ϕ−1C ′′2ξ2xξβξtβ + C′′((ξxxβξtβ)s + 4ξxβξtxβ)+C ′′′(2ξx(ξβξtβ)x + ξxxξβξtβ)+C(4)ξ2xξβξtβ]}=: ϕ−1(S − σ2u2Q).The asymptotic covariance matrix of β̂ML is then found to beΣML = ϕ(S−1 +σ2u2S−1QS−1) + O(σ4u).Noting that(ξxxβξtβ)s + 4ξxβξtxβ = (ξβξtβ)xx + 2ξxβξtxβand2ξx(ξβξtβ)x + ξxxξβξtβ = 2(ξxξβξtβ)x − ξxxξβξtβ,we see that ΣML equals the expression for ΣSQS in (21) of KSW up to theorder of σ2u, i.e.:ΣML = ΣSQS + O(σ4u).168 Nuisance parametersUp to now we assumed the nuisance parameters ϕ, μ, and σ2 to be known.We shall now drop this assumption.Let ν := (ϕ, μ, σ2)t be the vector of the nuisance parameters and θ :=(βt, νt)t the vector of all unknown parameters of the model. Suppose θ isestimated by Maximum Likelihood. The covariance matrix of θ̂ is then givenas the inverse of the information matrix of θ, which can be decomposed inthe following wayIθ =(Iβ FF t Iν),where Iβ is as in Section 7, Iν is the information matrix of ν:Iν = Eρ−2ρνρtν ,andF = Eρ−2ρβρtν .By arguments as in the previous sections but without the need to computeall the terms, it is seen that F can be expanded asF ≈ F0 + F1σ2u2.In the error-free model (i.e., σ2u = 0), the ML estimators of β, ϕ, μ, and σ2 areasymptotically independent. Thus, in this case, F = 0, and therefore, in the17error-ridden model, F0 = 0 and thus F = O(σ2u). In addition, in the error-freemodel, Iβ and Iν are positive definite and so, in the error-ridden model, Iβand Iν are also positive definite, at least for small σ2u. It then follows thatI−1θ =(I−1β −I−1β FI−1ν−I−1ν F tI−1β I−1ν)+ O(σ4u).Considering only the upper left corner of this matrix, we see that the covari-ance matrix of β̂ML is now given byΣML = I−1β + O(σ4u).But according to our main result of the previous section,I−1β = ΣSQS + O(σ4u).Therefore we have again, even in the presence of nuisance parameters,ΣML = ΣSQS + O(σ4u).9 ConclusionIn a measurement error model Maximum Likelihood (ML) methods are oftenextremely difficult to apply. For this reason other estimation techniques thatare easier to apply and yet lead to consistent estimators have been proposed.Among these Structural Quasi Score (SQS) and Corrected Score (CS) meth-ods are most prominent. These methods are, however, less efficient than ML.Fortunately, it can be shown that for small error variances these three meth-ods are almost equally efficient. This result is a further justification for theuse of the simpler methods in practical applications.18ML and SQS, as far as these methods have been developed here, bothrely on the assumption that the latent covariate is normally distributed. De-viations from normality lead to biased estimators, see Cheng and Schneeweiss(2003). One can, however, modify the ML and SQS methods so that theytake into account a (finite) mixture of normal distributions instead of justone normal distribution, for SQS see Cheng and Schneeweiss (2003). Thispossibility renders these methods rather flexible in so far as any (continuous)distribution can be approximated by a mixture of normals.The near equal efficiency result was proved for a scalar covariate X (butfor a vector valued parameter β). One should be able to prove the same resultfor a vector of covariates each measured with measurement errors.AppendixProof of the Lemma in Section 6 :First note that W ∼ N(μ, σ2w) with σ2w = σ2 + σ2u. Therefore, with Vfrom (12),Ef(W )V = Ef(W )W−μσ2wσ2wσ2= σ2wσ21√2πσw∫f(w)w−μσ2we− (w−μ)22σ2w dw,which by partial integration is equal to19Ef(W )V = σ2wσ21√2πσw∫f ′(w)e− (w−μ)22σ2w dw= σ2wσ2Ef ′(W )= (1 + σ2uσ2)Ef ′(W ).References[1] Kukush, A., Schneeweiss, H., Wolf, R. (2002): Comparing different esti-mators in a nonlinear measurement error model. Discussion Paper 244,SFB 386, University of Munich.[2] Shklyar, S., Schneeweiss, H. (2002): A comparison of asymptotic covari-ance matrices of three consistent estimators in the Poisson regressionmodel with measurement error. Discussion Paper 283, SFB 386, Uni-versity of Munich. To appear in Journ. Multivariate Analysis.[3] Cheng, C.L., Schneeweis, H. (2003): Bias of the quasi score estimatorof a measurement error model under misspecification of the regressordistribution. Discussion Paper 340, SFB 386, University of Munich.

A comparison of asymptotic covariance matrices of three consistent estimators in the Poisson regression model with measurement error. Discussion Paper 283, SFB 386,U n i -versity of Munich. To appear in Journ. Multivariate Analysis.

Bias of the quasi score estimator of a measurement error model under misspeciﬁcation of the regressor distribution. Discussion Paper 340, SFB 386,

Comparing diﬀerent estimators in a nonlinear measurement error model. Discussion Paper 244, SFB 386,

https://epub.ub.uni-muenchen.de/1766/1/paper_396.pdf

Relative Efficiency of Maximum Likelihood and Other Estimators in a Nonlinear Regression Model with Small Measurement Errors

Abstract

Similar works

Full text

Available Versions

Universität München: Elektronischen Publikationen

Open Access LMU