This paper presents a general loss function under quadratic loss structure and discusses the comparison of risk functions associated with the unbiased least squares and biased Stein-rule estimators of the coefficients in a linear regression model

Heumann, Christian

Shalabh

Toutenburg, Helge

English

EconStor (ZBW Kiel)

Risk performance of Stein-rule estimators over the least squares estimators of regression coefficients under quadratic loss structures

Open Access LMU

Shalabh, Toutenburg, Heumann:Risk Performance Of Stein-Rule Estimators Over TheLeast Squares Estimators Of Regression CoefficientsUnder Quadratic Loss StructuresSonderforschungsbereich 386, Paper 495 (2006)Online unter: http://epub.ub.uni-muenchen.de/ProjektpartnerRisk Performance Of Stein-Rule EstimatorsOver The Least Squares Estimators OfRegression Coefficients Under Quadratic LossStructuresShalabh∗, H. Toutenburg†and C. Heumann‡Discussion Paper 493 SFB386, University of Munich, MunichNovember 3, 2006AbstractThis paper presents a general loss function under quadratic lossstructure and discusses the comparison of risk functions associatedwith the unbiased least squares and biased Stein-rule estimators ofthe coefficients in a linear regression model.∗Department of Mathematics and Statistics, Indian Institute of Technology, Kanpur208016, India. E-mail: shalab@iitk.ac.in, shalabh1@yahoo.com (Corresponding Author)†Department of Statistics, Ludwig–Maximilians–University Munich, Ludwigstr. 33,80539 Munich, Germany. E-mail: helge.toutenburg@stat.uni-muenchen.de‡Department of Statistics, Ludwig–Maximilians–University Munich, Ludwigstr. 33,80539 Munich, Germany. E-mail: chris@stat.uni-muenchen.de11 IntroductionPerformance of any estimation procedure for the parameters in a model isgenerally evaluated by either the goodness of fitted model or the concen-tration of the estimates around the true parameter values. In practice, itmay often be desirable to employ both the criteria simultaneously; see, forinstance, Toutenburg and Shalabh (1996), Shalabh (1995, 2000) and Zell-ner (1994) for some illustrative examples. Accordingly, Zellner (1994) hasintroduced the balanced loss function. As the goodness of fitted model canbe interpreted as the goodness of the predictions for the actual values of thestudy variable within the sample while the concentration of estimates may bemeasured by the goodness of predictions for the average values of the studyvariable within the sample, Shalabh (1995) has presented the predictive lossfunction.In this paper, we present a general loss function of which the loss functionsconsidered by Shalabh (1995) and Zellner (1994) are particular cases, andexpose the unbiased least squares and biased Stein-rule estimators of theregression coefficients. In Section 2 we describe the linear regression modeland present a general loss function under quadratic loss structure. Section3 gives a comparison of the risk functions associated with the least squaresand Stein-rule estimators, and a condition on the characterizing scalar forthe superiority of the Stein-rule estimators over the least squares estimatoris obtained. Several particular cases are also considered. Some concludingremarks are then placed in Section 4. Lastly, the Appendix gives the proofof Theorem.22 Linear Regression Model And The Loss Func-tionLet us consider the following linear model:y = Xβ + σ² (2.1)where y is a n×1 vector of n observations on the study variable, X is a n×pfull column rank matrix of n observations on p explanatory variables, σ is anunknown positive scalar and ² is a n× 1 vector of disturbances.It is assumed that the elements of ² are independently and identically dis-tributed following a distribution with mean 0, variance 1 and third momentγ1 measuring skewness.If β˜ denotes any estimator of β, the goodness of the fitted model is reflectedin the residual vector (Xβ˜ − y). Similarly, the pivotal quantity for measur-ing the concentration of estimates around the true parameter values is theestimation error (β˜ − β). Accordingly, the quadratic loss function for thegoodness of fit of the model is(Xβ˜ − y)′(Xβ˜ − y) (2.2)while the commonly employed loss function for the precision of estimationare(β˜ − β)′(β˜ − β) (2.3)or(β˜ − β)′X ′X(β˜ − β) (2.4)Taking both the criteria of the goodness of fit and precision of estimationtogether, Zellner (1994) has proposed the following balanced loss function:BL(β˜) = ω(Xβ˜ − y)′(Xβ˜ − y) + (1− ω)(β˜ − β)′X ′X(β˜ − β) (2.5)3where ω is a scalar lying between 0 and 1.From the viewpoint of the prediction of the values of the study variable withinthe sample, the loss functions (2.2) and (2.4) can be regarded as arising fromthe prediction of actual values y by Xβ˜ and the prediction of the averagevalues E(y) = Xβ by Xβ˜ respectively. Accordingly, Shalabh (1995) hasdefined a target functionT = ωy + (1− ω)E(y) (2.6)and has presented the following predictive loss functionPL(β˜) = (Xβ˜ − T )′(Xβ˜ − T )= ω2(Xβ˜ − y)′(Xβ˜ − y)+(1− ω)2(β˜ − β)′X ′X(β˜ − β)+2ω(1− ω)(Xβ˜ − T )′X(Xβ˜ − T ) (2.7)where ω is a scalar between 0 and 1. Note that ω = 0 and ω = 1 in (2.6)provides predictions for average and actual values of y. Any other value0 < ω < 1 provides the weight assigned to the actual value prediction andprovides simultaneous prediction of actual and average values of y.Looking at the functional forms of the balanced loss function and the pre-dictive loss function, we propose the following weighted loss function:WL(β˜) = λ1(Xβ˜ − y)′(Xβ˜ − y)+λ2(β˜ − β)′X ′X(β˜ − β)+(1− λ1 − λ2)(Xβ˜ − y)′X(β˜ − β) (2.8)where λ1 and λ2 are scalars characterizing the loss functions.Clearly, the function (2.8) encompasses the loss functions (2.2), (2.4), (2.5)and (2.7) as particular cases. Thus it is fairly general and sufficiently flexible.43 Comparisons Of Least Squares And Stein-Rule EstimatorsThe least squares estimator of β is given byb = (X ′X)−1X ′y (3.1)which is well known for its optimality in the class of linear and unbiasedestimators.If we drop the linearity and unbiasedness, there exist estimators with betterperformance than the least squares estimator under the risk criterion. Onesuch interesting family of nonlinear and biased estimators of β, popularlyknown as Stein-rule estimators is defined byβˆ =[1−(kn− p+ 2)y′ (I −H) yy′Hy]b (3.2)whereH = X(X ′X)−1X ′ (3.3)and k is a positive nonstochastic scalar; see, e.g. Judge and Bock (1978) andSaleh (2006).For comparing the estimators, let us take the criterion as risk, i.e., the ex-pected value of the weighted loss function (2.8).The exact expressions for the risk functions can be derived bur their naturewould be sufficiently intricate. We therefore consider their large sample as-ymptotic approximations. For this purpose, we assume that the explanatoryvariables are asymptotically cooperative, i.e., the limiting form of the matrixn−1X ′X is finite and nonsingular, as n tends to infinity.The large sample asymptotic expressions for the risk functions are derived in5the Appendix and are presented below.Theorem 1 :The risk function of b and βˆ to order O(n−1) are given byR(b) = E [WL(b)]= σ2λ1n− σ2p(λ1 − λ2) (3.4)R(βˆ) = E[WL(βˆ)]= σ2λ1n− σ2p(λ1 − λ2)− σ4knβ′Sβ[(1− λ1 + λ2)(γ1σX¯ ′β + p− 2)− k](3.5)where S = 1nX ′X and X¯ is a p × 1 vector of the means of the observationson the p explanatory variables.Comparing (3.4) and (3.5), it is observed that the Stein-rule estimator hassmaller risk to the order of our approximations, in comparison to the leastsquares estimator whenk < (1− λ1 + λ2)(γ1σX¯ ′β + p− 2)(3.6)provided that(λ1 − λ2) < 1 and(γ1σX¯ ′β + p− 2)> 0 (3.7)or(λ1 − λ2) > 1 and(γ1σX¯ ′β + p− 2)< 0. (3.8)When the distribution of disturbances is symmetric and/or X¯ is a null vec-tor, i.e., the observations on the explanatory variables are taken as deviationsfrom their corresponding means, then the condition (3.6) becomes free fromunknown parameters β and is satisfied when either of the following two con-6ditions holds true:k < (1− λ1 + λ2)(p− 2) and (λ1 − λ2) < 1 if p > 2 (3.9)k < (λ1 − λ2 − 1)(2− p) and (λ1 − λ2) > 1 if p = 1, 2. (3.10)Now let us examine the performance of estimators under some interestingloss functions.3.1 Loss Function: (Xβ˜ − y)′(Xβ˜ − y)This loss function is a particular case of (2.8) with λ1 = 1 and λ2 = 0. It isindeed the residual sum of squares and is the sum of squares of predictionerrors when the aim is to predict the actual values of the study of the studyvariable within the sample.In this case, it is interesting to observe from (3.4) and (3.5) that the leastsquares estimator remains unbeaten by all the Stein-rule estimators irrespec-tive of the nature of the observations on the explanatory variables and thedistribution of disturbances. This matches with the result obtained by Sri-vastava and Shalabh (1996, p.143) on the basis of exact risk expressions.3.2 Loss Function: (β˜ − β)′X ′X(β˜ − β)If we set λ1 = 0 and λ2 = 1 in (2.8), we get this loss function. It is essen-tially the weighted sum of squares of the estimation errors and is the sum ofsquares of the prediction errors when the aim is to predict the average valuesof the study variable within the sample.In this case, the Stein-rule estimators are better than the least squares esti-7mator whenk < 2(γ1σX¯ ′β + p− 2)(3.11)with the rider that the quantity on the right hand side is positive; see alsoVinod and Srivastatave (1995).This condition reduces tok < 2(p− 2); p > 2 (3.12)when the distribution of disturbances is symmetric irrespective of the natureof data on the explanatory variables or X¯ is a null vector whether the distri-butions of disturbances is symmetric or asymmetric.Similarly, the condition (3.11) is satisfied as long as (3.12) holds true pro-vided that γ1 and X¯′β have the same sign, i.e., X¯ ′β is positive for positivelyskewed distributions of disturbances and is negative for negatively skeweddistributions of disturbances. In fact, it is possible to find Stein-rule estima-tors with better performance than the least squares estimator even for p = 1and p = 2 whenγ1X¯′β > 2σ. (3.13)It may be noticed (3.12) is a well-known condition for the superiority of Stein-rule estimators on the basis of exact risk under the normality of disturbances;see, e.g., Judge and Bock (1978) and Saleh (2006).3.3 Loss Function: (Xβ˜)′X(β˜ − β)This loss function is obtained from (2.8) by putting λ1 = λ2 = 0 and canbe regarded as measuring the covariability between the residuals and the8estimation errors. From the viewpoint of prediction within the sample, itis the sum of cross products of errors arising from the prediction of theactual and average values of the study variable by Xβ˜. This loss functionis, however, not interesting because the exact risk (3.4) of the least squaresestimator turns out to be zero which is the risk of Stein-rule estimators toorder O(n−1) which may be negative.3.4 Loss Function: ω(Xβ˜ − y)′(Xβ˜ − y) + (1 − ω)(β˜ −β)′X ′X(β˜ − β)If we put λ1 = ω and λ2 = (1−ω) in (2.8), we get the balanced loss functionproposed by Zellner (1994). This is indeed a convex combination of the sumof squares of the residuals and the weighted sum of squares of the estimationerrors. It can also be interpreted as a convex combination of the two sumsof squares of the prediction errors arising from the prediction of the actualand average values of the study variable within the sample.For 0 ≤ ω < 1, the Stein-rule estimators perform better than the leastsquares estimator whenk < 2(1− ω)(γ1σX¯ ′β + p− 2). (3.14)When γ1 is zero and/or X¯ is a null vector, the condition (3.14) assumes asimple form:k < 2(1− ω)(p− 2); p > 2. (3.15)This serves as a sufficient condition for the superiority of the Stein-rule es-timators over the least squares estimator in case of the asymmetric distrib-utions of the disturbances provided that the skewness coefficient γ1 has the9same sign as X¯ ′β. Further, if (3.14) holds true, one can find Stein-rule es-timators better than the least squares estimator even when there is simplyone or two explanatory variables in the model.It may be observed that the condition (3.14) has been derived by Giles,Giles and Ohtani (1996) by considering the exact risk under the normalityof disturbances; see also Ohtani (1998).3.5 Loss Function: ω2(Xβ˜ − y)′(Xβ˜ − y) + (1 − ω)2(β˜ −β)′X ′X(β˜ − β) + 2ω(1− ω)(Xβ˜ − y)′X(β˜ − β)This loss function is a particular case of (2.8) with λ1 = ω2 and λ2 = (1−ω)2.It is a combination of the sum of squares of the residuals, the weighted sumof squares of the estimation errors and the weighted sum of cross productsof the residuals and the estimation errors. This is also equal to the sumof squares of prediction errors when Xβ˜ is employed for the prediction of aconvex combination of the actual and average values of the study variable;see Shalabh (1995).From (3.4) and (3.5), it is seen that the Stein-rule estimators have smallerrisk in comparison to the least squares estimator whenk < 2(1− ω)(γ1σX¯ ′β + p− 2)(3.16)which is precisely the same as (3.14) obtained from risk comparison underthe balanced loss function. The condition of (3.16) with γ1 = 0 matches withthe condition of Shalabh (1995) on the basis of exact risk; see also Shalabh(1999).104 Some RemarksAppreciating the simultaneous use of the two performance criteria, viz, thegoodness of fitted model and the concentration of estimates around the trueparameter values, for judging the efficiency of any estimation procedure forthe coefficients in a linear regression model, we have presented a general lossfunction using the quadratic loss structure. Several popular loss functionsare found to be the particular cases of it, and thus the properties of lossfunction is fairly general and sufficiently flexible.For the regression coefficient vector, we have considered the unbiased leastsquares and biased Stein-rule estimators. We have compared their perfor-mance according to the risk criterion under the proposed loss function andhave obtained a condition on the characterizing scalar for the superiority ofthe Stein-rule estimates over the least squares estimator.We have not assumed any functional form for the distribution of the distur-bances; we have simply supposed the finiteness of the first three moments.Accordingly, the large sample approximations for risk functions are used forthe purpose of comparison. An interesting observation emerging from ourinvestigations is that the condition on the characterizing scalar for the supe-riority of the Stein-rule estimators over the least squares estimator deducedfrom the exact risks under the normality of disturbances remains valid for avariety of symmetric and asymmetric distributions. Further, the Stein-ruleestimators are found to be quite robust with respect to the choice of lossfunctions.11A AppendixFrom (2.1) and (2.8), we observe thatWL(β˜) = σ2λ1²′²− σ(1 + λ1 − λ2)²′X(β˜ − β) + (β˜ − β)′X ′X(β˜ − β).(A.1)Setting β˜ = b, it is easy to see thatR(b) = E[WL(b)] = σ2λ1n− σ2p(λ1 − λ2) (A.2)which is the result (3.4) of the Theorem.Now, if we writeu = n12X ′²andv = n12(²′²n− 1),we haveb− β = σn12S−1u. (A.3)Next, consider the quantityy′(I −H)y(n− p+ 2)y′Hy =σ2(n+ n12v − u′S−1u)(n− p+ 2)(nβ′Sβ + 2n 12σβ′u+ σ2u′S−1u)=σ2nβ′Sβ(1 +vn12− 1nu′S−1u)(1− p− 2n)−1·(1 +2σβ′uγ12β′Sβ+σ2u′S−1unβ′Sβ)−1.Expanding and retaining terms to order O(n−32 ), we findy′(I −H)y(n− p+ 2)y′Hy =σ2nβ′Sβ+σ2n32β′Sβ(v − 2σβ′uβ′Sβ)+Op(n−2). (A.4)12Substituting (A.3) and (A.4) in (3.2), we find(βˆ − β) = σn12S−1u− σ2knβ′Sββ− σ2kn32β′Sβ[vβ + σ(S−1 − 2β′Sβββ′)u]+Op(n−2).(A.5)Using the distributional properties of ², it is easy to verify thatE(²′A²) = tr(A)E(²′A² · ²) = γ1(I ∗ A)ewhere A is any n × n symmetric matrix with nonstochastic elements, e is an × 1 vector with all elements unity and * denotes the Hadamard productoperator of matrices.Making use of these results and neglecting terms of higher order of smallnessthan O(n−1), we see from (A.5) thatE[²′X(βˆ − β)] = σp− σ2knβ′Sβ[γ1ne′Xβ + σ(p− 2)](A.6)E[(βˆ − β)′X ′X(βˆ − β)] = σ2p− σ3knβ′Sβ[2γ1ne′Xβ + 2σ(p− 2)− σk].(A.7)Setting β˜ = βˆ in (A.1), utilizing the above results and retaining terms toorder O(n−1), we findR(βˆ) = E[WL(βˆ)]= σ2λ1n− σ2p(λ1 − λ2)− σ4knβ′Sβ[(1− λ1 + λ2)( γ1nσe′Xβ + p− 2− k)]which is the result (3.5) of the Theorem.AcknowledgementThe first author gratefully acknowledges the support from Alexander vonHumboldt Foundation, Germany in the form of Humboldt Fellowship.13ReferencesGiles, J.A., Giles, D.E.A. and Ohtani K. (1996): ‘The exact risksof some pre-test and Stein-type regression estimates underbalanced loss’, Communications in Statistics - Theory andMethods, 25, p. 2901-2924.Judge, G.G. and Bock, M.E. (1978): ‘The Statistical Implica-tion of Pre-Test and Stein-Rule Estimators in Econo-metrics’, North Holland.Ohtani K. (1998): ‘The exact risk of a weighted average estima-tor of the OLS and Stein-rule estimators in regression underbalanced loss’, Statistics and Decisions, 16, p. 35-45.Saleh, A.K.E. (2006): ‘Theory of Preliminary Test and Stein-type Estimation with Application’, John Wiley.Shalabh (1995): ‘Performance of Stein-rule procedure for simul-taneous prediction of actual and average values of study vari-able in linear regression model’, Jan Tinbergen Award Paper,Bulletin of the 50th Session of the International StatisticalInstitute, p. 1375-1390.Shalabh (1999): ‘Improving the predictions in linear regressionmodels’, Journal of Statistical Research, 33, p. 33-39.Shalabh (2000): ‘Prediction of values of variables in linear mea-surement error model’, Journal of Applied Statistics, 27, p.475-482.Srivastava, A.K. and Shalabh (1996): ‘Efficiency properties ofleast squares and Stein-rule predictions in linear regression14model’, Journal of Applied Statistical Science, 4, p. 141-145.Toutenburg, H. and Shalabh (1996): ‘Predictive performance ofthe methods of restricted and mixed regression estimators’,Biometrical Journal, 38, p. 951-959.Vinod, H.D. and Srivastava, V.K. (1995): ‘Large sample asymp-totic properties of the double k-class estimators in linear re-gression models’, Econometric Reviews, 14, p. 75-100.15

Risk Performance Of Stein-Rule Estimators Over The Least Squares Estimators Of Regression Coefficients Under Quadratic Loss Structures

http://epub.ub.uni-muenchen.de/1863/1/paper_495.pdf

Risk Performance Of Stein-Rule Estimators Over The Least Squares Estimators Of Regression Coefficients Under Quadratic Loss Structures

Abstract

Similar works

Full text

Available Versions

EconStor (ZBW Kiel)

Open Access LMU