Pynnönen, Seppo

English

Osuva

Working Papers of The University of Vaasa,Department of Mathematics and Statistics,16The joint distribution of a lineartransformation of internally studentizedleast squares residualsSeppo Pynno¨nenPreprint, September 2010University of VaasaDepartment of Mathematics and StatisticsP.O.Box 700, FI-95101 Vaasa, FinlandPreprints available at: http://lipas.uwasa.fi/julkaisu/ewp.htmlThe joint distribution of an arbitrary lineartransformation of internally studentized least squaresresiduals of a linear regressionSeppo Pynnonen, Department of Mathematics and Statistics, University ofVaasa, P.O.Box 700, FI-65101, Vaasa, Finland, Email address: sjp@uwasa.fi,Office phone: +358-6-3248259, Fax: +358-6-3248557Abstract. The term ’studentization’ is commonly used to describe a scaleparameter dependent quantity U by a scale estimate S such that the resultingratio, U/S, has a distribution that is free of from the nuisance unknown scaleparameter. External studentization refers to a ratio in which the nominatorand denominator are independent, while internal studentization refers to a ra-tio in which they are dependent. The advantage of the internal studentizationis that typically one can use a single common scale estimator, while in the ex-ternal studentization every single residual is scaled by different scale estimatorto gain the independence. For normal regression errors the joint distribution ofand arbitrary (linearly independent) subset of internally studentized residualsis well known. However, in some application a linear combination of inter-nally studentized residuals may be useful. The boundedness of an arbitrarylinear combination these residuals is well documented, but the distribution ofa linear combination seems not have been derived in literature. This papercontributes the existing literature by deriving the joint distribution of a gen-eral linear transformation of internally studentized residuals. The distributionof a univariate linear combination and joint distributions of all major versionsof commonly utilized internally studentized residuals are obtained as specialcases. The paper shows that the joint distribution of all linear combinations(including an arbitrary linearly independent subset of residuals) belong to thesame family of distributions. Some test statistics for inference purposes arealso derived.Key words: Linear combination of residuals; normed residuals; distribution ofresiduals; bounds for linear combination of studentized residuals; t-statistic forstudentized residuals; F -statistic for studentized residuals; regression diagnos-ticsMSC: 62E15; 62J051. IntroductionConsider the usual regression model with n observationsy = Xβ + u, (1)1where X is an n×p′ nonstochastic matrix of rank p ≤ p′ < n, y is an n-vector ofobservable responses, β is a p′-vector of slope parameters, and u ∼ N(0, σ2I) isan n-vector of unobservable homoscedastic independent normally distributederrors with (unkown) variance parameter σ2 and I is an n×n identity matrix.The least squares residuals are given byuˆ = Qy= Qu, (2)whereQ = I−X(X′X)−X′ (3)is an n×n idempotent matrix with rank n− p in which the prime denotes thematrix transposition and (X′X)− is a generalized inverse of X′X. The errorsum of squares u′u can be estimated by the residual sum of squaresS2 = uˆ′uˆ = u′Qu. (4)By equation (2) the least squares residuals are linear combinations of normalrandom variables, which implies that uˆ follows a (singular) normal distribu-tion, such thatuˆ ∼ N (0, σ2Q) . (5)Studentization is a common term used to describe division of a scale parameterdependent statistic, say U , by a scale estimate S such that the distributionof the resulting ratio U/S is free from the nuisance scale parameters [see e.g.Margolin (1977)]. Typically U and S are derived from the same data, in whichcase the ratio U/S is called internally studentized if U and S are dependentand externally studentized if they are independent [see, Cook and Weisberg(1982, p.18)].2In least squares regression, the internally studentized residuals are defined asri =uˆis√qii, (6)where uˆi is the ith component of the vector uˆ, s =√uˆ′uˆ/(n− p), and qii isthe ith diagonal element of the matrix Q. Although ri is the familiar ratio of anormally distributed random variable and a square root of a scaled Chi-squarerandom variable, the end result is not a t-distributed random variable. Thereason is that the nominator and the denominator are not independent due tothe fact that uˆ2i ≤ uˆ′uˆ for all i = 1, . . . , n. Furthermore, because qii ≥ 1 + 1/n[Cook and Weisberg (1982, p. 12)], it follows that unlike a t-distributed randomvariable that can assume all real values, r2i ≤ n− p.Stefansky (1972), Ellenberg (1973), and Dı´az-Garc´ıa and Gutie´rrez-Ja´imez(2007) derived the joint distribution of an arbitrary (nonsingular) subset of theinternally studentized residuals defined in (6). Beckman and Trussel (1974)derive the distribution for a t-statistic for a single ri. The present paper derivesthe joint distribution of an arbitrary non-singular linear transformation ofinternally studentized residuals.2. The Main ResultDefinition 1 Let M be an m× n matrix. We callrM =1sMuˆ (7)a studentized linear transformation of least squares residuals uˆ, where s =√uˆ′uˆ/(n− p). Furthermore, we call the transformation nonsingular if thematrix MQM′ is positive definite, in which case m ≤ n− p.All the major classes of residuals defined in literature can be obtained as3special cases of (7). For example, consider in addition to the internally stu-dentized residuals defined in equation (6), also the following alternatives [see,e.g., Chatterjee and Hadi (1988) and Lloynes (1979)]:Normalized residuals:uˆi/√uˆ′uˆ. (8)Standardized residuals:uˆi/s. (9)Abrahamse-Koerts residuals (normalized):B′uˆ/√uˆ′uˆ, (10)where B is an n × n matrix defined in Abrahamse and Koerts (1971),satisfying uˆ′BB′uˆ = uˆ′uˆ.Consider an arbitrary linearly independent subset Im = {i1, i2, . . . , im} ⊂{1, . . . n}, m ≤ n − p of the above classes of residuals. It is easily seenthat each of these is a special case of the linear transformation defined inequation (7). For the purpose, define MI as a m × n matrix in which eachrow j = 1, . . . ,m is a 1 × n vector with element ij = 1 and zeros elsewhere,ij ∈ Im. Furthermore, let D−1/2 denote an n×n diagonal matrix with elements(q11)−1/2, . . . , (qnn)−1/2. Then a set, Im, of internally studentized residuals, de-fined in equation (6), is obtained by defining in equation (7), M = MID−1/2,a set of normalized residuals, defined in equation (8), is obtained by definingM = (n − p)−1/2MI , a set of standardized residuals, defined in equation (9),is obtained by defining M = MI , and a set of Abrahamse-Koerts residuals areobtained, defined in equation (10), by defining M = (n− p)−1/2MIB′.The following lemma is useful.4Lemma 1 Let M be a non-singular transformation as given in Definition 1,thenVM = uˆ′uˆ− uˆ′M′(MQM′)−Muˆ (11)is distributed as σ2χ2(n− p− q), where q = rank(MQ) andUM = uˆ′M′(MQM′)−Muˆ (12)is distributed as σ2χ2(q).Proof. By equation (2), uˆ = Qu, where Q is an indempotent matrix withtr Q = n− p. Thus VM in (11) can be written asVM = u′ (Q−QM′(MQM′)−MQ)u. (13)Because u ∼ N(0, σ2I), VM is σ2χ2(n − p − q) distributed if and only if thematrix, Q−QM′(MQM′)−MQ, in the quadratic form in (13) is idempotentof rank n−p−q. Direct multiplication of the matrix Q−QM′(MQM′)−MQby itself shows that it is idempotent (the only trick is to show that the matrixQM′(MQM′)−MQM′(MQM′)−MQ equals QM′(MQM′)−MQ, which canbe easily shown by taking the difference of these matrices and multiplying byitself to find out that the product is zero). The rank of an idempotent matrixis the same as its trace, which in this case istr(Q−QM′(MQM′)−MQ) = tr Q− tr (QM′(MQM)−MQ)= n− p− tr ((MQM′)−MQM′)= n− p− q,because (MQM′)−MQM′ is idempotent with trace equalling the rank of thematrix. Thus, the proof that VM ∼ σ2χ2(n− p− q) is complete.In the same manner, writing UM = u′QM′(MQM′)−MQu, we find immedi-ately that QM′(MQM′)−MQ is idempotent and tr (QM′(MQM′)−MQ) =5tr ((MQM′)−(MQM′)) = q, which together with u ∼ N(0, σ2I) imply thatUM ∼ σ2χ2(q), completing the proof.Corollary 1 The studentized residuals defined in equation (7) are boundedsuch thatr′M (MQM′)− rM ≤ n− p. (14)Proof. Let s2 = uˆ′uˆ/(n − p). Then by Lemma 1 and by the definition ofrM in equation (7), 0 ≤ VM = uˆ′uˆ − uˆ′M′(MQM′)−Muˆ = (n − p)s2 −s2r′M (MQM′)− rM , from which the result follows immediately, completingthe proof.Lemma 2 Under the assumptions of Lemma 1, VM and Muˆ are independent.Proof. Because Muˆ = MQu is a normal random vector and VM is thequadratic form of the normal random vectoruM =(Q−QM′(MQM′)−MQ)u, (15)a sufficient condition for the independence of Muˆ and VM is that MQu anduM , defined in equation(15), are uncorrelated. This is the case if the defin-ing matrices, MQ and Q−QM′(MQM′)−MQ, of these random vectors areorthogonal. NowMQ(Q−QM′(MQM′)−MQ) = MQ−MQM′(MQM′)−MQ = P.It is straightforward to show that PP′ = 0, which implies that P = 0, fromwhich the orthogonality and hence uncorrelation follow, completing the proofof the lemma.6Lemma 2 implies immediately:Corollary 2 Under the assumptions of Lemma 2 the chi-square random vari-ables VM and UM , defined in equations (11) and (12), respectively, are inde-pendent.With these results we can derive the main result of this paper.Theorem 1 Under the assumptions of the linear regression model in (1), thejoint distribution of the nonsingular studentized linear transform of residuals,defined in equation (7) when m < n− p, isfrM (r) =Γ [(n− p)/2] |MQM′|−1/2(pi(n− p))m/2 Γ [(n− p−m)/2](16)×(1− r′ (MQM′)−1 rn− p) 12(n−p−m)−1, r′ (MQM′)−1 r ≤ n− p,where Γ(·) is the Gamma function.Proof. Under the nonsingularity assumption MQM′ is positive definite, theinverse (MQM′)−1 exists and that r = m, i.e., the rank of the matrix is m.The bounds, r′ (MQM′)−1 r ≤ n−p, follow from Corollary 1. The rest can beproceeded analogously to Ellenberg (1973). That is, due to the χ2-distributionresult of VM in Lemma 1, normality of uM = Muˆ, and independence of VMand uM by Lemma 2, their joint density is the product of their appropriatedensities, resulting tofuM ,VM (x, v) =1pim2 (2σ2)12(n−p)|MQM′| 12Γ[(n− p−m)/2]v12(n−p−m)−1× exp{− 12σ2[x′(MQM′)−1x+ v]}. (17)7Define next the transformationsr = x/√y/(n− p)y = x′(MQM′)−1x+ vThe Jacobian of this transformation is(yn− p) 12m. (18)Using these, the joint density of rM and S2 = uˆ′uˆ becomesfrM ,S2(r, y) =(yn− p)m2fuM ,VM(r√y/(n− p), y − x′(MQM′)−1x)=|MQM′|− 12(pi(n− p))m2 Γ[(n− p−m)/2]×(1− r′(MQM′)−1rn− p) 12(n−p−m)−1× 12(n−p)/2σ2( yσ2) 12(n−p)−1e−y2σ2 .Introducing a further transformation z = y/σ2 with Jacobian σ2 and integrat-ing with respect to z yields finally the marginal density of rM given in (16),which completes the proof of the theorem.Theorem 1 gives the joint distribution of the studentized nonsingular trans-formation, rM , when m < n− p. The next theorem gives the distribution fora nonsingular studentized transformation with m = n− p.Theorem 2 Under the assumption of Theorem 1, when m = n− p,rMd=√n− p M˜U (19)where U is uniformly distributed (n− p)× 1 random vector on the unit sphereSn−p = {x ∈ Rn−p : x′x = 1}, d= means that the random variables have8the same distribution, and M˜ = MHm is an m × m matrix in which Hmare the m = n − p columns of the eigenvectors of Q that correspond the uniteigenvalues, such that M˜M˜′ = MQM′.Proof: A proof is given in Pynno¨nen (2010b).3. ApplicationsIn particular, selecting m = 1 such that M becomes a row vector vector,Theorem 1 implies that the density function of the distribution of an arbitrary(nonsingular) linear combination rm = m′uˆ, where m is an n × 1 vector ofreal numbers satisfying m′Qm > 0, isfrm(r) =Γ[(n− p)/2](m′Qm)−1/2Γ[(n− p− 1)/2]Γ[1/2]√n− p(1− r2(n− p)m′Qm) 12(n−p−1)−1(20)for |r| ≤√(n− p)m′Qm. Thus, r2m/[(n− p)m′Qm] follows a Beta distribu-tion with parameters 1/2 and (n− p− 1)/2. Setting the ith component in mequal to 1 and all others equal to zero gives the density function for a singleinternally studentized residual ti = uˆi/s, where s =√uˆuˆ/(n− p),fti(r) =Γ[(n− p)/2](qii)−1/2Γ[(n− p− 1)/2]Γ[1/2]√n− p(1− r2(n− p)qii) 12(n−p−1)−1, (21)|r| ≤√(n− p)qii, where qii is the ith diagonal element of Q.Again, r2i /[(n − p)qii] follows a Beta distribution with parameters 1/2 and(n−p−1)/2. Thus, the distributions of single internally studentized residualsand their arbitrary linear combinations belong to the same family of distribu-tions such that through simple transformations they are identically distributedas the Beta distribution with parameters 1/2 and (n−p−1)/2. This obviouslyfacilitates inference based on individual residuals or their linear combinations.9Moreover, as is shown below, the situation can be further facilitated by intro-ducing a transformation that leads to a common t-distribution.Individual residuals are typically used as diagnostic tools for model checkingand testing for outliers and influential observations [see, Cook and Weisberg(1982)]. A potential field of applications of using linear combinations of in-ternally studentized residuals is for example in testing for various seasonalpatterns in the residuals of a time series regression. Typically seasonal ef-fects are tested using appropriate dummy-variables, which can be shown to beequivalent of using a statistic based on an externally studentized sum of resid-uals. Perhaps a more concrete application, where the distribution results ofthis paper give additional insight is in the field of event studies in financial eco-nomics [an excellent review is in Campbell, Lo, and MacKinlay (1997, Ch. 4)].The traditional parametric approach is based on external studentized residu-als. However, a non-parametric approach based on Wilcoxon-type rank sums,suggested by Corrado (1989) for testing single period returns, and extendedin Campbell and Wasley (1993) for testing cumulative multi-day rank sums,is gaining increasingly popularity. The used test statistics are obviously of thetype of internally studentized residuals where the nominator and denominatorare not independent. Thus, the distribution theory developed in this papercan be readily utilized in examination of the distributional properties of thetest statistics in these cases.Regarding externally studentized residuals in which the ratio of residual andthe estimator of standard deviation are independent, we find thattm = rm(n− p− 1n− p− r2m) 12=m′uˆ/√m′Qm√1n−p−1Vm, (22)where rm = m′uˆ/s√qm, qm = m′Qm, and Vm = uˆ′uˆ− (m′uˆ)2/(m′Qm), is10a t-distributed random variable with n− p− 1 degrees of freedom by virtue ofLemma 1, Lemma 2, and m′uˆ/√m′Qm ∼ N(0, σ2) . Again, a special caseof this is a single residual, in which case the right-most expression of (22) canbe written asti =uˆis(i)√qii(23)wheres2(i) =(n− p)s2 − uˆ2i /qiin− p− 1 (24)is the residual means square from a sample with the ith observation removedfrom the regression [Beckman and Trussell (1974)]. By equation (22), therelationship between the externally studentized residual, ti, and the internallystudentized residual, ri, given in equation (6), isti = ri(n− p− 1n− p− r2i) 12(25)[c.f. Cook and Weisberg (1980, p. 20)]. The end result is that an arbitrary(nonsingular) linear function of internally studentized residuals can be trans-formed to a t-statistic with n − p − 1 degrees of freedom. Thus, utilizing thesimple transformation in (22), rather than relying on the Beta distribution thefamiliar t-distribution can be used instead in the related statistical inference.Finally, we note that for the multivariate applications Lemma 1 and Corol-lary 2 imply thatt2M =uˆ′M′(MQM′)−1Muˆ/muˆ′ (I−M′(MQM′)−1M) uˆ/(n− p−m) (26)follows the F -distribution with m and n− p−m degrees of freedom. This is ageneralization of the test statistic for testing multiple outlying cases in CookWeisberg (1982, p. 30).A multivariate analog to tm, defined in (22), can be forumlated in terms ofr2M =uˆ′M′(MQM′)−1Muˆs, (27)11such that we can rewrite (26) ast2M =r2M(n− p−m)m(n− p− r2M). (28)4. ConclusionsThis paper derives the joint distribution of a general linear transformation ofinternally studentized least squares residuals from a linear regression. Othertypes of scaled residuals, commonly used in practical applications, can be easilyobtained as special cases by defining the linear transformation appropriately.The distributions of arbitrary subsets as well as marginal distributions of singleresiduals are obtained as special cases from the general distribution by definingthe linear transformation in a suitable manner. The paper discusses also somepotential applications in which the results can be readily applied.ReferencesAbrahamse, A.P.J. and J. Koerts (1971). New estimators in regresion anal-ysis. Journal of the American Statistical Associstion 66, 71–74.Beckman, R.J. and H.J. Trussell (1974). The distribution of an arbitrarystudentized residual and the effects of updating in multiple regression.Journal of the American Statistical Association 69, 199–201.Campbell, Cynthia, J. and Charles E. Wasley (1993). Measuring securityprice performance using NASDAQ returns. Journal of Financial Eco-nomics 33, 73–92.Campbell, John Y., Andrew W. Lo, and A. Craig MacKinlay (1997). TheEconometrics of Financial Markets. Princeton, NJ: Princeton UniversityPress.12Chatterjee, Samprit and Ali S. Hadi (1988). Sensitivity Analysis in LinearRegression, Wiley, New York.Corrado, Charles, J. (1989). A nonparametric test for abnormal security-price performance in event studies. Journal of Financial Economics 23,385–395.Cook, Dennis R. and Sanford Weisberg (1982). Residuals and Influence inRegression. Chapman and Hall.Dı´az-Garc´ıa, Jose´ A. and Ramo´n Gutie´rrez-Ja´imez (2007). The distributionof residuals from a general elliptical linear model. Journal of StatisticalPlanning and Inference 137, 2347–2354.Ellenberg, Jonas K. (1973). The joint distribution of the standardized leastsquares residuals from a general linear regression. Journal of the Ameri-can Statistical Association 68, 941–943.Loynes, R.M. (1979). A note on Prescott’s upper bound for normed residuals.Biometrika 66, 387–389.Margolin, Barry H. (1977). The distribution of internally studentized statis-tics via Laplace transform inversion. Biometrika 64, 573–582.Pynno¨nen, Seppo (2010). Joint distribution of a linear transformation of OLSregression residuals with general spherical error distributio. WorkingPaper, department of Mathematics and Statistics, University of VaasaStefansky, Wilhelmine (1972). Rejecting outliers in fractional designs. Tech-nometrics 14. 469–479.13

The joint distribution of a linear transformation of internally studentized least squares residuals

https://osuva.uwasa.fi/bitstream/handle/10024/9903/mat16.pdf?sequence=2&isAllowed=y

The joint distribution of a linear transformation of internally studentized least squares residuals

Abstract

Similar works

Full text

Available Versions

Osuva