Cataloged from PDF version of article.We consider the behavior of model selection criteria in AR models where the error terms are not normal by varying skewness and kurtosis. The probability of estimating the true lag order for varying degrees of freedom (k) is the interest. For both small and large samples skewness does not effect the performance of criteria under consideration. On the other hand, kurtosis does effect some of the criteria considerably. In large samples and for large values of k the usual asymptotic theory results for normal models are confirmed. Moreover, we showed that for small sample sizes performance of some newly introduced criteria which were not considered in Monte Carlo studies before are better. (C) 1998 Elsevier Science S.A

Zaman, A.

Economics Letters

Bilkent University Institutional Repository

Economics Letters 59 (1998) 17–22Effects of skewness and kurtosis on model selection criteria*Sıdıka Bas¸c¸ı , Asad ZamanDepartment of Economics, Bilkent University, 06533, Bilkent, Ankara, TurkeyReceived 29 July 1997; accepted 4 February 1998AbstractWe consider the behavior of model selection criteria in AR models where the error terms are not normal by varyingskewness and kurtosis. The probability of estimating the true lag order for varying degrees of freedom (k) is the interest. Forboth small and large samples skewness does not effect the performance of criteria under consideration. On the other hand,kurtosis does effect some of the criteria considerably. In large samples and for large values of k the usual asymptotic theoryresults for normal models are confirmed. Moreover, we showed that for small sample sizes performance of some newlyintroduced criteria which were not considered in Monte Carlo studies before are better. Ó 1998 Elsevier Science S.A.Keywords: Model Selection Criteria; AR lag order determination; RobustnessJEL classification: C22; C511. Introduction¨In a Monte-Carlo study Lutkepohl (1985) compares the finite-sample performance of 12 differentidentification approaches for AR models. In his study, Schwarz Criterion (SC) (Schwarz, 1978;Rissanen, 1978) and Hannan-Quinn Criterion (HQC) (Hannan and Quinn, 1979; Quinn, 1980) emergeas being the best among the compared criteria. Akaike Information Criterion (AIC) (Akaike, 1973,¨1974) also performs well. Koreisha and Pukkila (1993) augment Lutkepohl (1985) and show that theperformance of the above mentioned criteria depends on the number of nonzero elements of thematrices of the AR parameters and the maximum possible lag order that is used. These two studiesand some other prior theoretical and empirical studies assume that the distribution of the error termsare normal. Our object in this paper is to relax this assumption and compare the performance of abovementioned criteria, as well as a few others, when the AR process may have nonnormal errors. We dothis by varying skewness and kurtosis. Since it is known that some of the financial data possesskewness and kurtosis it is important for applied econometricians to know the behaviour of variouscriteria under such circumstances.We find that although skewness does not have much effect on the performance of criteria whileestimating the lag order of the AR model, kurtosis effects the performance of some of the criteria. We¨reached same conclusion as Lutkepohl (1985) that SC and HQC are best criteria under normal error*Corresponding author. Tel.: 190 312 2664769; fax: 190 312 2664960/2665140; e-mail: sbasci@bilkent.edu.tr0165-1765/98/$19.00 Ó 1998 Elsevier Science S.A. All rights reserved.PII S0165-1765( 98 )00036-618 S. Bas¸c¸ı, A. Zaman / Economics Letters 59 (1998) 17 –22distributions and large samples. More than that we showed that they are also best for nonnormal errordistributions if the sample size is large. For small sample sizes performance of other criteria which are¨not included in Lutkepohl (1985) are better than SC and HQC under the assumptions of both normaland nonnormal error distributions.2. Criteria for AR lag order selectionAssume that a given set of data X 5 (X , X , . . . ,X )9 is generated by a stationary AR( p) process0 1 T(allowing for nonzero mean a ):0X 5 a 1 a X 1 . . . 1 a X 1 ut 0 1 t21 p t2p twhere the a , i50, . . . p, are unknown parameters and u is an i.i.d error term with commoni tdistribution F. Our object is to find p, the true unknown lag order. It is assumed that there is an a priormaximum order M, so that our estimate for p can be any integer between 0,1, . . . ,M. All prior studiesof the issue, both theoretical and empirical, assume the errors to be normally distributed. In this paper,our goal is to assess the performance of these criteria when errors may be nonnormal. From the theoryof Edgeworth expansions, we know that the first two terms in asymptotic approximations arising fromlack of normality correspond to skewness and kurtosis respectively. Thus a good approach to assessrobustness is to vary skewness and kurtosis and study the behavior of the model selection criteria. Itappears that skewness does not have much effect on the relative performance of the criteria westudied, while kurtosis does have a great effect.We first briefly review the model selection criteria included in this study. To describe the criteria,ˆˆ ˆ ˆ ˆlet a , . . . ,a be the OLS estimates of the parameters of the AR( p) model. Let X 5a 1a X 10 p t 0 1 t212 21 Tˆ ˆ. . . 1a X be the OLS estimate of X . Define the usual variance estimate s 5(T2( p11)) op t2p t p t5p2ˆ(X 2X ) . The first criteria AIC was introduced by Akaike (1973), and has been a very populart t2 2pˆ ]criterion for model selection. Define AIC( p)5ln(s )1 , and estimate p to be that integer betweenp T1, . . . ,M which minimizes the criterion AIC. Shibata (1976) shows that AIC is not consistent butoverestimate p asymptotically with positive probability if M$p. Zaman (1984) shows how toˆcalculate the probability distribution of the estimate p produced by AIC. Shibata (1980) discusses anasymptotic efficiency property of AIC.2ˆHurvich and Tsai (1989) makes a bias correction to the AIC, defined by AIC ( p)5T ln(s )1C pT [11( p /T )] / [12( p12)T ]. They show that it is asymptotically efficient if the true model is infinitedimensional. When the true model is finite dimensional AIC chooses the true lag order most oftenCcompared to other asymptotically efficient criteria.2ˆThe Schwarz Criterion, SC( p)5ln (s )1p ln(T ) /T was introduced by Schwarz (1978) based onpBayesian reasoning. It has the advantage of being consistent over the AIC. This and the Hannan-2ˆ ¨Quinn criterion HQC( p)5ln (s )12p ln(ln(T )) /T both perform well in the Lutkepohl (1985) study,pand hence are included in our study.¨In addition to the AIC which was not part of the Lutkepohl (1985) study, we introduce two otherCcriterion which have not been studied in this context. The criterion PRESS, closely related toQuenouille (1949) jacknife, was first introduced by Allen (1974). This is defined as PRESS( p)5S. Bas¸c¸ı, A. Zaman / Economics Letters 59 (1998) 17 –22 19T 2˜ ˜o (x 2x ) where x , t51, . . . T, is the predicted value of x from an AR( p) model after omittingt51 t t t t1the t-th observation in the process of estimation. The value of p minimizing PRESS is selected.The last criterion included in our study is the often-used sequential F test. We start with the largestˆmodel AR(M), and examine the t statistic of a . If this is insignificant, we drop it and re-estimate anMAR(M21) model. We keep dropping the last coefficient until we get a significant one. In thesimulation study we took the significance level as 5%.3. Simulation resultsTo compare the small sample performance of the criteria discussed in the previous section we havetwo types of processes where one assumes three lags and the other assumes two lags as the true lag2order. a is one. In order to have a stationary process we generate a , i51,2,3, from a uniform0 idistribution in the region (21/2,1 /2). We set M56. For our robustness studies, we use two classes ofiid2error distributions. Let X|&(k,2);x . To study the effect of skewness, we considered u |F, where2k t] ] ]3Œ Œ ŒF is the distribution of (X /(2 k))2 k. This has mean 0, variance 1, and Eu 52 k so that skewnesstincreases with k.Fig. 1a,b below give the probability of correctly estimating p using the various criteria for varyingk. Fig. 1a has sample size 30, representing the small sample situation, and Fig. 1b has sample size200, representing the large sample situation. From the figures we can see that skewness does not havemuch effect on the performance of criteria while estimating the true lag order. Although AIC is theCbest criterion for sample size 30, it becomes one of the worst criteria for sample size 200.Performance of PRESS is similar to AIC . Asymptotically consistent criteria, SC and HQC, are notCdoing that well for sample size 30 but they estimate the true lag most often when we increase samplesize to 200. Performance of AIC and sequential F test are close to each other.iidTo study the effect of kurtosis, let u |F where F is a Student’s t distribution scaled to havet kvariance 1. As k increases, the kurtosis decrease, converging to that of the normal asymptotically.Note that skewness is 0 for this class of distributions. When k is one we have the Slash distributiondefined in Morgenthaler and Tukey (1991) as a normal divided by a Uniform (0,1).Fig. 2a,b below give the probabilities for correctly estimating p. We see from the figures thatkurtosis has a considerable effect on the criteria under consideration, especially on AIC and PRESS.CWhen k is small, that is for heavy-tailed error distributions, performance of PRESS and sequential Ftest are good but as k increase, that is when we get closer to normal distribution, we see a decline intheir performance. Converse is true for AIC . The result for AIC is what is to be expected becauseC CHurvich and Tsai (1989) makes the bias correction under the assumption of normality. Results can beseen more clearly for the case of sample size 200. Performance of other criteria slightly increase as wemove towards normal distribution when sample size is 30. For sample size 200, it seems that kurtosis1There are potential problems which arise from omitting middle observations in time series. We made adjustments to accountfor such problems but found that such adjustments made no difference in the outcome, and hence have chosen to reportresults without any adjustments.2Since the results for three lags are similar to the ones for two lags, we only present in this paper the results for two lags.20 S. Bas¸c¸ı, A. Zaman / Economics Letters 59 (1998) 17 –22Fig. 1. (a) Probability of estimating the true lag order (true lag order52, sample size530, a skewed distribution). (b) Probability ofestimating the true lag order (true lag order52, sample size5200, a skewed distribution).S. Bas¸c¸ı, A. Zaman / Economics Letters 59 (1998) 17 –22 21Fig. 2. (a) Probability of estimating the true lag order (true lag order52, sample size530, t distribution). (b) Probability of estimating thetrue lag order (true lag order52, sample size5200, t distribution).does not effect these criteria. Once again SC and HQC are performing badly when sample size is 30but they are the best ones when we increase sample size to 200. PRESS and AIC are the worstCcriteria for sample size 200.22 S. Bas¸c¸ı, A. Zaman / Economics Letters 59 (1998) 17 –224. ConclusionsTo conclude we can say that skewness does not effect the probability of estimating the true lagorder but kurtosis effects it especially for the criteria PRESS and AIC . When the sample size is smallCand error distributions are not heavy-tailed (corresponding to large values of degrees of freedom, k)we suggest the use of AIC for lag order selection. On the other hand, for heavy-tailed errorCdistributions PRESS and sequential F test perform very well. For large samples SC and HQC aredefinitely the best criteria to be used.We also studied forecasting performance based on models selected by the various criteria. This doesnot correlate very well with the ability of the model to pick out the true p. In addition, we also studiedthe probability distribution of the estimate of p. This reveals that PRESS has higher probabilities formodels size m satisfying m$p relative to the other criteria. If the maximum size M is reduced theperformance of PRESS improves substantially. Results for the normal distribution are very similar tothose for t .100References`Akaike, H., 1973. Information theory and an extension of the maximum likelihood principle. In: Petrov, B.N., Csaki, F., 2nd` `International Symposium on Information Theory. Akademiai Kiado, Budapest, pp. 267–281.Akaike, H., 1974. A new look at the statistical model identification. IEEE Transactions on Automatic Control AC 19,716–723.Allen, D.M., 1974. The relationship between variable selection and data augmentation and a method for prediction.Technometrics 16, 125–127.Hannan, E.J., Quinn, B.G., 1979. The determination of the order of an autoregression. Journal of the Royal Statistical SocietyB41, 190–195.Hurvich, C.M., Tsai, C.L., 1989. Regression and time series model selection in small samples. Biometrika 76 (2), 297–307.Koreisha, S.G., Pukkila, T., 1993. Determining the order of a vector autoregression when the number of component series islarge. Journal of Time Series Analysis 14 (1), 47–69.¨Lutkepohl, H., 1985. Comparison of criteria for estimating the order of a vector autoregressive process. Journal of TimeSeries Analysis 6 (1), 35–52.Quenouille, M., 1949. Approximate tests of correlation in time series. J. Royal Statistical Society Series B11, 18–84.Morgenthaler, S., Tukey, J.W. (Eds.), 1991. Configural Polysampling: A Route to Practicle Robustness. Wiley, New York.Quinn, B.G., 1980. Order determination for a multivariate autoregression. Journal of the Royal Statistical Society B42,182–185.Rissanen, J., 1978. Modeling by shortest data description. Automatica 14, 465–471.Schwarz, G., 1978. Estimating the dimension of a model. The Annals of Statistics 6, 461–464.Shibata, R., 1976. Selection of the order of an autoregressive model by Akaike’s information criterion. Biometrika 63,117–126.Shibata, R., 1980. Asymptotically efficient selection of the order of the model for estimating parameters of a linear process.The Annals of Statistics 8, 147–164.Zaman, A., 1984. Avoiding model selection by the use of shrinkage techniques. Journal of Econometrics 25, 239–346.

Effects of skewness and kurtosis on model selection criteria

Abstract

Similar works

Full text

Available Versions

Bilkent University Institutional Repository