Tests of overidentifying restrictions are widely used in practice. However, there is often confusion about the nature of their null hypothesis and about the interpretation of their outcome. In this note we argue that these tests give little information on whether the instruments are correlated with the errors of the underlaying economic model and on whether they identify parameters of interest

Parente, Paulo M D C

Santos Silva, Joao M C

University of Essex Research Repository

A Cautionary Note on Tests for Overidentifying Restrictions

ISSN 1755-5361                   Discussion Paper Series              A cautionary note on tests for overidentifying restrictions    Paulo M.D.C. Parente & J.M.C. Santos Silva       Note : The Discussion Papers in this series are prepared by members of the Department of Economics, University of Essex, for private circulation to interested readers. They often represent preliminary reports on work in progress and should therefore be neither quoted nor referred to in published work without the written consent of the author.                                 University of Essex           Department of Economics    No. 699 October 2011 A cautionary note on tests for overidentifyingrestrictionsPaulo M.D.C. Parentey J.M.C. Santos SilvazOctober 13, 2011AbstractTests of overidentifying restrictions are widely used in practice. However, there isoften confusion about the nature of their null hypothesis and about the interpretationof their outcome. In this note we argue that these tests give little information onwhether the instruments are correlated with the errors of the underlaying economicmodel and on whether they identify parameters of interest.JEL classication code: C12, C13, C51, C52.Key words: GMM, Hansens J-test, Instrumental variables, Sargan test.1. INTRODUCTIONEconomists doing empirical work often use instrumental variables (IV) and thegeneralized method of moments (GMM) to identify the parameter of interest.1 TheWe are grateful to an anonymous referee and to Holger Breinlich, Emanuele Ciani, DanieleMassacci, Patrick Nolen, Matthias Parey, Esmeralda Ramalho, and Joaquim Ramalho for manyuseful comments and discussions. Santos Silva gratefully acknowledges partial nancial supportfrom Fundação para a Ciência e Tecnologia (FEDER/POCI 2010).yDepartment of Economics, University of Exeter. Email: P.M.Parente@exeter.ac.uk.zUniversity of Essex and CEMAPRE. E-mail: jmcss@essex.ac.uk.1See, e.g., Wooldridge (2010) for details on these methods.1success of this approach critically depends on the validity of a set of moment condi-tions and therefore it is not surprising that researchers often try to check whether theassumed moment conditions are valid.It is well known that when the model is exactly identied it is not possible to checkthe validity of the moment conditions (e.g., Wooldridge, 2009, p. 529). However,when the model is over identied, researchers often use tests of the overidentifying re-strictions to assess the validity of the moment conditions.2 This practice is misleadingbecause the validity of the overidentifying restrictions is neither su¢ cient nor neces-sary for the validity of the moment conditions implied by the underlaying economicmodel, and therefore provides little information on the possibility of identifying theparameters of interest. Indeed, as noted for example by Deaton (2010), the validityof the moment conditions is an identifying assumption that cannot be tested.Although this result is known, it is rarely mentioned in the literature. Moreover,when it is mentioned, often no justication is provided for it, or the justication that isprovided is either inaccurate or not immediately clear. In this note we present the testsfor overidentifying restrictions in a way that makes their nature very transparent, andprovide illustrative examples that highlight important characteristics of these tests.2. TESTS FOR OVERIDENTIFYING RESTRICTIONSFor simplicity, consider a linear model of the formy = x0 + u; (1)where x is a k-vector of regressors,  is the k-vector of parameters of interest, and uare the errors of the model, which are correlated with x. If a p-vector of instrumentsz uncorrelated with u is available and p  k,  can be consistently estimated fromthe following moment conditionsE [z (y   x0)] = 0: (2)2Examples of popular tests for over identifying restrictions are the ones proposed by Sargan(1958) and by Hansen (1982).2Consider a random sample f(yi; x0i; z0i)gni=1 and let b denote the estimate of  ob-tained from the sample analog of (2). Then, if p > k, tests for overidentifying restric-tions check for possible correlation between the residuals (yi   x0ib) i = 1; :::; n andthe instruments. Hence, these tests are often interpreted as checking the validity of(2).The crucial point to note is that tests for overidentifying restrictions do not checkwhether (2) holds but rather whether there is some vector  = plim (b) such thatE [z (y   x0)] = 0; (3)where  is implicitly dened by the instruments used. Therefore, (2) and (3) implythe orthogonality between the instruments and di¤erent errors, and consequently therelation between the validity of the overidentifying restrictions and of the validity ofthe moment conditions implied by the underlaying economic model is very tenuous.It is easy to see that the overidentifying restrictions may be valid even if the in-struments are correlated with u. Indeed, substituting (1) in (3) we obtainE [z (x0 (   ) + u)] = 0; (4a)E (zx0) (   ) = E (zu) : (4b)Under the usual assumption that E(zx0) has rank equal to k, the expression aboveshows that (3) will hold as long as it is possible to nd a vector  = (   ) suchthat E(zu) =E(zx0).When the instruments are valid, E(zu) = 0 and the solution to (4b) is  = 0. Inthis case the moment conditions are valid and the estimator identies the parametersof interest. However, E(zu) = 0 is a su¢ cient but not necessary condition for (4b)to have a solution.3 Indeed, even if E(zu) 6= 0, it may be possible to nd a vector such that (4b) holds. In this case, the overidentifying restrictions are still valid butthe estimator identies  =  +  rather than the parameters of interest.3The necessary and su¢ cient condition for (4b) to have a solution is that E(uz) is in the span ofthe columns of E(zx0), i.e., that rankhE(zx0) E(uz)i= k.3The result that the validity of the overidentifying restrictions is not su¢ cient toensure the identication of the parameters of interest is not entirely new. In hisstudy of the local power of tests for overidentifying restrictions, Newey (1985) notedthat these restrictions may be valid even if the instruments are not. Neweys (1985)result for local alternatives is presented in more detail by Hall (2005). In turn, DeBlander (2008) considers non-local alternatives and gives somewhat less transparentversion of the result presented above. In the same vein, some authors note that whenall instruments have the same rational, the fact that the model passes the test ofoveridentifying restrictions o¤ers little comfort (see, e.g., Murray, 2006. p. 117, andWooldridge, 2009, p. 529), but do not provide a clear explanation for why this isthe case. More recently, Wooldridge (2010, pp. 134-7) and Deaton (2010, pp. 430-2)provided deeper and clearer discussions of this issue.4To see that the validity of the overidentifying restrictions is also not necessary forthe parameters of interest to be successfully identied, consider a setting where thepopulation of interest is a mixture of S sub-populations such that E [zs (y   x0s)] = 0,where expectations are taken over the entire population and s are the parametersof interest for sub-population s = 1; : : : ; S.5 With a random sample from the entirepopulation, it is clear that s can be estimated by using zs as instruments. However,if s 6= t 9t 6= s, in general there will be no value  such that E [zs (y   x0)] =0, 8s, and therefore the overidentifying restrictions will be invalid if the full set ofinstruments z1; : : : ; zS is used. Hence, the set of overidentifying restrictions may beinvalid even if each individual orthogonality condition holds and each instrumentidenties a parameter of interest.4It is worth noting that this issue is related to the possibility of existence of observationallyequivalent models in the GMM framework. That is, there may be multiple sets of valid momentrestrictions, involving di¤erent sets of parameters. This issue was studied recently by Hall andPelletier (2007).5This situation considered by Imbens and Angrist (1994), who have pointed out that di¤erentsets of instruments lead to the estimation of di¤erent objects.4That the rejection of the overidentifying restrictions can be the result of parameterheterogeneity was pointed out by Angrist, Graddy, and Imbens (2000) and it is alsonoted by Angrist and Pischke (2009, p. 166), who remark that testing overidentifyingrestrictions is out the window in a fully heterogeneous world.In short, whether or not the overidentifying restrictions are valid gives little infor-mation on whether the instruments are correlated with the errors of the underlayingeconomic model, and on whether parameters of interest can be successfully identied.Below we provide simple examples that illustrate this point.3. ILLUSTRATIVE EXAMPLESFor simplicity, we focus on the case where the researcher wants to estimate thereturns to education using a wage equation of the formln (w) = 0 + 1s+ u; (5)where w denotes the wage, s is a measure of the level of schooling, and 1 measuresthe returns to education and is the parameter of interest. As usual, s is not assumedto be uncorrelated with u, namely due to the possible omission of important regres-sors. Therefore, consistent estimation of 1 requires the availability of a vector z ofinstrumental variables. In what follows, we illustrate the lack of relation between thevalidity of the instruments and the validity of the overidentifying restrictions usingthe data studied by Card (1995), which is used by Wooldridge (2009 and 2010) toexemplify the use of instrumental variables estimators.63.1. Overidentifying restrictions are valid irrespective of the instrumentsvaliditySuppose that, as for example in Wooldridge (2009, p. 522), the wage equation isestimated by IV using mothers and fathers schooling, respectively ms and fs, asinstruments for s. That is, z is the vectorh1 ms fsi0.6The data are available at http://www.stata.com/data/jwooldridge/eacsap/card.dta.5Then, assuming that E (zz0) has full rank, the rst stage regression consists inestimating the linear projectionL (sjms; fs) = 0 + 1ms+ 2fs.Now, because ms and fs essentially measure the same thing, 1 and 2 are likely tobe similar.7 Indeed, with the data considered here, the estimates of 1 and 2 are,respectively, 0:20 and 0:22, with standard errors of about 0:02. This situation is notparticular to this example and it is likely to occur whenever the instruments haveessentially the same motivation and are measured on the same scale.By the same reasoning, it is likely to be the case that ms and fs will have similarcoe¢ cients in the linear projection of u on the instruments. That is, it is likely thatL (ujms; fs) = 0 + 1ms+ 2fs = 0 + 1 (ms+ fs) ;with the instruments being invalid if 1 6= 0.Suppose now that indeed 1 = 2 6= 0 and 1 = 2.8 Then, it is possible to showthat, even if 1 = 2 6= 0,E[(ln (w)  0   1s)z] = 0for 0 = (0   01=1 + 0) and 1 = (1 + 1=1). Indeed, from (5) we have thatE[(ln (w)  0   1s)z] = E[(u+ 0   0 + (1   1) s)z];= E[(u+ 01=1   0   1=1s)z]:Now, writing " = s  L (sjms; fs) and  = u  L (ujms; fs), we haveE[(ln (w)  0   1s)z] = E[(u+ 01=1   0   1=1 (0 + 1 (ms+ fs) + "))z];= E[(0 + 1 (ms+ fs) +    0   1 (ms+ fs)  1=1")z];= E[(   1=1")z] = 0;7See Holmlund, Lindahl and Plug (2010) for a recent study on the e¤ects of parents educationon childrens schooling.8In fact, all that is needed is that 21 =21which implies that evidently the problem persists evenif the instruments are measured in di¤erent scales.6with the last equality holding because both " and  are dened as di¤erences betweena variable and its linear projection on z, and consequently are uncorrelated with theinstruments. Therefore, whether or not z is correlated with u, the overidentifyingrestrictions will be valid.Going back to the illustrative data set, usingms and fs as instruments we obtain anestimate of the returns to education of 7% and the (robust) Hansens J-test statistichas a p-value of 0:22. Of course, given the similarity between the instruments used,this result o¤ers little information on whether or not z is correlated with u.3.2. Overidentifying restrictions are invalid when each instrument is validSuppose now that the e¤ect of education on wages is heterogeneous (see, e.g.,Card, 1998, and Ichino and Winter-Ebmer, 1999). More specically, suppose thatthe population is divided into two groups and that the returns to schooling for therst group are equal to h1 , whereas for the second group the returns are l1, withh1 > l1. Using  to denote an unobservable indicator that is 1 for individuals in therst group, being zero otherwise, the wage equation can be written asln (w) = 0 +h1 + l1 (1  )s+ u:Consider now the case in which two sets of binary instruments are available: zhis uncorrelated with u and with (1  ) s, but is correlated with s, whereas zl isuncorrelated with u and s, but is correlated with (1  ) s. That is, zl is an instru-ment that is correlated with the level of schooling only for the individuals for whichthe returns to education are low, whereas zh is only correlated with schooling forindividuals with high returns to education.Under standard regularity conditions, the following moment conditions holdE ln (w)  0   h1szh= 0;E ln (w)  0   l1szl= 0;7and therefore both instruments are valid in the sense that each of them will allow theidentication of a parameter of interest. However, when both instruments are used,the IV or GMM regression of ln (w) on s generally will only identify a mixture of h1and l1. In this case, in general, the overidentifying restrictions are invalid becausethere is no single parameter that makes the errors of the model orthogonal to bothinstruments.Ichino and Winter-Ebmer (1999) argue that, if they are valid, instruments likeparental education will identify l1 because they are likely to a¤ect mainly the school-ing of individuals with limited ability who may receive more schooling if their parentsare highly educated. Therefore, the if ms and fs are valid instruments, the results inSubsection 3.1 would be an estimate of l1. Ichino and Winter-Ebmer (1999) also ar-gue that h1 can be estimated using as instruments variables that identify individualswho, thanks to their ability, choose more schooling in the absence extraneous con-straints, but drop out of school if constrained. Ichino and Winter-Ebmer (1999) giveas an example of such instrument a dummy for the fathers participation in WWII.In the spirit of Ichino and Winter-Ebmer (1999), here we use the variables livingwith a single mother at the age of 14and living with stepparent at the age of 14to estimate h1 . With these instruments, the returns to education are estimated to be23% and the (robust) Hansens J-test statistic has a p-value of 0:30. Therefore, thenew set of instruments leads to an estimate of the returns to education that is indeedmuch larger than the one previously obtained, and again there are no indication thatthe overidentifying restrictions are violated.If the model is estimated using simultaneously both sets of instruments, however,the (robust) Hansens J-test statistic has a p-value of 0:01, which would lead to therejection of the null at the usual 5% level. At rst sight this result is puzzling becauseif each sub-set of overidentifying restrictions is valid, the full set should also be valid.However, this naïve interpretation is awed because the residual whose orthogonalityto the instruments is checked by the test for overidentifying restrictions depends onthe chosen set of instruments, and therefore the set of restrictions tested when zh and8zl are used together is not the union of the sets of restrictions tested when zh and zlare considered separately.Overall, the three test statistics computed in this example suggest that zh and zlestimate di¤erent sets of parameters, but are mute about the ability of these sets ofinstruments to identify parameters of interest.4. CONCLUDING REMARKSThe examples presented in this note clearly illustrate a number of interesting points.First and foremost, they show that the validity of the overidentifying restrictionsprovides little information on the ability of the instruments to identify the parameterof interest. It is important to note that this is not a nite sample limitation ofthe test, but rather it is one of its intrinsic characteristics. Second, these examplesshow that, contrarily to what is often stated, the interpretation of the outcome of atest for overidentifying restrictions does not depend on the presence of enough validinstruments. Finally, these examples suggest that it is more appropriate to interprettests for overidentifying restrictions as checks for whether or not all the instrumentsidentify the same vector of parameters, as proposed by Hausman (1983). That is, thetests check the coherency of the instruments rather than their validity.REFERENCESAngrist, J.D., Graddy, K., and Imbens, G.W. (2000). The Interpretation of In-strumental Variables Estimators in Simultaneous Equations Models with anApplication to the Demand for Fish,The Review of Economic Studies, 67,499-527.Angrist, J. and Pischke, J.-S. (2009). Mostly Harmless Econometrics: An Empiri-cists Companion, Princeton (NJ): Princeton University Press.Card, D. (1995). Using Geographic Variation in College Proximity to Estimatethe Return to Schooling. In: Christophides, L.N., Grant, E.K., and Swidin-9sky, R. (Eds), Aspects of Labour Market Behavior: Essays in Honour of JohnVanderkamp. Toronto: University of Toronto Press.Card, D. (1998). The Causal E¤ect of Education on Earnings. In: Ashenfelter,O., Card, D., (Eds.), Handbook of Labor Economics, vol. 3. Amsterdam: North-Holland.Deaton, A. (2010). Instruments, Randomization, and Learning about Develop-ment.Journal of Economic Literature, 48, 42455.De Blander, R. (2008). Which Null Hypothesis do Overidentication RestrictionsActually Test?. Economics Bulletin, 3, 1-9.Hall, A. (2005). Generalized Method of Moments, New York (NY): Oxford UniversityPress.Hall, A. R. and D. Pelletier (2007): Non-Nested Testing in Models Estimated viaGeneralized Method of Moments,forthcoming in Econometric Theory.Hausman, J.A. (1983). Specication and Estimation of Simultaneous Models.In:Griliches, Z. and Intriligator. M., (Eds.), Handbook of Econometrics, vol. 1.Amsterdam: North-Holland.Holmlund, H., Lindahl, M., and Plug, E. (2010). The Causal E¤ect of ParentsSchooling on Childrens Schooling - A Comparison of Estimation Methods, CE-Sifo Working Paper Series 3234, CESifo Group Munich.Ichino, A. and Winter-Ebmer, R. (1999). Lower and Upper Bounds of Returns toSchooling: An Exercise in IV Estimation with Di¤erent Instruments,EuropeanEconomic Review, 43, 889-901.Imbens, G.W. and Angrist, J.D. (1994). Identication and Estimation of LocalAverage Treatment E¤ects,Econometrica, 62, 467-475.Murray, M. P. (2006). Avoiding Invalid Instruments and Coping with Weak In-struments,Journal of Economic Perspectives, Vol. 20, 111-132.Newey, W. (1985). Generalized Method of Moments Specication Testing,Journalof Econometrics, 29, 229-256.10Sargan, J.D. (1958). The Estimation of Economic Relationships using InstrumentalVariables,Econometrica, 26, 393-415.Wooldridge, J.M. (2009). Introductory Econometrics, A Modern Approach, 4th ed.,Mason (OH): South-Western.Wooldridge, J.M. (2010). Econometric Analysis of Cross Section and Panel Data,2nd ed., Cambridge (MA): MIT Press.11

A cautionary note on tests for overidentifying restrictions

Abstract

Similar works

Full text

Available Versions

University of Essex Research Repository

University of Essex Research Repository