In this talk I discuss Pearson's X2 as a measure of goodness of fit for quantal response models, including binary outcome models (logit, probit, gompit), multinomial logistic regression (mlogit) and conditional logistic regression (clogit). Large sample results for X2 have been derived by a.o. McCullagh and Windmeijer. A Stata implementation of the test will be illustrated.

Jeroen Weesie

English

Research Papers in Economics

On Pearson’s X2
for categorical response variables
Jeroen Weesie
Department of Sociology
University of Utrecht
The Netherlands
email J.Weesie@fss.uu.nl
tel +31–30–2531922
fax +31–30–2534405
Prepared for
Belgian/Dutch/German Stata User’s Meeting
Maastricht May 23, 2002
Pearson’s X2 1/15 jeroen weesie June 10, 2002Testing goodness of ﬁt
• embedd model in larger model
• constructive method: suggests model
improvement
but: often violation of some assumption leads to
rejections for other forms of misspeciﬁcation
e.g.: White test for heteroscedasticity in
regression is also very sensitive to misspeciﬁcation
of mean
• may require estimating more complicated models
(LR, Wald), though sometimes score testing may
be feasible
Check testomit
• sometimes saturated model can be estimated,
e.g., regression models with categorical covariates
• goodness of ﬁt statistics
• derive distribution of d(obs,ﬁt) under Ho
• example with categorical response
d(obs,ﬁt) = sum (obs-exp)2/exp
• d() can often be seen as an aggregate of residuals
• See Cressie-Read (1984) for details
• All models are wrong ...
Pearson’s X2 2/15 jeroen weesie June 10, 2002Pearson’s X2 for binary data
Pearson X2 measure for goodness of ﬁt after binary
regression models
πi = F(x0
iβ)
With replication of the xi (HL: ”m-asymptotics”)
X2 =
X
pattern
(obs − exp)2
exp
With large number of obs per pattern, X2 is
approximately χ2 (with df = #patterns - #parameters).
Stata’s lfit command provides this test for logistic
regression
lfit also allows essentially unique covariates, i.e., with
small number of replications per pattern. The manual
warns that this is “not necessarily incorrect.”
Pearson’s X2 3/15 jeroen weesie June 10, 2002Unaggregated measure of ﬁt
With unique covariates, the unaggregated Pearson’s
statistic Tn is
Tn =
n X
i=1
(yi − ˆ πi)2
ˆ πi(1 − ˆ πi)
With replicated data, Tn does not equal X2, but usually
is close.
Claim: Tn is not χ2 distributed (”n-asymptotics”).
Correct Theory: Subject to regularity conditions
(Windmeijer ’90; McCullagh ’86)
Tn − n
√
nσn
→ N(0,1)
σ2
n =
1
n
n X
i=1
(1 − 2πi)2
πi(1 − πi)
− v0
nΩ−1
n vn
vn =
1
n
n X
i=1
1 − πi
πi(1 − πi)
F0(x0
iβ)xi
Ωn =
1
n
n X
i=1
F0(x0
iβ)
πi(1 − πi)
Fisher information
Condition for Tn to be χ2 distributed: σ2 = 2.
Counter example: logistic regression with 1 x-var
xi ∼ U[−1,2] β = 1
σ2
n → 0.034
Pearson’s X2 4/15 jeroen weesie June 10, 2002Extensions
Extensions available for (Windmeijer 1995)
• multinomial logit (Stata: mlogit)
• conditional logistic regression (one success/group;
= Luce-McFadden choice model) (Stata: clogit).
Conditional logistic regression (k alternatives)
πij =
expxijβ
Pk
h=1expxihβ
Asymptotic result for
Tn =
n X
i=1
k X
j=1
(Yij − ˆ πij)2
ˆ πij(1 − ˆ πij)
Tn − nk
√
nσn
→ N(0,1)
σ2
n =
1
n
n X
i=1
k X
j=1

  
  
1 − 2πij
πij
qij −
k X
h6=j
qijqih

  
   − ν0
nΩ−1
n νn
νn =
1
n
n X
i=1
k X
j=1
qij(xij −
k X
h=1
πihxih)
qij =
1 − 2πij
1 − πij
Pearson’s X2 5/15 jeroen weesie June 10, 2002Problem with application
(1) Sensitivity with respect to observations with large
residuals (small πi for observed response)
Ad hoc modiﬁcations of test statistics
• ignore observations with some π < 
• ”round-up” probabilities to 
• or: “leave as is”
See also Hosmer & Lemeshow - 2nd edition.
(2) Quality of asymptotic approximation unknown
(3) Power against meaningfull misspeciﬁcations
unknown
Pearson’s X2 6/15 jeroen weesie June 10, 2002A Stata command
Post-estimation command
pearsonx2 [, eps(#) table]
available after the following commands
logit / logistic
probit
cloglog
mlogit
clogit -- one positive response per groups
Options
eps(#) specifies that only observations for which the
estimated probability for all possible outcomes are
greater than # are used in computing the test.
# defaults to 1E-2.
table specifies that Windmeijer’s test is conducted for
various eps (.1,.01,.001,etc) in order to assess the
sensitivity of the test to very small probabilities
of some outcomes.
Pearson’s X2 7/15 jeroen weesie June 10, 2002Example logistic regression
. use barcelona_lbw
(Hosmer & Lemeshow data)
. xi: logistic low age lwt i.race smoke ptl ht ui
i.race _Irace_1-3 (naturally coded; _Irace_1 omitted)
Logit estimates Number of obs = 189
LR chi2(8) = 33.22
Prob > chi2 = 0.0001
Log likelihood = -100.724 Pseudo R2 = 0.1416
------------------------------------------------------------------------------
low | Odds Ratio Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
age | .9732636 .0354759 -0.74 0.457 .9061578 1.045339
lwt | .9849634 .0068217 -2.19 0.029 .9716834 .9984249
_Irace_2 | 3.534767 1.860737 2.40 0.016 1.259736 9.918406
_Irace_3 | 2.368079 1.039949 1.96 0.050 1.001356 5.600207
smoke | 2.517698 1.00916 2.30 0.021 1.147676 5.523162
ptl | 1.719161 .5952579 1.56 0.118 .8721455 3.388787
ht | 6.249602 4.322408 2.65 0.008 1.611152 24.24199
ui | 2.1351 .9808153 1.65 0.099 .8677528 5.2534
------------------------------------------------------------------------------
. lfit
Logistic model for low, goodness-of-fit test
number of observations = 189
number of covariate patterns = 182
Pearson chi2(173) = 179.24
Prob > chi2 = 0.3567
. lfit, group(10)
Logistic model for low, goodness-of-fit test
(Table collapsed on quantiles of estimated probabilities)
number of observations = 189
number of groups = 10
Hosmer-Lemeshow chi2(8) = 9.65
Prob > chi2 = 0.2904
Pearson’s X2 8/15 jeroen weesie June 10, 2002Logistic regression (cont)
. pearsonx2
Pearson-Windmeijer goodness-of-fit test after logistic low
number of observations = 189
Pearson’s X2 (ungrouped) = 182.02
Windmeijer’s H = norm(X2) = 0.61
Prob > chi2(1) = 0.4334
. pearsonx2, table
Pearson-Windmeijer goodness-of-fit test after logistic low
number of observations = 189
Pearson’s X2 (ungrouped) = 182.02
Windmeijer’s H = norm(X2) = 0.61
Prob > chi2(1) = 0.4334
----------------------------------------------------------------
eps | Obs X2 se(X2) H p
-----------+----------------------------------------------------
0.10000000 | 163 161.46 3.01 0.26 0.6095
0.01000000 | 189 182.02 8.90 0.61 0.4334
----------------------------------------------------------------
All obs with some p<eps are ignored in computing the test
Pearson’s X2 9/15 jeroen weesie June 10, 2002Example conditional logistic regression
clogit choice sexJap incJap japan sexEur incEur europe, group(id) nolog
Conditional (fixed-effects) logistic regression Number of obs = 885
LR chi2(6) = 142.74
Prob > chi2 = 0.0000
Log likelihood = -252.72012 Pseudo R2 = 0.2202
------------------------------------------------------------------------------
choice | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
sexJap | -.4694799 .3114939 -1.51 0.132 -1.079997 .141037
incJap | .0276854 .0123666 2.24 0.025 .0034472 .0519236
japan | -1.962652 .6216804 -3.16 0.002 -3.181123 -.7441806
sexEur | .5388442 .4525278 1.19 0.234 -.348094 1.425782
incEur | .0273669 .013787 1.98 0.047 .000345 .0543889
europe | -3.180029 .7546837 -4.21 0.000 -4.659182 -1.700876
------------------------------------------------------------------------------
. pearsonx2, table
Pearson-Windmeijer goodness-of-fit test after clogit choice
number of observations = 295
Pearson’s X2 (ungrouped) = 870.42
Windmeijer’s H = norm(X2) = 19.48
Prob > chi2(1) = 0.0000
----------------------------------------------------------------
eps | Obs X2 se(X2) H p
-----------+----------------------------------------------------
0.10000000 | 200 625.43 2.26 126.67 0.0000
0.01000000 | 295 870.42 3.30 19.48 0.0000
----------------------------------------------------------------
All obs with some p<eps are ignored in computing the test
Pearson’s X2 10/15 jeroen weesie June 10, 2002Simulation: Goodness-of-link test
Design
True : logit(πi) = γx1 + γx2
xij iid N(0,1)
Fitted: logit(πi) = β1x1 + β2x2
probit(πi) = β1x1 + β2x2
Results (proportion of rejections in 1000 replications)
pearsonx2 linktest
γ n ﬁtted 0.10 0.05 0.01 0.10 0.05 0.01
1 100 logit .048 .027 .015 .105 .049 .005
1 400 logit .070 .045 .021 .100 .050 .008
1 1600 logit .064 .046 .027 .119 .062 .013
1 100 probit .046 .032 .018 .112 .055 .010
1 400 probit .096 .076 .047 .120 .059 .009
1 1600 probit .102 .080 .058 .147 .085 .021
3 100 logit .105 .063 .028 .100 .064 .017
3 400 logit .127 .071 .021 .142 .116 .058
3 1600 logit .133 .074 .023 .103 .062 .014
3 100 probit .160 .119 .055 .194 .112 .037
3 400 probit .248 .178 .066 .253 .204 .134
3 1600 probit .269 .201 .080 .254 .164 .078
Pearson’s X2 11/15 jeroen weesie June 10, 2002Simulation: Omitted variables in logit
Design
true logitπi = xi1 + xi2 + γxi1xi2
xij iid N(0,1)
ﬁtted logitπi = β1xi1 + β2xi2
Results (proportion of rejections in 1000 replications)
n γ α = 0.10 α = 0.05 α = 0.01
100 0 .037 .028 .020
400 0 .085 .053 .026
1600 0 .076 .037 .010
100 1/3 .087 .070 .053
400 1/3 .112 .089 .036
1600 1/3 .248 .181 .092
100 2/3 .230 .201 .161
400 2/3 .273 .194 .113
1600 2/3 .847 .781 .641
100 1 .488 .459 .394
400 1 .621 .536 .348
1600 1 .999 .999 .995
Pearson’s X2 12/15 jeroen weesie June 10, 2002Simulation: Omitted variables in clogit
Design – k alternatives
True LPij = x1ij + x2ij + γx1ijx2ij
xhij iid N(0,1)
Fitted LPij = β1x1ij + β2x2ij
and
πij =
expLPij
Pk
l=1expLPil
Results (proportion of rejections in 1000 replications)
γ = 0 γ = 1
k n .100 .050 .010 .100 .050 .010
3 100 .031 .023 .017 .378 .352 .284
3 200 .040 .030 .017 .267 .211 .106
3 400 .046 .033 .024 .506 .475 .436
3 400 .057 .035 .021 .268 .194 .103
4 100 .061 .044 .026 .261 .229 .192
4 200 .022 .019 .006 .813 .747 .619
4 400 .051 .030 .017 .714 .636 .473
4 800 .059 .035 .019 .995 .920 .817
5 100 .052 .046 .024 .180 .133 .071
5 200 .028 .019 .007 .920 .905 .881
5 400 .061 .046 .022 .771 .722 .594
5 800 .046 .030 .014 .996 .994 .990
Pearson’s X2 13/15 jeroen weesie June 10, 2002Discussion and conclusion
(Based on many more simulations than reported here)
Dedicated tests (eg omitted vars test) have more power than the
omnibus gof test (surprise?)
Asymptotic results for binary cases (logit, probit) seem adequate
I am not sure yet about cloglog
Asymptotic results for mlogit / clogit are reasaonbly accurate only
for LARGE n. For small and moderate n, tests are severely biased.
Turn to higher order asymptotics?
The methods of Windmeijer (1994) and Weesie (199) for reducing
the sensitivity of the tests to very small probabilities are not
ambiguous improvements.
Consider other statistics from the power family suggested by
Cressie-Read.
Pearson’s X2 14/15 jeroen weesie June 10, 2002References
• Read, T.R.C. and N.A.C. Cressie. 1988. Goodness-of-Fit
Statistics for Discrete Multivariate Data. New York:
Springer Verlag.
See also Stata command multgof, a revision of gof published
in STB 36 1997.
• Hosmer, D.W. and S. Lemeshow. 1989. Applied logistic
regression. 2nd edition. Wiley: New York.
• McCullagh, P. 1986. “The conditional distribution of
goodness-of-ﬁt statistics for discrete data.” Journal of the
American Statistical Association 81: 104–107.
• Weesie, J. 1998. “Windmeijer’s goodness-of-ﬁt test for logistic
regression.” Stata Technical Bulletin Reprints Vol 8, 153–160.
• Weesie, J. 1999 (2001). Tests for omitted variables. Utrecht,
memeo.
Describes the command testomittedvars that performs LR,
Wald, and three forms of score tests for omitted variables.
• Windmeijer, F.A.G. 1990. “The asymptotic distribution of the
sum of weighted squared residuals in binary choice models.”
Statistica Neerlandica 44, 2: 69–78.
• Windmeijer, F.A.G. 1994. “A goodness-of-ﬁt test in the
multinomial logit model based on weighted squared residuals.”
Statistica Neerlandica 48, 3: 271–283.
Pearson’s X2 15/15 jeroen weesie June 10, 2002

On Pearson's X^2 for categorical response variables

http://fmwww.bc.edu/RePEc/dsug2002/nsgg_pearsonx2.pdf

On Pearson's X^2 for categorical response variables

Abstract

Similar works

Full text

Available Versions

Research Papers in Economics