Belle, F. van

Cremers, P.

Kleijnen, J.P.C.

English

Research Papers in Economics

t
CBM
R
7626 ~
1984
142
TIJDSC~?RIFTEN
BUREAU
gestemming KA11I ;. , ~ L.~~~',~
i~ ' -.:~~JOL
~ HOOL~ILBURG ~
FACULTEIT DER ECONOMISCHE WETENSCHAPPEN
RESEARCH MEMORANllti`? ~i i
~ C I N 0 0 4 3 5~
THE POWER OF WEIGHTED AND ORDINARY LEAST
SQL'ARES WITH ESTIMATED UNEQL'AL VARIANCES
IN EXPERIMENTAL DESIGN
Jack F.C. kleijnen
Pieter Cremers
Frie van Belle
January 1984
TILBURG UNIVERSITY
DEPARTP~NT OF ECONOMICS
Postbus 90153 - 5000 LE Tilburg
Netherlands1
THE POWER OF
WEIGHTED AND ORDINARY LEAST SQUARES
WITH ESTIMATED UNEQUAL VARIANCES
IN EXPERIMENTAL DESIGN
Jack P.C. Kleijnen
Pieter Cremers
Frie van Belle
Department of Business and Economics
Tilburg Uníversity (Katholieke Hogeschool Tilburg)
Tilburg, Netherlands
Keywords and Phrases: Student t test; bias; standard errors; a error
ABSTRACT
Response variances var(yi) are estimated from m replications per
experimental condition. The resulting estimated variances si can be used
to derive the correct variances of the Ordinàry Least Squares (OLS)
estimators B. The estimates si can also be used to compute the Estimated
~~
Weighted Least Squares (EWLS) estimators S. The asymptotic covariance
~~
formula for EWLS might be utilized to test these estimators s. The type
I and type II errors of this test procedure are compared to the corres-
ponding errors of the OLS test.
1. INTRODUCTION
This paper is a continuatíon of Kleíjnen, Brent and Brouwers
(1981) and Nozari (1984); also see Deaton, Reynolds and Myers (19,83).
The problem we face is: we have the classical general linear model
but the errors e may show strong heterogeneity of variance. We have
variance estimators s2 based on replicating the experimental conditions
i
i, say mi times:2
mi
Si - E (Yij - Yi)2I(mi-1) ( 1 - 1,...,n)
j-1
We examine the following questions:
(1) Are Ordinary Least Squares (OLS) robust? So we compute
B -
(X'.X)-1.X,.x
and
n .. - (X'
.X)-1.a2
a
n mi
0 2- E E(Yij - Yij)2I(N-q)
i-1 j-1
(1.3)
(1.4)
(1.5)
n „ „
where q denotes the number of parameters, N- E mi and yi ~ yi, The
j
classical t statistic with v- N-q degrees of frée~om is:
s. - s.
t~ - ~~ ,. ~ ~ ( j - i,...,q)
{var (Bj)}
(1.6)
(2) Can we use the OLS estimator g of (eq. 1.3) combined with the cor-
rect expression for the covariance matrix S2„ in case of unequal varianc-
es? Obviously we have: NS
s~.. - (x'.x)-i.x'.s~y.x.(x',X)-i (i.~)
s
We can estimate S2 using si of eq. (1.2). But how many degrees of free-
Y
dom has the t statistic of eq. ( 1.6)? It is easy to derive that eq.
(1.3) reduces to
., n
6j - E xij.ylln ( j - 1,....q) (1.8)
i-1
where xi is the (i ,j)th element of the n x q matrix g formed by the n
j
different rows of the N x q matrix X(remember: N- E mi; mi replicat-
es), and we restrict this study to experimental designs with X'.X
- n.I. Because (X ) 2 equals plus one, and the observations are indepen-
ij
dent we obtain:
n 2
var (B.) - E var (yi)In
~ i-1
(1.9)3
Using :he estimator si of eq. (1.2) we get:
n
var (B.) - E (silmi)In2
~ ial
(1.10)
We further restricted our study to equal numbers of replications (mi -
m) so that var (Bj) reduces to a sum of X2 variates. Because of the
additivity of X2 varíates the t statistic of eq. (1.6) has degrees of
freedom v - n.(m-1).
Note: If mi ~ m then we would recommend mi - c ~ar (yi~) so that
var (yi) is (approximately) constant.
2
(3) Can we use the variance estimators si in Estimated Weighted Least
Squares (EWLS)? Oz
B - (X' .S2-:X) ".X' .SZ-:x
~Y ~Y
The asymptotic covariance matrix of EWLS ís:
S?~~ - (X' .S2 1.X)
Y
(1.12)
Eqs. (1.11) and (1.12) result in the analogue of the t statistic of eq.
(1.6). However, it is more difficult to determine the correct degrees of
freedom v~. We might investigate:
(í) v~ - N-q - n.m-q; see the classical OLS formulas.
n ~
(ii) v- E(mi-1) - n.(m-1) ~ n.m-n; see eq. (1.10).
(iii) v~` - min (mi-1) - m-1; see Scheffé (1964).
~ i
(iv) v- m or t~ - z with z~ N(0,1); the asymptotic case.
Actually we did not investigate approach (i). One reason is that ap-
proach (i) assumes a correctly specified regression model whereas the
other approaches use the unbiased estimators si. The difference between
(ï) and (ii) is minor if q n(with q~ n).4
2. MONTE CARLO INPUT PARAMETERS
The parameters of our Monte Carlo experiment are as follows: All
n X q matríces X are orthogonal. One X is a 16 x 13 matrix X taken from
a simulation study on the Rotterdam harbor (with design generators
1- 5.6 and 3~ 4.5) and corresponding s vector; see Kleijnen et al.
(1981; note 3). The other matrices X are 8 X 4 and 4 X 3 respectively.
We combine each of these three cases with several degrees of heteroge-
neity measured by
H - ( omax - omin), oQ
where a~x (and
amin)
is the maximum (and minimum) element of Sty. H
varies between zero (constant variances) and 1,455.69 (taken from the
harbor study). The variances are estimated from m replications where we
varied m between two (a technical minimum) and twenty-five.
We repeated each Monte Carlo experiment (specified by X, B, ny,
and m) 150 times to reduce chance effects. We used a multiplicative
random number generator with multiplier 1313 and modulus 259. This
generator was developed by NAG (Numerical Algorithms Group) and it is
standard on our ICL 2960 computer.
3. MONTE CARLO RESULTS
In Appendix 1 we present the results that substantiate the experimental
results of Kleijnen et al. (1981). In other words we repeat the experi-
ment of Kleijnen et al. (1981) with different random numbers and find
the following results:
(i) Bias: OLS gíves unbiased estimators S as we knew, and EWLS gives
~~
unbiased estimators B too.
(íi) Standard errors: The asymptotic covariance formula of eq. (1.12)
applies if we estimate var (y) from twenty-five replications (m ~
25). For m~ 9 our results deviate from Kleijnen et al. (1981):
the asymptotic formula may very well underestimate the variance.
(iii) Relative efficiency: In case of strong heterogeneity EWLS gives
smaller variances for the B estimators provided we have more than
two replications (m ~ 2).5
Next we try to answer a new set of questíons, namely can we use
the Student t statistic tv when we estimate the unknown variances
var (y~) and apply OLS and EWLS respectively, where the degrees of
freedoa v may equal n(m-1) for OLS and (m-1), n(m-1) or m for EWLS. We
estimate the true distribution from 150 realizations, and apply three
popular goodness-of-fit tests, namely the X2, the Kolmogorov-Smirnov,
and the Anderson-Darling test. We apply each goodness-of-fit test to
each of the q parameters Bj, with a li significance level. We do not
present the mass of data but report our preliminary results (whích are
further investigated below): EWLS based on only two replícations result
in distributions not well approximated by any Student distribu[ion. If
we have more replications (m ~ 2) then we may use the Student t statist-
ic with the (conservative) degrees of freedom equal to m-1. If m is as
high as 25 then we may use the normal approximation. OLS with the cor-
rected variance formula accounting for unequal variances (eq. 1.7)
results in a t distribution with degrees of freedom equal to n(m-1),
provided n(m-1) ~ 15 (as n íncreases the variance of var (B) decreases).
We shall give more detailed results for the following more specialized
question.
Because we use the t distribution only to select the critical
constant t~,2, we test the hypothesis:
; S. - B. .
HO : P{ „~ „ ~ ~ ~ tV~2} - o (J ' 1,....q) (3.1)
{var (B~)}
versus the alternative hypothesis H1 : P{e} ~ a or the one-sided and
conservative alternative hypothesis H1 : P{e} ~ n, where e denotes the
event within the brackets of eq. (3.1). To test HO we use the binomial
test as follows. We estímate P{e} from 150 independent replications and
compute a confidence interval. For example, for the one-sided H1 the
lower limit of th,~ 1-YO confidence interv~l i s given by the following
expression where z 0 is defined by P(z ~ z 0) ~ YO and z~ N(0,1):
Y ..
p-z0.{p.(1-p)~150}1` (3.2)6
We reject ii0 if a is smaller than this limit. We reject HO if for a~ of
the q parameters Sj we exceed the critical level: Applying the Bonfer-
roni inequality we reject HO if:
l~~~q IPj-zY~q.{pj.(1-pj)I150}~] ~ a (3.3)
We fix Y in eq. (3.3) at 5X. We apply this procedure for three classi-
cal a values in eq. (3.1), namely 1X, 57 and lOZ. This approach results
in Tables I and II where the symbol ~ meane that we reject ii0. These
tables suggest the following conclusions: If the n responses y have
different variances and we can estimate these variances from more than
two replications (m ~ 2) then the OLS estimators 9 can be tested using a
Student t statistic with degrees of freedom equal to v a n(m-1), provid-
~~
ed we test B with an a exceeding li. Testing the EWLS estimators B re-
quires more replications, say m~ 25 (and a~ 0.01). This conclusion
agrees with Nozari (1983)'s conclusion.
If both OLS and EWLS result in (roughly) the same a error then
we may proceed to a comparison of their power functions. We estimated
the power function ín a few points (from eight to ten points) using
different random numbers per point. (In Kleijnen (1984) we shall present
a more efficient procedure.) For each point we generate 150 replicates.
In all experiments the estimated power of EWLS dominated that of OLS (as
we míght expect because in previous experiments we found that
~~
var (S ) t var (6)) which we tested through the sign test.
4. CONCLUSIONS
If we suspect heterogeneity of variances then we should try to
estimate the n different variances, using more than two replications (m
~ 2). We can use these estimated variances to derive the correct vari-
ances of the OLS estimators S and to test their significance, through
the Student t statistic with n(m-1) degrees of freedom. If we have firm
estimators of the response variances - say 25 replications - then it is
better to use the EWLS eatimators g~ with the t distribution with de-
grees of freedom equal to n(m-1). We should test OLS and EWLS estimators
using an a higher than 1X.TABLE I
Testing the tail of the t~ distribution; one-sided test
a - li a ~ 5Í
OLS
tn(m-1)
EWLS OLS
tm-1 tn(m-1)
C1Hllm2 ~ ~
tn(m-1)
C1H1455m2 ~ ~ ~ ~ ~ ~t ,~ ,~
C2H lOm2 ~ ~ ~ ,r ,t ,t
C2H1455m2 ~ ~ ~ ~ ~ ~r ~ ~t ,~
C3H lOm4 ~ ~ ~ ~ ~ ,~
C3111289m4 ~ ~ ~ ~t ~ ~
C3H10m5 ~ ,r ~
C3H 1289m5 ~ ~ ~ ~ ,~ ,~
C1HOm9 ~ ,t ,t ,r
C1Hllm9
C1H1455m9
C2HOm9
C2H l Om9 ~ ~ ~ ~t ,t ~
C2H1455m9 ~ ~ ~t ~t ,~
C3H10m9
C3H1289m9 ~ ~ ~t ~ ~t ~ ,t ~t
C1HOm25
C1Hllm25
~itti4i~mti
C2HOm25
C2H10m25
r,2H 1455m25
C3Hi~m25
C3H1289m25
C1Hllm2 means: Case 1(n - 16, q- 13), Heterogeneity factor H- 11, replications m- 2. Etc.
EWLS OLS
a - l0Y
EWLS
tm-1 tn(m-1) ~ tn(m-1) tm-1 tn(m-1) ~TABLE II
Testing the tail of the t~ distribution; two-sided test
C1Hllm2
C1H1455m2
C2H10m2
C2H1455m2
C3H10m4
C3H1289m4
C3H l Om5
C3H1289m5
C1HOm9
C1Hllm9
C1H1455m9
C2HOm9
C2H10m9
CZH1455m9
C3H10m9
C3H1289m9
C1HOm25
C1Hllm25
C1H1455m25
C2HOm25
C2H10m25
C2H1455m25
C3N10m25
C3H1289m25
a a li a- 5Y
tm-1 tn(m-1)
~ ~
OLS EWLS OLS EWLS OLS EWLS
tn(m-1)
~
~ tn(m-1) tm-1 tn(m-1) ~ tn(m-1) tm-1 tn(m-1) t
~ ,t ~ ,t ~t ~ ,~
a - 10~
~ ~ ~ ~ ~ ~ ~ ~ ~ ~
~ ~
~ ~ ~ ~ ~ ~ ~ ~ ~9
APPENDIX 1: REPEATING THE EXPERIMENT OF KLEIJNEN ET AL. (1981)
We first check the correctness of our Monte Carlo computer
progran as follows. We know that the OLS estimator 6 of eq. (1.3) or
(1.8) is unbiased and that its covariance matrix is given by eq. (1.7)
or (1.9) where St or var (y) is known in the Monte Carlo experiment. So
~
we estimated the aexpected values E(Bj) and the variances var (gj) from
the 150 Monte Carlo repetitions and tested these values using the stan-
dard normal z statistic respectively the X2 statistic with 149 degrees
of freedom.
Next we examine the quality of the various B estimators in
several steps:
(i) Bias of B estimator
We know that OLS give unbiased estimators s, and that under mild techni-
~~
cal assumptions EWLS also give unbiased estimators S. In the preceding
paragraph we verified the lack of bias ín OLS. For EWLS we computed the
(approximate) Student t statistic:
150 „~
( E S, ~150) - S
t149 0 Jg 150
~ - - - --- -- ( j - 1, ....4) (A.1)
{ E (S~ - E S~ ~150)2~(149 X 150)}~
g-1 Jg g-1 Jg
Note: We do not use the equality sign in eq. (A.1) because the EWLS
-~t - ~~
estimator S is not a linear transformation of x; ~ also uses the
random vector with the elements si. However, the t statistic is supposed
to be robust, especially with as many observations as 150.
We obtain 160 realizations of t149 (the number 160 follows from
Table III later on). We use a 5~ significance level per realization so
that we expect eight false significances. We find zero significances for
OLS and six for EWLS. We conclude that OLS and EWLS do give unbiased
estimators of S which agrees with Kleijnen et al. (1981).
(ii) Standard error of S estimator
The standard errors of the OLS estimators S follow from eq. (1.7) or eq.
(1.10). For EWLS we have the asymptotic formula of eq. (1.12). We com-
pute the X2 approximation:10
2íj)
X149
150 .~ 150 „~ 2
~1 (Bjg - ~1 BjgI150) ~149
(X'.HY1.X)jj
íj s 1,...~q) íA.2)
where () j j means the j-th element on the main diagonal of (). Table
III displays the maximimm and the minimum of the q realizations. We
compare the maximum and minimum using a two-sided
X149
test with li
significance, resulting in the critical values 0.73 and 1.32. Table III
suggests the following conclusions. With only two replications (m) to
~~
estimate var(y) we underestimate the true variance of B. With m- 25
~~
the asymptotic formula gives unbiased estimators of var(B ). With m- 9
we may very well underes~imate the variance; our results for m~ 9
conflict with Kleijnen et al. (1981) who reported unbiased estimators.
~~
Note: We use the X2 statistic even though 6 may be nonnormal and we
know that the X2 statistic is not robust. However, we do not apply a
distribution-free procedure because we have 149 degrees of freedom and
because ultimately we are not interested in the standard errors them-
selves but in their role when using a t statistic like eq. (1.6); see
Section 3.
(iii) Efficiency of OLS versus EWLS
We measure the efficiency by the variance. Therefore we compare the
estimated variance of the EWLS estimator (the numerator of eq. (A.2)) to
the known variance of the OLS estimator (see eq. (1.9)) which results ín
a X149
statistic analogous to eq. (A.2). Table IV suggests the followíng
conclusions (which agree with Kleijnen et al. (1981)):
(i) If we knew that the variances var(y) are constant (H - 0), then we
should not estimate them, i .e., we should not use EWLS.
(ii) In case of strong heterogeneity we should use EWLS provided we can
estímate var(y) from more than two observations.
BIBLIOGRAPHY
DEATON, M.,M.R. REYNOLDS and R.M. MYERS, Estimation and hypothesis
testing in regression in the presence of nonhomogeneous
error variances. COMMUNICATIONS IN STATISTICS, SIMULATION
AND COMPUTATION, 12, no. 1, 1983, pp. 45-66.11
KLEIJNEV, J.P.C., Efficient estímation of power functions in simulation
experiments. Tilburg University, Tilburg (Neth.), Nov. 1983.
KLEIJNEV, J.P.C., R. BRENT and R. BROUWERS, Small-sample behavior of
weighted least squares in experimental design applications.
COMMUNICATIONS IN STATISTICS, SIMULATION AND COMPUTATION,
B10, no. 3, 1981, pp. 303-313.
NOZARI, A., Dealing with unequal variances ín analysis of experimental
design applications. COMM[1NICATIONS IN STATISTICS, SIMUL-
ATION AND COMPUTATION, 1984 (forthcoming).
SCHEFFE, H., THE ANALYSIS OF VARIANCE. John Wiley 5 Sons, Inc., New
York, 4th printing, 1964.12
TABLE III
Adequacy of asymptotic variance formula
Case 1: n a 16, q 3 13
Heterogeneity H
0 11.84 1,455.69
m 9 25 2 9 25 2 9 25
max X2 1.399~ 1.207 1.643~ 1.215 1.238 1.792~ 1.186 1.224
min X2 0.834 0.823 1.013 0.914 0.674~ 1.236 0.923 1.005
Case 2: n 3 8, q- 4
Heterogeneity H
0 10.83 1,455.69
m 9 25 2 9 25 2 9 25
max X2 1.195 1.211 2.427~` 1.348~ 1.038 3.635~` 1.183 0.918
min X2 0.948 1.047 1.680 1.074 0.899 2.642 0.947 0.873
~
Case 3: n- 4, q- 3
Heterogeneity H
10.38 1,289.15
m 4 5 9 25 4 5 9 25
max X2 1.251 1.100 1.092 1.120 1.315 1.184 1.399~ 1.049
min X2 1.169 0.903 0.945 0.962 1.012 0.978 0.993 0.792
m13
TABLE IV
Effíciency of OLS versus EWLS
Case 1: n- 16, q- 13
Heterogeneity H
0 11.84 1,455.69
m 9 25 2 9 25 2 9 25
max X2 1.399~ 1.207 1.453~` 1.195 1.217 1.436~ 0.949 0.941
min x2 0.834 0.823 0.993 0.864 0.673~` 0.097~ 0.077~ 0.075~`
Case 2: n- 8, q- 4
Heterogeneity H
0 10.83 1,455.69
m 9 25 2 9 25 2 9 25
max X2 1.195 1.211 1.827~ 1.203 0.989 0.560 0.190 0.162
min X2 0.948 1.047 1.389 0.732 0.635~ 0.150~ 0.052~ 0.046~`
Case 3: n- 4, q- 3
Heterogeneity H
10.38 1,289.15
m 4 5 9 25 4 5 9 25
max X2 1.237 1.100 1.017 1.096 1.214 1.119 1.242 0.915
min X2 0.763 0.589~ 0.684~ 0.730 0.352~ 0.395~ 0.345~ 0.275~`i
?`; 1983 1:EEDS VEYSCH3NEN
126 H.H. Tigelaar
Iaeiitification of noisy linear systems with multiple arma inputs.
127 J.P.C. Kleijnen
Sta:istical Analysis of Steady-State Simulations: Survey of Recent
Pro;;ress.
12~3 A.J. de Zeeuw
Two notes on Nash and Information.
129 H.L. Theuns en A.M.L. Passier-Grootjans
Toeristische ontwikkeling - voorwaarden en systematiek; een selec-
tief líteratuuroverzicht.
130 J. Plasmans en V. Somers
A Maximum Likelihood Estimation Method of a Three Market Disequili-
brium Model.
131 R. van Montfort, R. Schippers, R. Heuts
Johnson SU-transformations for parameter estimation in arma-models
when data are non-gaussian.
132 J. Glombowski en M. Kruger
On the R81e of Distribution í n Different Theories of Cyclícal
Growth.
133 J.W.A. Vingerhoets en H.J.A. Coppens
Internationale Grondstoffenovereenkomsten.
Effecten, kosten en oligopolisten.
134 W.J. Oomens
The economic interpretation of the advertising effect of Lydia
Pinkham.
135 J.P.C. Kleijnen
Regression analysis: assumptions, alternatives, applications.
136 J.P.C. Kleí:ren
On the interpretation of variables.
137 G, van der Laan en A.J.J. Talman
Simplicial approximation of solutions to the nonlinear complemen-
tarity problem with lower and upper bounds.ai~
IN 1984 REEDS VERSCHENEN
Bibliotheek K. U. Brabant
i iii ~ iu i i
1 7 000 01 05981 6 8
138 G.J. Cuypers, J.P.C. Kleijnen en J.W.M. van Rooyen
Testing the Mean of an Asymetric Population:
Four Procedures Evaluated
139 T. Wansbeek en A. Kapteyn
Estimation in a linear model with serially correlated errors when
observations are missing
140 A. Kapteyn, S. van de Geer, H. van de Stadt, T. Wansbeek
Interdependent preferences: an econometric analysis
141 W.J.H. van Groenendaal
Diecrete and continuous univariate modelling
0

0.046~` Case 3: n- 4, q- 3 Heterogeneity H

0.345~ 0.275~`i ?`;

0.673~` 0.097~ 0.077~ 0.075~` Case 2: n- 8, q- 4 Heterogeneity H

de Zeeuw Two notes on Nash and Information.

Estimation and hypothesis testing in regression in the presence of nonhomogeneous error variances. COMMUNICATIONS IN STATISTICS,

Groenendaal Diecrete and continuous univariate modelling

Heuts Johnson SU-transformations for parameter estimation in arma-models when data are non-gaussian.

Kapteyn Estimation in a linear model with serially correlated errors when observations are missing

On the interpretation of variables.

On the R81e of Distribution í n Different Theories of Cyclícal Growth.

Regression analysis: assumptions, alternatives, applications.

Somers A Maximum Likelihood Estimation Method of a Three Market Disequilibrium Model.

Sta:istical Analysis of Steady-State Simulations: Survey of Recent Pro;;ress.

Talman Simplicial approximation of solutions to the nonlinear complementarity problem with lower and upper bounds.ai~ IN

Testing the Mean of an Asymetric Population: Four Procedures Evaluated

The economic interpretation of the advertising effect of Lydia Pinkham.

Theuns en A.M.L. Passier-Grootjans Toeristische ontwikkeling - voorwaarden en systematiek; een selectief líteratuuroverzicht.

Tigelaar Iaeiitification of noisy linear systems with multiple arma inputs.

Vingerhoets en H.J.A. Coppens Internationale Grondstoffenovereenkomsten. Effecten, kosten en oligopolisten.

Wansbeek Interdependent preferences: an econometric analysis

The power of weighted and ordinary least squares with estimated unequal variances in experimental design

The power of weighted and ordinary least squares with estimated unequal variances in experimental design

Abstract

Similar works

Full text

Available Versions

Research Papers in Economics