Properties of various types of estimators of the regression coefficients in linear logistic regression models are considered. The estimators include those based on maximum likelihood, minimum chi-square and weighted least squares. Theoretical approximations to the biases of the estimators are developed. The results of a large scale simulation investigation evaluating the moment properties of the estimators are presented for the case of a logistic model with a single explanatory variable

Al-Sarraf, Z

Young, D H

English

Brunel University Research Archive

                 TR/05/85                                                   March   1985 Properties   of   Estimators   of Parameters    in   Logistic   Regression  Models. by Z.   Al-Sarraf   and  D.   H.   Young                                    z1486051 Summary Properties  of  various   types   of   estimators   of   the   regression coefficients   in   linear   logistic  regression   models  are  considered. The   estimators   include   those   based   on  maximum  likelihood,   minimum  chi-square  and  weighted  least   squares.    Theoretical approximations to   the  biases   of   the  estimators  are  developed.      The   results   of   a   large  scale   simulation   investigation  evaluating   the  moment  properties  of  the  estimators   are   presented   for   the   case  of    a   logistic   model  with   a   single explanatory    variable. Contents 1. Introduction 2. Bias  Approximations 3. Moment   properties   of   estimators.                                                                                         1. 1·     IntroductionIn  the   statistical  analysis  of  binary  data  when   explanatory variables   are  present,   the  logistic  regression  model  plays   a  central role.     To   introduce  the  model, let  Y1   ,Y 2 ,… ,Yg represent   g independent binomial  random  variables  where  Yi  represents   the  number  of  successes in  a   set  of  ni   independent  trials.     For   the   ith  group,   let   xi  1    , ...,xik, denote   the   values  on   k  explanatory  variables  which  are  thought  to influence   the   individual   trial   probability  of   success,   denoted   by  Pi, for   the  ith  group,   i=1,.,,,g.     For  this   situation,   the   linear  logistic regression  model   is                      g,...1,i,~β'i~x)iQ/i(PLog ==                                    (1.1) where  Qi   =   1   -  Pi     and                         ) k β , . . . , 1 β , 0 ( β ' ~β , ) ik x , . . . , i1 x , (1 ' i ~ x = =          (1.2) The  regression  coefficients   in        are  usually   all  unknown   and  there ~βare  a  number  of  well-known  methods for estimating   them  (see Berkson,(1955)) which  we  now  review. (i)     Maximum  LikelihoodThe most commonly used method of estimation is probably maximum likelihood (ML), since these estimates can now be routinely obtained using   statistical  packages  such  as  GLIM   (Baker  and  Belder   1978), The  kernel   of   the  log-likelihood  may  be  written  as                (1.3)g1i)}~β!i~xelog(1~β!i~xi{pin)~βL(∑=+−= where  pi   -  yi/ni   denotes   the  observed  proportion  of   successes   in  the  ith group.      In   matrix   form  the  first  and  second  order  derivatives  of  the log-likelihood   are   given   by                                                                                                                               ⎥⎥⎥⎥⎥⎥⎥⎥⎦⎤⎢⎢⎢⎢⎢⎢⎢⎢⎣⎡−∑⋅−∑∑ −=∂∂)ipi(pi ikxin:            )ipi(pi ilXini)ipi(pi0Xin)~β()~βL(                                                 (1.4) 3.       ( )⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦⎤⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎣⎡⎟⎟⎟⎠⎞⎜⎜⎜⎝⎛−∑⎟⎟⎟⎠⎞⎜⎜⎜⎝⎛−∑⎟⎟⎟⎠⎞⎜⎜⎜⎝⎛−∑=∂∂ipiQ2ipiQipiqi ikxinipiQ2ipiQiP2iqi i1xinipiQ2ipiQip2iqi i0xin~ββR               (( ( )))(1.15)~2βˆ~β2~V2~V2~βˆ~β~β~)(R2~DputweIf(1.14)iQiP2iqipiQ2ipindiag2~V Where(1.13)~X2~V~X''~β~β~β)R(2=⎟⎠⎞⎜⎝⎛==⎟⎟⎟⎠⎞⎜⎜⎜⎝⎛∂∂=+==∂∂∂β  then   is  given  by   the   solution  of   the  k+1   equations  given  by 2~βˆ      ~02~D =                (1.16) An   iterative   solution   can   again  be  found   using  a  Newton-Raphson  approach similar   to   that   outlined   for   the   maximum  likelihood   estimation   procedure. The   calculations   are  conveniently   performed  using   GLIM  as  follows.   If  we let                                  (1.18))~β!i~x(expi2μ~β)'i~X(expilμ(1.17)2iqini2Y,2ipinilY−==== then   from   (1,11),   minimization   of   )~(R β  is   equivalent   to   minimization of                                       ( ) (1.19))1i2μi2Y1ilμi il(y~β*R−+−∑=  Minimisation  of   R* )~( β    is   seen   to  be  equivalent   to  maximisation  of   the log-likelihood  when   the   {yi  1}   and   {yi 2}   are   treated  as   observations  on  independent    exponentially  distributed   random  variables  with  means    1iμand   μ i  2   respectively. To  use   GLIM,   the   data   are  entered  as  g  pairs   of vectors  of  observations,   the  vectors  for  the   tth  pair  being   5.                           (1.27)say1*iw)1ini)(q1ini(p3in2)i1)(ni(n −=−+−+++                                                        A     modified    WLS   estimate   is  theresore   given   by    the   value  of  ~β   which Minimizes                                                                                                (1.28) 2)g1i ~β!i~X*i(Z*iW)~β(*S ∑= −=for    which   the   solution    is                                           *~Z*~W'~X1)~X*~W~X'(4~βˆ −=                                                     (1.29) .))1i2n(1)1in(1/)1ini(q)1inip(in((diag*~wand)*g~z,...,*1~z*'~zwhere −+−+−+−+==             If   we  let    N= ∑=g1i ni    and    assume    that   with  fixed    g                                     i  =  1, . . . ,g                                         (1.30) ,iλN/ininim1 =∞→ where  0   <   λi   <   1,   then   if   the   logistic   regression  model   is   correct,   it is  well-known   that                                                  )1)~~VX'~X(~0,MN(d)~β~βˆ(21N −→−                                 (1.31) where     diag ((  λ~V iPiQi ))   and   we  use   to  demote   any   estimator   from   the   ~βˆset  It    follows   that   the  four   estimators   all   have   the   same ~4β,3~β,2~β,1~β ˆˆˆˆasymptoric   properties   with                                  1)~X1~V~X'(~)βˆ(acov~β,~)βˆ(aE−==                                  (1.32) In   section   2,   we  develop  approximations   to   the  biases   of   the  estimators correct   to   order  N     .      In  section  3,   the   results   of  a   fairly   large   scale simulation    investigation   to   compare   the  moment   properties  of   the   esti- mators   for  a  number  of   sample   sizes   and  parameter  configurations  when there   is   a   single   explanatory  variable  are  presented.     These  results considerably   extend  the   findings  made  by   Berkson   (1955)   who considered the  particular  case  g   =  3,   ni   =   10,   i   =   1,2,3   and   showed  that   the   simple WLS  method  was  more  efficient   than  the   ML  and  MCS  methods of estimation under   a   number   of    success    probability    configurations. 2.     Approximate   Biases   of   EstimatorsIn   this   section  we   develop  approximations   to   order  N-1  for   the  biases of   the  ML,   MCS  and  WLS   estimators.     Initially   it   is  convenient  to  consider a   general   class   of   estimation   procedures   in   which  the estimates kB,...,2B,1B)))  (i)     Maximum Likelihood          ⎭⎬⎫⎩⎨⎧ −∑=φ iQlogiPiQlogipiniputting                                              (2.10) we  obtain    iQiPinisxirxirsV,)ipi(pinirxirU ∑=−∑−=                  (2.11) (2.12))2i6Pi6p(1iQiPiniuxi itxisxirxsrtuZ)iPi(QiQiPini itxisxirxrstW +−∑=−∑=   The  derivatives  higher  than  first  order  are  all  constants  and  are  0(N)   so Ar s t  is  0(N -2). and Brs , tu   and Crstu   are 0(N -3). We  also  have E(Ur  )   =  0                                                                                     (2.13)  sayrsIisxirxiiini)sr( QPUUE ∑= =                                  (2.14)where  I rs  =  Vrs  is  the   (r,s)th  element   in  the  information  matrix  and                  rstW)iPi(iiinitxisxirxi)tsrE( QQPUUU −=−∑−=                     (2.15) Since   E(Ur  Us  Ut  )  is  0(N),   the  last  two  terms  in  (2.5)  which  are  neglected in   (2.9)  are  0(N -2 ). Hence  the  bias  of  the  ML  estimator  correct   to 0(N-1)   is  ab I cd I bd I ac I d c using stu W tu I rs I u t s 2 1 rst W tc I sb I ra I c b a st I t s 2 1 ) 1 ( r b = ∑ ∑ ∑ ∑ ∑ − = ∑ ∑ ∑ ∑ ∑ − =                       (2.16) (ii) Minimum Chi-Square             Putting                                                                                                                             (2.17) ii/2)iPi(PiniQP−∑=φ    we obtain  (2.20))iPi(PiQi2P2)iPi(P1)i(2PinitxisxirxirstW(2.19)iQiP2)iPi(P4121)i2(P)iPi(P1)i2(PiQiPinisxirxi2rsV(2.18))iPi(P2iQiP2)iPi(P1)i(2PirxinirU⎪⎭⎪⎬⎫⎪⎩⎪⎨⎧⎥⎥⎦⎤⎢⎢⎣⎡⎪⎭⎪⎬⎫⎪⎩⎪⎨⎧⎪⎭⎪⎬⎫⎪⎩⎪⎨⎧−−−−∑=−+−+−−−∑=−−−−∑=9. Since  V rs   is  independent  of , W~β rst  ,  Zrstu   and  all  higher  order derivatives  are   zero,   we  have   from   (2.5)                                           )34.2()tuλtu(ssuλrtλutssrsλssrssrβrβˆVUUUV−∑∑∑+∑−=∑−=−using   the  same  approximation as in   (2,22), where                             isxirxii1)i(ni2)rs(rsλ QPVE −∑==                                          (2.35) Standard  calculations  using  Taylor   series   approximations  gives (2.37)0(1))ii(iiiniQi1)Pi(niqipiniQiPlogqipilogiqipinEand(2.36)2in10i2niPiQiQiPlogqipilogiqipEPQQP +−=⎥⎥⎥⎦⎤⎢⎢⎢⎣⎡⎪⎭⎪⎬⎫⎪⎩⎪⎨⎧−−−⎪⎭⎪⎬⎫⎪⎩⎪⎨⎧⎟⎟⎟⎠⎞⎜⎜⎜⎝⎛−⎟⎟⎠⎞⎜⎜⎝⎛⎟⎟⎟⎠⎞⎜⎜⎜⎝⎛+−=⎥⎥⎥⎦⎤⎢⎢⎢⎣⎡⎪⎭⎪⎬⎫⎪⎩⎪⎨⎧⎟⎟⎟⎠⎞⎜⎜⎜⎝⎛−⎟⎟⎠⎞⎜⎜⎝⎛ Hence                                                                            (2.38))1N(0)i-i(isxi)s( PQUE−+∑= and       0(1))iPi(QiQiPiniuxitxisxi4})tuλtu(VsE{U +−∑−=−                   (2.39) Using  these  results   in   (2.34)   and  noting   that   λrs =  21 Irs +  0(N -2 ), we  obtain  for   the  bias  of   the  WLS  estimator        )iPi(QiQiPiniuxitxisxisuIrtIuts)iPi(QisxirsIs21(3)rb −∑∑∑∑−−∑∑=          (2.40) Thus to order N -1 , thebiases of the MCS and WLS estimators are equal. The bias of the ML estimator will be greater than the biases of the MCS and WLS estimators   if                     (2.41))iPi(QisxirsIs)iPi(QiQiPiniuxitxisxisuIrtIut3 −∑∑>−∑∑∑∑s 3.     Moment  Properties   Of   The  EstimatorsIn order to investigate  the  properties  of  the  ML,  MCS,   WLS  and  MWLS estimators,   a  large   scale   simulation   investigation  was  made  for  the  case of    a   single   explanatory  variable  with   equally   spaced   values.      Without   loss of    generality,   the    linear    logistic   regression    model  was   taken  aslog(Pi/Qi)   =  β0    +  β1( i -1). i   =   1,...,g                                (3.1)         11. For   the  MCS   estimators,   the  biases   to   0(N  -1)   are             )0β(1)0βˆ2E()}iPi(Qixi12I)iPi(Qi11{I21)0β(2)0βˆE( −+−∑+−∑=−            (3.7)             )1β(1)1βˆ2E()}iPi(Qixi22I)iPi(Qi21{I21)1β(2)1βˆE( −+−∑+−∑=−          (3.8) the   same   results  holding   for   the  biases   of   the  WLS  estimators. In    table   2,   the  biases   of   the   estimators   obtained  by   simulation  are given  together  with  the  approximation  by   (3 .4) ,     (3 -5) ,    (3.7)   and   (3.8). The   results   show  that   the  absolute  values   of   the  biases   for   the  MWLS estimators  were   consistently   larger  than   those  of   the  other   three   estima- tors.   The  bias   advantage  of   the  WLS   estimator  compared  with   the  MWLS  estimator   is   in  agreement  with  the   suggestions  made  by  Hitchcock   (1962). In   the   case  of  β1   it   is   seen   that   the  ML  estimates  were  systematically too  high  while   the  other   three  methods   gave  negative  biases   in  nearly  all cases. Table   2 Biases   x   102   of   estimators   for   configurations   shown   in   table   1 . a)β0 Configuration ML Approx(3.4) MCS WLS Approx(3.7) MWL   (i) -9.31 -5.10 -1 .52 2.05 2.80 9.18 n=25 (ii) -4.29 -2.35 -1 .86 -1 .37 -0.11 2.41  (iii) 2.14 0.30 1 .66 2.67 -0.15 0.12  (iv) -2.87 -2.50 6.67 9.82 7.48 15.80 n=25 (v) -0.92 -0.64 0.35 0.63 0.71 2.03  (vi) -0.07 0. 14 -0.99 -0.40 -0.57 -2.38  (i) -3.21 -2.55 0.61 1 .48 1 .40 6.60 n=50 (ii) -0.13 -1. 17 0.90 1 .02 -0.06 3.11  (iii) 0. 16 0. 15 -0.14 0.14 -0.08 -0.66  (iv) -0.21 -1.25 4.29 5.12 3.74 9.82 n=50 (v) -1.48 -0.32 -0.79 -0.69 0.35 0.16  (vi) -0.04 0.07 -0.57 -0.35 -0.28 -1 .05  (i) -1 .48 -1 .28 0.35 0.66 0.70 3.59 n=100 (ii) -0.73 -2.35 -0.15 -0.12 -0.11 0.98  (iii) -0.54 0.07 -0.70 -0.61 -0.04 -0.86                                                                                                                   13. Table  3 Variances  of  estimators   for  configurations   shown  in  table   1.                                                     a)β0  Conf iguration ML MCS      WLS   MWLS Approx(3. 3)  (i) 0,2118 0,1848 0,1768 0.1581 0.1889 n=25 (ii) 0.1143 0.1083 0.1070 0.0974 0.1143  (iii) 0.1272 0.1187 0.1190 0.1073 0.1176  (iv) 0.1127 0.0998 0.0983 0.0858 0,1018 n=25 (v) 0.0603 0.0560 0.0552 0.0517 0.0586  (vi) 0.0661 0.0617 0.0620 0.0560 0.0688  (i) 0.1017 0.0957 0.0940 0.0863 0.0944 n=50 (ii) 0.0597 0.0581 0.0579 0.0551 0.0571  (iii) 0.0627 0.0607 0.0605 0.0573 0.0588  (iv) 0.0538 0.0513 0.0510 0.0467 0.0509 n=50 (v) 0.0282 0.0273 0.0273 0.0262 0.0293  (vi) 0.0383 0.0372 0.0370 0.0351 0.0344  (i) 0.0480 0.0464 0.0459 0.0437 0.0472 n=100 (ii) 0.0301 0.0297 0.0296 0.0289 0.0286  (iii) 0.0319 0.0316 0.0315 0.0306 0.0294  (iv) 0.0270 0.0264 0.0264 0.0251 0.0255 n=100 (v) 0.0141 0.0139 0.0139 0.0136 0.0146  (vi) 0.0174 0.0172 0.0172 0.0167 0.0172 b)β1   (variances  ×  102)  Configuration ML MCS WLS MWLS Approx(3.3)  (i) 2.5560 2.2950 2.2240 2.0080 2.5013 n=25 (ii) 1 .8980 1.7930 1.7740 1.6120 1.9580  (iii) 3.7530 3.1790 3.2030 2.7190 3.1078  (iv) 0.3037 0.2721 0.2668 0.2408 0.2903 n=25 (v) 0.2476 0.2278 0.2234 0,2065 0.2360  (vi) 0.3098 0.2758 0.2761 0.2390 0.3386        15 Table  4Mean   square  errors  and  efficiencies  of  estimators  relative  to   the  ML estimators  for  configurations   shown   in  table   1. a)β0 Configuration  Mean   Square     Efficiencies   ML MCS WLS MWLS MCS WLS MWLS (i) 0.2205 0.1851 0. 1772 0.1665 119. 1 124.4 132.4 n=25 (ii) 0. 1 161 0.1087 0. 1072 0.0980 106.8 108.3 118.5  (iii) 0.1277 0.1190 0 .1197 0.1073 107.3 106.7 119.0  (iv) 0.1135 0.1043 0.1079 0.1108 108.8 105.2 102.4 n=25 (v) 0.0604 0.0560 0.0552 0.0521 107.9 109.5 115.9  (vi) 0.0661 0.0618 0.0620 0.0566 107.0 106.6 116.8  (i) 0. 1027 0.0957 0.0942 0.0907 107.3 109.0 1 13.2 n=50 (ii) 0.0597 0.0582 0.0580 0.0560 102.6 102.9 106.6  (iii) 0.0627 0.0607 0.0605 0.0574 103.3 103.6 109.2  (iv) 0.0538 0.0531 0.0536 0.0564 101 .3 100.4 95.4 n=50 (v) 0.0284 0.0273 0.0273 0.0262 104.0 104.0 108.4  (vi) 0.0383 0.0372 0.0371 0.0352 103.0 103.2 108.8  (i) 0.0482 0.0464 0.0460 0.0450 103.9 104.8 107.1 n=100 (ii) 0-0302 0.0297 0.0296 0.0290 101 .7 102.0 104.1  (iii) 0.0319 0.0316 0.0315 0.0307 101 .0 101.3 103.9  (iv) 0.0270 0.0269 0.0270 0.0278 100.4 100.0 97.1 n=100 (v) 0.0141 0.0139 0.0139 0.0137 101 .4 101 .4 102.9  (vi) 0.0174 0.0172 0.0172 0.0167 101 .2 101.2 104.2                               (b)β1   (mean   square  errors   x   102)  (i) 2.6311 2.3040 2.2240 2.0293 114.2 1 18.3 129.7 n=25 (ii) 1.9311 1.7970   1 .7752 1 .6353 107.5 108.8 118.1  (iii) 3.7684 3.2160 3.3745 2.9735 117.2 111.7 126.7  (iv) 0.3051 0.2767 0.2783 0.2685 110.3 109.6 113.6 n=25 (v) 0.2491 0.2238 0.2255 0.2192 108.9 110,5 113.6  (vi) 0.3099 0.2897 0.3099 0.2888 107.0 100.0 107.3                                                           17 Acknowledgement We wish  to thank  Mr. Dennis  Scrimshaw  for  many  helpful   discussions during  the  course  of   this  work. REFERENCESBaker,   R.J.   and  Nelder,   J.A.    (1978).     Generalised    Linear    Interactive Modelling   (GLIM).     Numerical  Algorithms   Group,   Oxford. Berkson,   J.    (1955).     Maximum  likelihood  and  minimum x2   estimates   of   the logistic    function.        J.Amer  .Statist  .Assoc.   50,   130-162. Gart,   J.J.   and  Zweifel,   J.R.    (1967).      On   the   bias   of  various   estimators of   the  logistic   function  and  its  variance  with  application  to  quantal bioassay.     Biometrika,  54,181-187. Hitchcock, S.E. (1962). A  note  on  the  estimation of the parameters of  the logistic function, using the minimum logit x2 method. Biometrika,  49, 250-252.  

A note on the estimation of the parameters of the logistic function, using the minimum logit x2 method.

estimates of the logistic function. J.Amer .Statist .Assoc.

Generalised Linear Interactive Modelling (GLIM). Numerical Algorithms Group,

Maximum likelihood and minimum x2 estimates of the logistic function.

On the bias of various estimators of the logistic function and its variance with application to quantal bioassay.

Properties of estimators of parameters in logistic regression models

Properties of estimators of parameters in logistic regression models

Abstract

Similar works

Full text

Available Versions

Brunel University Research Archive