Arne Risa Hole

English

Research Papers in Economics

 
 
 
Sheffield Economic Research Paper Series 
 
SERP Number: 2011011 
 
 
ISSN 1749-8368 
 
 
 
 
Arne Risa Hole 
  
 
A comment on ‘Mixed logit models: Accuracy and software choice.’ 
 
 
May 2011 
 
 
 
Department of Economics 
University of Sheffield 
9 Mappin Street 
Sheffield 
S1 4DT  
United Kingdom 
www.shef.ac.uk/economics A comment on ‘Mixed logit models:
Accuracy and software choice.’
Arne Risa Hole￿
Department of Economics
University of She¢eld
May 4, 2011
The recent paper by Chang and Lusk (2011) contains a useful review of
the facilities for estimating mixed logit models in three econometric software
packages. The packages considered are SAS, NLOGIT-LIMDEP, and my
mixlogit module for Stata (Hole, 2007). The review gives details of the dif-
ferent options and capabilities of the packages and also aims to evaluate their
accuracy by using a Monte Carlo experiment. In this comment I provide ev-
idence which suggests that the data generation process used in the paper by
Chang and Lusk may produce datasets that do not contain su¢cient varia-
tion to identify the parameters in the model empirically. This suggests that
their experiment is not well-suited to evaluate the accuracy of the di¤erent
mixed logit routines.
It has been demonstrated by Chiou and Walker (2007) that simulation
methods can mask identi…cation problems in the estimation of mixed logit
models. This is particularly an issue when a small number of draws is used
in the estimation process, although Chiou and Walker (2007) show that em-
pirical unidenti…cation can in some cases be masked at as many as 5000
pseudo-random draws and 2000 Halton draws. One sign of empirical uniden-
ti…cation is that the optimisation routine does not converge and/or parameter
estimates ‘explode’. Lack of convergence can therefore be an indication of
an identi…cation problem. In the following I will replicate the experiment
conducted by Chang and Lusk (2011) with the aim of further examining the
cases where the Stata mixlogit routine does not converge.1
￿Department of Economics, University of She¢eld, 9 Mappin Street, She¢eld S1 4DT,
UK. Tel.: +44 1142 223411, Fax: + 44 1142 223458. E-mail: a.r.hole@she¢eld.ac.uk.
1The Stata code that I used to replicate the experiment is available on request.I generated 500 datasets with size N = 200 following section 4.1 in
Chang and Lusk (2011) using Stata 11. The mixlogit routine declared non-
convergence with one of these datasets which was chosen for further examina-
tion. With this dataset I re-estimated the mixed logit model using three alter-
native packages: NLOGIT 4, Biogeme 1.8 (Bierlaire, 2003) and Matlab 7, us-
ing the Matlab routine written by Kenneth Train (http://elsa.berkeley.edu/
~train/software.html).2 The models were run using 50, 500 and 5000 Hal-
ton draws to evaluate the simulated log-likelihood function. In all the runs
multinomial logit estimates were used as starting values for the means and
the starting values for the standard deviations were set to 0.1.3
Table 1. Mixed logit estimation results. 50 Halton draws.
NLOGIT Biogeme Matlab
Parameter True value Coef. Std.Err. Coef. Std.Err. Coef. Std.Err.
￿1 2 3.95 1.30 3.34 1.06 3.91 1.27
￿1 1 1.63 0.95 1.32 0.87 1.59 0.94
￿2 5 10.81 3.71 8.68 2.84 10.69 3.67
￿2 1 3.66 1.64 2.15 1.50 3.61 1.63
Number of obs. 200 200 200
Simulated LL -50.97 -51.75 -51.18
In the runs using 50 Halton draws (Table 1) all the packages declare
convergence and produce reasonable-looking results. When increasing the
number of draws to 500, however, the packages show signs of the model
being unidenti…ed: NLOGIT displays a warning message saying that the
‘log likelihood is ‡at at the current estimates’, Biogeme does not declare
convergence after 1000 iterations (the default maximum) and the Matlab
results show signs of ‘exploding’ parameters.4 Further increasing the number
of draws to 5000 gives similar results, and in this case Biogeme issues a
2The Stata, NLOGIT and Biogeme analysis was carried out on a PC with an In-
tel Core 2 Duo CPU processor, 3.00 GHz, with 3.23 GB of RAM running Microsoft
Windows XP Professional Version 2002 (Service Pack 3). The Matlab analysis was car-
ried out on Iceberg, the University of She¢eld’s high performance computing server, see
http://www.she¢eld.ac.uk/wrgrid/iceberg for detailed speci…cations.
3When using alternative starting values Biogeme sometimes converged to di¤erent
solutions but the value of the simulated log-likelihood at convergence for these runs was
lower than for the results presented here.
4The coe¢cient estimates at convergence are 52364.07 (￿1), 24081.74 (￿1), 159525.68
(￿2) and 66120.58 (￿2).warning that the model is ‘unidenti…able’. Taken together these …ndings
suggest that the model is empirically unidenti…ed using the criteria proposed
by Chiou and Walker (2007).
As further evidence of this conclusion Figure 1 plots the maximum of the
simulated log-likelihood function over a range of values for ￿1.5 It is clear
from the …gure that the likelihood function is virtually ‡at over a wide range
of ￿1 values, which again suggests that the model is empirically unidenti…ed.
Figure 1. Plot of the maximum of the simulated log-likelihood function over
a range of values for ￿1
-
6
5
-
6
0
-
5
5
-
5
0
-
4
5
S
i
m
u
l
a
t
e
d
 
l
o
g
-
l
i
k
e
l
i
h
o
o
d
0 5 10 15 20 25 30 35 40
β1
The …ndings presented in this note suggest that the data generation
process used in Chang and Lusk (2011) is not well-suited to evaluate the
accuracy of the software packages considered in the review, as at least in
some cases the generated dataset does not contain enough variation to iden-
tify the parameters in the model. While the authors’ conclusions regarding
the options and capabilities of the various mixed logit routines are uncon-
troversial, further research is needed to determine which software package is
the most accurate.
5The …gure was generated in Stata 11 using the mixlogit module with 500 Halton
draws to obtain the simulated log-likelihood values.References
[1] Bierlaire M. 2003. BIOGEME: A free package for the estimation of dis-
crete choice models. Proceedings of the 3rd Swiss Transportation Research
Conference, Ascona, Switzerland.
[2] Chang JB, Lusk JL. 2011. Mixed logit models: Accuracy and software
choice. Journal of Applied Econometrics 26: 167-172.
[3] Chiou L, Walker JL. 2007. Masking identi…cation of discrete choice mod-
els under simulation methods. Journal of Econometrics 141: 683–703.
[4] Hole AR. 2007. Fitting mixed logit models by using maximum simulated
likelihood. The Stata Journal 7: 388-401.

A comment on ‘Mixed logit models: Accuracy and software choice.'

Abstract

Similar works

Full text

Available Versions

Research Papers in Economics