The Design of Stated Choice Experiments: The State of Practice and Future Challenges

Since the work of Louviere and Woodworth (1983) and Louviere and Hensher (1983), stated choice (SC) methods have become the dominant data paradigm in the study of behavioural responses of individuals and households as well as other organizations, in fields as diverse as marketing, transport and environmental and health economics, to name but a few. In SC experiments, it is usual for sampled respondents to be asked to choose from amongst a number of labelled or unlabelled alternatives defined on a number of attribute dimensions, each in turn described by pre-specified levels drawn from some underlying experimental design. The choice task is then repeated a number of times, up to the total number of choice sets being offered over the experiment. Several experimental design strategies are available to the practitioner, however, within the transport literature, it appears that the most common form of experimental design used are orthogonal fractional factorial designs. In this paper we review the properties of such designs, and demonstrate that these properties are unlikely to be retained through to the estimation process. We also discuss an alternative design construction strategy, used to construct statistically optimal designs

Efficiency and Sample Size Requirements for Stated Choice Studies

Stated choice (SC) experiments represent the dominant data paradigm in the study of behavioral responses of individuals, households as well as other organizations, yet little is known about the sample size requirements for models estimated from such data. Current sampling theory does not adequately address the issue and hence researchers have had to resort to simple rules of thumb or ignore the issue and collect samples of arbitrary size, hoping that the sample is sufficiently large enough to produce reliable parameter estimates. In this paper, we demonstrate how to generate efficient designs (based on D-efficiency and a newly proposed sample size S-efficiency measure) using prior parameter values to estimate multinomial logit models containing both generic and alternative-specific parameters. Sample size requirements for such designs in SC studies are investigated. In a numerical case study is shown that a D-efficient and even more an Sefficient design needs a (much) smaller sample size than a random orthogonal design. Furthermore, it is shown that wide level range has a significant positive influence on the efficiency of the design and therefore on the reliability of the parameter estimates

Sample optimality in the design of stated choice experiments

Stated choice (SC) experiments represent the dominant data paradigm in the study of behavioral responses of individuals, households as well as other organizations, yet little is known about the sample size requirements for models estimated from such data. Current sampling theory does not adequately address the issue and hence researchers have had to resort to simple rules of thumb or ignore the issue and collect samples of arbitrary size, hoping that the sample is sufficiently large enough to produce reliable parameter estimates. In this paper, we demonstrate how to generate efficient designs (based on D-efficiency and a newly proposed sample size S-efficiency measure) using prior parameter values to estimate multinomial logit models containing both generic and alternative-specific parameters. Sample size requirements for such designs in SC studies are investigated. Using a numerical case study, we show that using S-efficiency can substantially reduce the sample size required of SC studies

Construction of experimental designs for mixed logit models allowing for correlation across choice observations

In each stated choice (SC) survey, there is an underlying experimental design from which the hypothetical choice situations are determined. These designs are constructed by the analyst, with several different ways of constructing these designs having been proposed in the past. Recently, there has been a move from so-called orthogonal designs to more efficient designs. Efficient designs optimize the design such that the data will lead to more reliable parameter estimates for the model under consideration. The main focus has been on the multinomial logit model, however this model is unable to take the dependency between choice situations into account, while in a stated choice survey usually multiple choice situations are presented to a single respondent. In this paper, we extend the literature by focusing on the panel mixed logit (ML) model with random parameters, which can take the above mentioned dependency into account. In deriving the analytical asymptotic variance-covariance matrix for the panel ML model, used to determine the efficiency of a design, we show that it is far more complex than the crosssectional ML model (assuming independent choice observations). Case studies illustrate that it matters for which model the design is optimized, and that it seems that a panel ML model SC experiment needs less respondents than a cross-sectional ML experiment for the same level of reliability of the parameter estimates

Confidence intervals of willingness-­‐to-­‐pay for random coefficient logit models

Random coefficient logit (RCL) models containing random parameters are increasingly used for modelling travel choices. Willingness-to-pay (WTP) measures, such as the value of travel time savings (VTTS) are, in the case of such RCL models, ratios of random parameters. In this paper we apply the Delta method to compute the confidence intervals of such WTP measures, taking into account the variancecovariance matrix of the estimates of the distributional parameters. Compared to simulation methods such as proposed by Krinsky and Robb, the Delta method is able to avoid some of the simulations by deriving partly analytical expressions for the standard errors. Examples of such computations are shown for different combinations of random distributions

Efficient Designs for Alternative Specific Choice Experiments

In the past, research on the construction of efficient designs for stated choice experiments has been limited to unlabeled experiments with generic parameter estimates. In this paper, by deriving the asymptotic (co)variance matrix for the alternative-specific MNL model, the authors are able to generate efficient alternative-specific experiments. The authors show that D-error assuming prior parameter values equal to zero is unable to explain statistical efficiency in orthogonal designs and that wide attribute levels are likely to yield more reliable parameter estimates than using narrow attribute levels. The authors also show that the D-optimality criterion may yield inefficient parameter estimates for some design attributes given that trade-offs are made between the efficiencies of different parameter estimates

Constructing Efficient Choice Experiments

Research on the construction of efficient designs for stated choice (SC) experiments has been limited to either unlabeled experiments with generic parameter estimates or labeled experiments with alternative specific parameter estimates. Designs combining both generic and alternative specific parameters have not yet been addressed. In this paper, by deriving the asymptotic (co)variance matrix for the most general MNL model, the authors are able to demonstrate how efficient experiments that allow for the estimation of both types of estimates may be generated. The authors go onto show how estimation of the asymptotic (co)variance matrix may also be used to determine sample size requirements for SC experiments

Detecting dominancy and accounting for scale differences when using stated choice data to estimate logit models

Stated choice surveys have been used for several decades to estimate preferences of agents using choice models, and are widely applied in the transportation domain. Typically orthogonal or efficient experimental designs underlie such surveys. These experimental designs may suffer from choice tasks containing a dominant alternative, which we show is problematic because it affects scale and therefore may bias parameter estimates. We propose a new measure based on minimum regret to calculate dominancy and automatically detect such choice tasks in an experimental design. This measure is then used to define a new experimental design type that ensures tradeoffs within the design. Finally, we propose a new regret-scaled multinomial logit model that takes the level of dominancy within a choice task into account. Results using simulated and empirical data show that the presence of dominant alternatives can bias model estimates, but that making scale a function of a smooth approximation of normalised minimum regret can properly account for scale differences without the need to remove choice tasks with dominant alternatives from the dataset

Incorporating model uncertainty into the generation of efficient stated choice experiments: A model averaging approach

Stated choice (SC) studies typically rely on the use of an underlying experimental design to construct the hypothetical choice situations shown to respondents. These designs are constructed by the analyst, with several different ways of constructing these designs having been proposed in the past. Recently, there has been a move from so-called orthogonal designs to more efficient designs. Efficient designs optimize the design such that the data will lead to more reliable parameter estimates for the model under consideration. The literature dealing with the generation of efficient designs has examined and largely solved the issue of a requirement for a prior knowledge of the parameter estimates that will be obtained post data collection. Nevertheless, problems related to the fact that the efficiency of a SC experiment is related to the variance-covariance matrix of the model to be estimated and that different econometric models will have different variance-covariance matrix, thus resulting in different levels of efficiency for the same design, has yet to be addressed. In this paper, we propose the use of a model averaging process over different econometric models to solve this problem. Via the use of a case study, we show that designs generated using the model averaging process prove robust to different model estimation as well as provide decent levels of protection against biased parameter estimates relative to designs generated specifically for a given model type