1,340 research outputs found
Parameter estimation for generalized thurstone choice models
We consider the maximum likelihood parameter estimation problem for a generalized Thurstone choice model, where choices are from comparison sets of two or more items. We provide tight characterizations of the mean square error, as well as necessary and sufficient conditions for correct classification when each item belongs to one of two classes. These results provide insights into how the estimation accuracy depends on the choice of a generalized Thurstone choice model and the structure of comparison sets. We find that for a priori unbiased structures of comparisons, e.g., when comparison sets are drawn independently and uniformly at random, the number of observations needed to achieve a prescribed estimation accuracy depends on the choice of a generalized Thurstone choice model. For a broad set of generalized Thurstone choice models, which includes all popular instances used in practice, the estimation error is shown to be largely insensitive to the cardinality of comparison sets. On the other hand, we found that there exist generalized Thurstone choice models for which the estimation error decreases much faster with the cardinality of comparison sets
Models for Paired Comparison Data: A Review with Emphasis on Dependent Data
Thurstonian and Bradley-Terry models are the most commonly applied models in
the analysis of paired comparison data. Since their introduction, numerous
developments have been proposed in different areas. This paper provides an
updated overview of these extensions, including how to account for object- and
subject-specific covariates and how to deal with ordinal paired comparison
data. Special emphasis is given to models for dependent comparisons. Although
these models are more realistic, their use is complicated by numerical
difficulties. We therefore concentrate on implementation issues. In particular,
a pairwise likelihood approach is explored for models for dependent paired
comparison data, and a simulation study is carried out to compare the
performance of maximum pairwise likelihood with other limited information
estimation methods. The methodology is illustrated throughout using a real data
set about university paired comparisons performed by students.Comment: Published in at http://dx.doi.org/10.1214/12-STS396 the Statistical
Science (http://www.imstat.org/sts/) by the Institute of Mathematical
Statistics (http://www.imstat.org
When is it Better to Compare than to Score?
When eliciting judgements from humans for an unknown quantity, one often has
the choice of making direct-scoring (cardinal) or comparative (ordinal)
measurements. In this paper we study the relative merits of either choice,
providing empirical and theoretical guidelines for the selection of a
measurement scheme. We provide empirical evidence based on experiments on
Amazon Mechanical Turk that in a variety of tasks, (pairwise-comparative)
ordinal measurements have lower per sample noise and are typically faster to
elicit than cardinal ones. Ordinal measurements however typically provide less
information. We then consider the popular Thurstone and Bradley-Terry-Luce
(BTL) models for ordinal measurements and characterize the minimax error rates
for estimating the unknown quantity. We compare these minimax error rates to
those under cardinal measurement models and quantify for what noise levels
ordinal measurements are better. Finally, we revisit the data collected from
our experiments and show that fitting these models confirms this prediction:
for tasks where the noise in ordinal measurements is sufficiently low, the
ordinal approach results in smaller errors in the estimation
Minimax-optimal Inference from Partial Rankings
This paper studies the problem of inferring a global preference based on the
partial rankings provided by many users over different subsets of items
according to the Plackett-Luce model. A question of particular interest is how
to optimally assign items to users for ranking and how many item assignments
are needed to achieve a target estimation error. For a given assignment of
items to users, we first derive an oracle lower bound of the estimation error
that holds even for the more general Thurstone models. Then we show that the
Cram\'er-Rao lower bound and our upper bounds inversely depend on the spectral
gap of the Laplacian of an appropriately defined comparison graph. When the
system is allowed to choose the item assignment, we propose a random assignment
scheme. Our oracle lower bound and upper bounds imply that it is
minimax-optimal up to a logarithmic factor among all assignment schemes and the
lower bound can be achieved by the maximum likelihood estimator as well as
popular rank-breaking schemes that decompose partial rankings into pairwise
comparisons. The numerical experiments corroborate our theoretical findings.Comment: 16 pages, 2 figure
Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependence
Data in the form of pairwise comparisons arises in many domains, including
preference elicitation, sporting competitions, and peer grading among others.
We consider parametric ordinal models for such pairwise comparison data
involving a latent vector that represents the
"qualities" of the items being compared; this class of models includes the
two most widely used parametric models--the Bradley-Terry-Luce (BTL) and the
Thurstone models. Working within a standard minimax framework, we provide tight
upper and lower bounds on the optimal error in estimating the quality score
vector under this class of models. The bounds depend on the topology of
the comparison graph induced by the subset of pairs being compared via its
Laplacian spectrum. Thus, in settings where the subset of pairs may be chosen,
our results provide principled guidelines for making this choice. Finally, we
compare these error rates to those under cardinal measurement models and show
that the error rates in the ordinal and cardinal settings have identical
scalings apart from constant pre-factors.Comment: 39 pages, 5 figures. Significant extension of arXiv:1406.661
Ranking Models in Conjoint Analysis
In this paper we consider the estimation of probabilisticranking models in the context of conjoint experiments. By usingapproximate rather than exact ranking probabilities, we do notneed to compute high-dimensional integrals. We extend theapproximation technique proposed by \\citet{Henery1981} in theThurstone-Mosteller-Daniels model for any Thurstone orderstatistics model and we show that our approach allows for aunified approach. Moreover, our approach also allows for theanalysis of any partial ranking. Partial rankings are essentialin practical conjoint analysis to collect data efficiently torelieve respondents' task burden.conjoint experiments;partial rankings;thurstone order statistics model
- …