121,725 research outputs found
Analysis of Crowdsourced Sampling Strategies for HodgeRank with Sparse Random Graphs
Crowdsourcing platforms are now extensively used for conducting subjective
pairwise comparison studies. In this setting, a pairwise comparison dataset is
typically gathered via random sampling, either \emph{with} or \emph{without}
replacement. In this paper, we use tools from random graph theory to analyze
these two random sampling methods for the HodgeRank estimator. Using the
Fiedler value of the graph as a measurement for estimator stability
(informativeness), we provide a new estimate of the Fiedler value for these two
random graph models. In the asymptotic limit as the number of vertices tends to
infinity, we prove the validity of the estimate. Based on our findings, for a
small number of items to be compared, we recommend a two-stage sampling
strategy where a greedy sampling method is used initially and random sampling
\emph{without} replacement is used in the second stage. When a large number of
items is to be compared, we recommend random sampling with replacement as this
is computationally inexpensive and trivially parallelizable. Experiments on
synthetic and real-world datasets support our analysis
Clustering and Inference From Pairwise Comparisons
Given a set of pairwise comparisons, the classical ranking problem computes a
single ranking that best represents the preferences of all users. In this
paper, we study the problem of inferring individual preferences, arising in the
context of making personalized recommendations. In particular, we assume that
there are users of types; users of the same type provide similar
pairwise comparisons for items according to the Bradley-Terry model. We
propose an efficient algorithm that accurately estimates the individual
preferences for almost all users, if there are
pairwise comparisons per type, which is near optimal in sample complexity when
only grows logarithmically with or . Our algorithm has three steps:
first, for each user, compute the \emph{net-win} vector which is a projection
of its -dimensional vector of pairwise comparisons onto an
-dimensional linear subspace; second, cluster the users based on the net-win
vectors; third, estimate a single preference for each cluster separately. The
net-win vectors are much less noisy than the high dimensional vectors of
pairwise comparisons and clustering is more accurate after the projection as
confirmed by numerical experiments. Moreover, we show that, when a cluster is
only approximately correct, the maximum likelihood estimation for the
Bradley-Terry model is still close to the true preference.Comment: Corrected typos in the abstrac
A practical guide and software for analysing pairwise comparison experiments
Most popular strategies to capture subjective judgments from humans involve
the construction of a unidimensional relative measurement scale, representing
order preferences or judgments about a set of objects or conditions. This
information is generally captured by means of direct scoring, either in the
form of a Likert or cardinal scale, or by comparative judgments in pairs or
sets. In this sense, the use of pairwise comparisons is becoming increasingly
popular because of the simplicity of this experimental procedure. However, this
strategy requires non-trivial data analysis to aggregate the comparison ranks
into a quality scale and analyse the results, in order to take full advantage
of the collected data. This paper explains the process of translating pairwise
comparison data into a measurement scale, discusses the benefits and
limitations of such scaling methods and introduces a publicly available
software in Matlab. We improve on existing scaling methods by introducing
outlier analysis, providing methods for computing confidence intervals and
statistical testing and introducing a prior, which reduces estimation error
when the number of observers is low. Most of our examples focus on image
quality assessment.Comment: Code available at https://github.com/mantiuk/pwcm
- …