44 research outputs found
Clustering and Inference From Pairwise Comparisons
Given a set of pairwise comparisons, the classical ranking problem computes a
single ranking that best represents the preferences of all users. In this
paper, we study the problem of inferring individual preferences, arising in the
context of making personalized recommendations. In particular, we assume that
there are users of types; users of the same type provide similar
pairwise comparisons for items according to the Bradley-Terry model. We
propose an efficient algorithm that accurately estimates the individual
preferences for almost all users, if there are
pairwise comparisons per type, which is near optimal in sample complexity when
only grows logarithmically with or . Our algorithm has three steps:
first, for each user, compute the \emph{net-win} vector which is a projection
of its -dimensional vector of pairwise comparisons onto an
-dimensional linear subspace; second, cluster the users based on the net-win
vectors; third, estimate a single preference for each cluster separately. The
net-win vectors are much less noisy than the high dimensional vectors of
pairwise comparisons and clustering is more accurate after the projection as
confirmed by numerical experiments. Moreover, we show that, when a cluster is
only approximately correct, the maximum likelihood estimation for the
Bradley-Terry model is still close to the true preference.Comment: Corrected typos in the abstrac
A practical guide and software for analysing pairwise comparison experiments
Most popular strategies to capture subjective judgments from humans involve
the construction of a unidimensional relative measurement scale, representing
order preferences or judgments about a set of objects or conditions. This
information is generally captured by means of direct scoring, either in the
form of a Likert or cardinal scale, or by comparative judgments in pairs or
sets. In this sense, the use of pairwise comparisons is becoming increasingly
popular because of the simplicity of this experimental procedure. However, this
strategy requires non-trivial data analysis to aggregate the comparison ranks
into a quality scale and analyse the results, in order to take full advantage
of the collected data. This paper explains the process of translating pairwise
comparison data into a measurement scale, discusses the benefits and
limitations of such scaling methods and introduces a publicly available
software in Matlab. We improve on existing scaling methods by introducing
outlier analysis, providing methods for computing confidence intervals and
statistical testing and introducing a prior, which reduces estimation error
when the number of observers is low. Most of our examples focus on image
quality assessment.Comment: Code available at https://github.com/mantiuk/pwcm
Optimal Data Collection For Informative Rankings Expose Well-Connected Graphs
Given a graph where vertices represent alternatives and arcs represent
pairwise comparison data, the statistical ranking problem is to find a
potential function, defined on the vertices, such that the gradient of the
potential function agrees with the pairwise comparisons. Our goal in this paper
is to develop a method for collecting data for which the least squares
estimator for the ranking problem has maximal Fisher information. Our approach,
based on experimental design, is to view data collection as a bi-level
optimization problem where the inner problem is the ranking problem and the
outer problem is to identify data which maximizes the informativeness of the
ranking. Under certain assumptions, the data collection problem decouples,
reducing to a problem of finding multigraphs with large algebraic connectivity.
This reduction of the data collection problem to graph-theoretic questions is
one of the primary contributions of this work. As an application, we study the
Yahoo! Movie user rating dataset and demonstrate that the addition of a small
number of well-chosen pairwise comparisons can significantly increase the
Fisher informativeness of the ranking. As another application, we study the
2011-12 NCAA football schedule and propose schedules with the same number of
games which are significantly more informative. Using spectral clustering
methods to identify highly-connected communities within the division, we argue
that the NCAA could improve its notoriously poor rankings by simply scheduling
more out-of-conference games.Comment: 31 pages, 10 figures, 3 table