26,398 research outputs found
Active Ranking using Pairwise Comparisons
This paper examines the problem of ranking a collection of objects using
pairwise comparisons (rankings of two objects). In general, the ranking of
objects can be identified by standard sorting methods using
pairwise comparisons. We are interested in natural situations in which
relationships among the objects may allow for ranking using far fewer pairwise
comparisons. Specifically, we assume that the objects can be embedded into a
-dimensional Euclidean space and that the rankings reflect their relative
distances from a common reference point in . We show that under this
assumption the number of possible rankings grows like and demonstrate
an algorithm that can identify a randomly selected ranking using just slightly
more than adaptively selected pairwise comparisons, on average. If
instead the comparisons are chosen at random, then almost all pairwise
comparisons must be made in order to identify any ranking. In addition, we
propose a robust, error-tolerant algorithm that only requires that the pairwise
comparisons are probably correct. Experimental studies with synthetic and real
datasets support the conclusions of our theoretical analysis.Comment: 17 pages, an extended version of our NIPS 2011 paper. The new version
revises the argument of the robust section and slightly modifies the result
there to give it more impac
Just Sort It! A Simple and Effective Approach to Active Preference Learning
We address the problem of learning a ranking by using adaptively chosen
pairwise comparisons. Our goal is to recover the ranking accurately but to
sample the comparisons sparingly. If all comparison outcomes are consistent
with the ranking, the optimal solution is to use an efficient sorting
algorithm, such as Quicksort. But how do sorting algorithms behave if some
comparison outcomes are inconsistent with the ranking? We give favorable
guarantees for Quicksort for the popular Bradley-Terry model, under natural
assumptions on the parameters. Furthermore, we empirically demonstrate that
sorting algorithms lead to a very simple and effective active learning
strategy: repeatedly sort the items. This strategy performs as well as
state-of-the-art methods (and much better than random sampling) at a minuscule
fraction of the computational cost.Comment: Accepted at ICML 201
Ordered Preference Elicitation Strategies for Supporting Multi-Objective Decision Making
In multi-objective decision planning and learning, much attention is paid to
producing optimal solution sets that contain an optimal policy for every
possible user preference profile. We argue that the step that follows, i.e,
determining which policy to execute by maximising the user's intrinsic utility
function over this (possibly infinite) set, is under-studied. This paper aims
to fill this gap. We build on previous work on Gaussian processes and pairwise
comparisons for preference modelling, extend it to the multi-objective decision
support scenario, and propose new ordered preference elicitation strategies
based on ranking and clustering. Our main contribution is an in-depth
evaluation of these strategies using computer and human-based experiments. We
show that our proposed elicitation strategies outperform the currently used
pairwise methods, and found that users prefer ranking most. Our experiments
further show that utilising monotonicity information in GPs by using a linear
prior mean at the start and virtual comparisons to the nadir and ideal points,
increases performance. We demonstrate our decision support framework in a
real-world study on traffic regulation, conducted with the city of Amsterdam.Comment: AAMAS 2018, Source code at
https://github.com/lmzintgraf/gp_pref_elici
Optimal Data Collection For Informative Rankings Expose Well-Connected Graphs
Given a graph where vertices represent alternatives and arcs represent
pairwise comparison data, the statistical ranking problem is to find a
potential function, defined on the vertices, such that the gradient of the
potential function agrees with the pairwise comparisons. Our goal in this paper
is to develop a method for collecting data for which the least squares
estimator for the ranking problem has maximal Fisher information. Our approach,
based on experimental design, is to view data collection as a bi-level
optimization problem where the inner problem is the ranking problem and the
outer problem is to identify data which maximizes the informativeness of the
ranking. Under certain assumptions, the data collection problem decouples,
reducing to a problem of finding multigraphs with large algebraic connectivity.
This reduction of the data collection problem to graph-theoretic questions is
one of the primary contributions of this work. As an application, we study the
Yahoo! Movie user rating dataset and demonstrate that the addition of a small
number of well-chosen pairwise comparisons can significantly increase the
Fisher informativeness of the ranking. As another application, we study the
2011-12 NCAA football schedule and propose schedules with the same number of
games which are significantly more informative. Using spectral clustering
methods to identify highly-connected communities within the division, we argue
that the NCAA could improve its notoriously poor rankings by simply scheduling
more out-of-conference games.Comment: 31 pages, 10 figures, 3 table
- …