2 research outputs found
A Weighted Generalization of the Graham-Diaconis Inequality for Ranked List Similarity
The Graham-Diaconis inequality shows the equivalence between two well-known
methods of measuring the similarity of two given ranked lists of items:
Spearman's footrule and Kendall's tau. The original inequality assumes
unweighted items in input lists. In this paper, we first define versions of
these methods for weighted items. We then prove a generalization of the
inequality for the weighted versions.Comment: 13 pages, 4 figure
How Reliable are University Rankings?
University or college rankings have almost become an industry of their own,
published by US News \& World Report (USNWR) and similar organizations. Most of
the rankings use a similar scheme: Rank universities in decreasing score order,
where each score is computed using a set of attributes and their weights; the
attributes can be objective or subjective while the weights are always
subjective. This scheme is general enough to be applied to ranking objects
other than universities. As shown in the related work, these rankings have
important implications and also many issues. In this paper, we take a fresh
look at this ranking scheme using the public College dataset; we both formally
and experimentally show in multiple ways that this ranking scheme is not
reliable and cannot be trusted as authoritative because it is too sensitive to
weight changes and can easily be gamed. For example, we show how to derive
reasonable weights programmatically to move multiple universities in our
dataset to the top rank; moreover, this task takes a few seconds for over 600
universities on a personal laptop. Our mathematical formulation, methods, and
results are applicable to ranking objects other than universities too. We
conclude by making the case that all the data and methods used for rankings
should be made open for validation and repeatability.Comment: 47 pages, 24 figure