6 research outputs found
Voting with Limited Information and Many Alternatives
The traditional axiomatic approach to voting is motivated by the problem of
reconciling differences in subjective preferences. In contrast, a dominant line
of work in the theory of voting over the past 15 years has considered a
different kind of scenario, also fundamental to voting, in which there is a
genuinely "best" outcome that voters would agree on if they only had enough
information. This type of scenario has its roots in the classical Condorcet
Jury Theorem; it includes cases such as jurors in a criminal trial who all want
to reach the correct verdict but disagree in their inferences from the
available evidence, or a corporate board of directors who all want to improve
the company's revenue, but who have different information that favors different
options.
This style of voting leads to a natural set of questions: each voter has a
{\em private signal} that provides probabilistic information about which option
is best, and a central question is whether a simple plurality voting system,
which tabulates votes for different options, can cause the group decision to
arrive at the correct option. We show that plurality voting is powerful enough
to achieve this: there is a way for voters to map their signals into votes for
options in such a way that --- with sufficiently many voters --- the correct
option receives the greatest number of votes with high probability. We show
further, however, that any process for achieving this is inherently expensive
in the number of voters it requires: succeeding in identifying the correct
option with probability at least requires voters, where is the number of options and is a
distributional measure of the minimum difference between the options
Aggregating partial rankings with applications to peer grading in massive online open courses
We investigate the potential of using ordinal peer grading for the evaluation of students in massive online open courses (MOOCs). According to such grading schemes, each student receives a few assignments (by other students) which she has to rank. Then, a global ranking (possibly translated into numerical scores) is produced by combining the individual ones. This is a novel application area for social choice concepts and methods where the important problem to be solved is as follows: how should the assignments be distributed so that the collected individual rankings can be easily merged into a global one that is as close as possible to the ranking that represents the relative performance of the students in the assignment? Our main theoretical result suggests that using very simple ways to distribute the assignments so that each student has to rank only k of them, a Borda-like aggregation method can recover a 1 - O(1/k) fraction of the true ranking when each student correctly ranks the assignments she receives. Experimental results strengthen our analysis further and also demonstrate that the same method is extremely robust even when students have imperfect capabilities as graders. Our results provide strong evidence that ordinal peer grading cam be a highly effective and scalable solution for evaluation in MOOCs
Who is in Your Top Three? Optimizing Learning in Elections with Many Candidates
Elections and opinion polls often have many candidates, with the aim to
either rank the candidates or identify a small set of winners according to
voters' preferences. In practice, voters do not provide a full ranking;
instead, each voter provides their favorite K candidates, potentially in ranked
order. The election organizer must choose K and an aggregation rule.
We provide a theoretical framework to make these choices. Each K-Approval or
K-partial ranking mechanism (with a corresponding positional scoring rule)
induces a learning rate for the speed at which the election correctly recovers
the asymptotic outcome. Given the voter choice distribution, the election
planner can thus identify the rate optimal mechanism. Earlier work in this area
provides coarse order-of-magnitude guaranties which are not sufficient to make
such choices. Our framework further resolves questions of when randomizing
between multiple mechanisms may improve learning, for arbitrary voter noise
models.
Finally, we use data from 5 large participatory budgeting elections that we
organized across several US cities, along with other ranking data, to
demonstrate the utility of our methods. In particular, we find that
historically such elections have set K too low and that picking the right
mechanism can be the difference between identifying the ultimate winner with
only a 80% probability or a 99.9% probability after 400 voters.Comment: To appear in HCOMP 201
How effective can simple ordinal peer grading be?
Ordinal peer grading has been proposed as a simple and scalable solution for computing reliable information about student performance in massive open online courses. The idea is to outsource the grading task to the students themselves as follows. After the end of an exam, each student is asked to rank — in terms of quality — a bundle of exam papers by fellow students. An aggregation rule then combines the individual rankings into a global one that contains all students. We define a broad class of simple aggregation rules, which we call type-ordering aggregation rules, and present a theoretical framework for assessing their effectiveness. When statistical information about the grading behaviour of students is available (in terms of a noise matrix that characterizes the grading behaviour of the average student from a student population), the framework can be used to compute the optimal rule from this class with respect to a series of performance objectives that compare the ranking returned by the aggregation rule to the underlying ground truth ranking. For example, a natural rule known as Borda is proved to be optimal when students grade correctly. In addition, we present extensive simulations that validate our theory and prove it to be extremely accurate in predicting the performance of aggregation rules even when only rough information about grading behaviour (i.e., an approximation of the noise matrix) is available. Both in the application of our theoretical framework and in our simulations, we exploit data about grading behaviour of students that have been extracted from two field experiments in the University of Patras