2,909 research outputs found
Optimizing positional scoring rules for rank aggregation
Nowadays, several crowdsourcing projects exploit social choice methods for computing an aggregate ranking of alternatives given individual rankings provided by workers. Motivated by such systems, we consider a setting where each worker is asked to rank a fixed (small) number of alternatives and, then, a positional scoring rule is used to compute the aggregate ranking. Among the apparently infinite such rules, what is the best one to use? To answer this question, we assume that we have partial access to an underlying true ranking. Then, the important optimization problem to be solved is to compute the positional scoring rule whose outcome, when applied to the profile of individual rankings, is as close as possible to the part of the underlying true ranking we know. We study this fundamental problem from a theoretical viewpoint and present positive and negative complexity results and, furthermore, complement our theoretical findings with experiments on real-world and synthetic data
Who is in Your Top Three? Optimizing Learning in Elections with Many Candidates
Elections and opinion polls often have many candidates, with the aim to
either rank the candidates or identify a small set of winners according to
voters' preferences. In practice, voters do not provide a full ranking;
instead, each voter provides their favorite K candidates, potentially in ranked
order. The election organizer must choose K and an aggregation rule.
We provide a theoretical framework to make these choices. Each K-Approval or
K-partial ranking mechanism (with a corresponding positional scoring rule)
induces a learning rate for the speed at which the election correctly recovers
the asymptotic outcome. Given the voter choice distribution, the election
planner can thus identify the rate optimal mechanism. Earlier work in this area
provides coarse order-of-magnitude guaranties which are not sufficient to make
such choices. Our framework further resolves questions of when randomizing
between multiple mechanisms may improve learning, for arbitrary voter noise
models.
Finally, we use data from 5 large participatory budgeting elections that we
organized across several US cities, along with other ranking data, to
demonstrate the utility of our methods. In particular, we find that
historically such elections have set K too low and that picking the right
mechanism can be the difference between identifying the ultimate winner with
only a 80% probability or a 99.9% probability after 400 voters.Comment: To appear in HCOMP 201
People on Drugs: Credibility of User Statements in Health Communities
Online health communities are a valuable source of information for patients
and physicians. However, such user-generated resources are often plagued by
inaccuracies and misinformation. In this work we propose a method for
automatically establishing the credibility of user-generated medical statements
and the trustworthiness of their authors by exploiting linguistic cues and
distant supervision from expert sources. To this end we introduce a
probabilistic graphical model that jointly learns user trustworthiness,
statement credibility, and language objectivity. We apply this methodology to
the task of extracting rare or unknown side-effects of medical drugs --- this
being one of the problems where large scale non-expert data has the potential
to complement expert medical knowledge. We show that our method can reliably
extract side-effects and filter out false statements, while identifying
trustworthy users that are likely to contribute valuable medical information
How reliable are annotations via crowdsourcing? a study about inter-annotator agreement for multi-label image annotation
The creation of golden standard datasets is a costly business. Optimally more than one judgment per document is obtained to ensure a high quality on annotations. In this context, we explore how much annotations from experts differ from each other, how different sets of annotations influence the ranking of systems and if these annotations can be obtained with a crowdsourcing approach. This study is applied to annotations of images with multiple concepts. A subset of the images employed in the latest ImageCLEF Photo Annotation competition was manually annotated by expert annotators and non-experts with Mechanical Turk. The inter-annotator agreement is computed at an image-based and concept-based level using majority vote, accuracy and kappa statistics. Further, the Kendall Ď„ and Kolmogorov-Smirnov correlation test is used to compare the ranking of systems regarding different ground-truths and different evaluation measures in a benchmark scenario. Results show that while the agreement between experts and non-experts varies depending on the measure used, its influence on the ranked lists of the systems is rather small. To sum up, the majority vote applied to generate one annotation set out of several opinions, is able to filter noisy judgments of non-experts to some extent. The resulting annotation set is of comparable quality to the annotations of experts
Voting with Random Classifiers (VORACE)
In many machine learning scenarios, looking for the best classifier that fits
a particular dataset can be very costly in terms of time and resources.
Moreover, it can require deep knowledge of the specific domain. We propose a
new technique which does not require profound expertise in the domain and
avoids the commonly used strategy of hyper-parameter tuning and model
selection. Our method is an innovative ensemble technique that uses voting
rules over a set of randomly-generated classifiers. Given a new input sample,
we interpret the output of each classifier as a ranking over the set of
possible classes. We then aggregate these output rankings using a voting rule,
which treats them as preferences over the classes. We show that our approach
obtains good results compared to the state-of-the-art, both providing a
theoretical analysis and an empirical evaluation of the approach on several
datasets
- …