Creation of Reliable Relevance Judgments in Information Retrieval Systems Evaluation Experimentation through Crowdsourcing: A Review

Test collection is used to evaluate the information retrieval systems in laboratory-based evaluation experimentation. In a classic setting, generating relevance judgments involves human assessors and is a costly and time consuming task. Researchers and practitioners are still being challenged in performing reliable and low-cost evaluation of retrieval systems. Crowdsourcing as a novel method of data acquisition is broadly used in many research fields. It has been proven that crowdsourcing is an inexpensive and quick solution as well as a reliable alternative for creating relevance judgments. One of the crowdsourcing applications in IR is to judge relevancy of query document pair. In order to have a successful crowdsourcing experiment, the relevance judgment tasks should be designed precisely to emphasize quality control. This paper is intended to explore different factors that have an influence on the accuracy of relevance judgments accomplished by workers and how to intensify the reliability of judgments in crowdsourcing experiment

Learning robust low-rank approximation for crowdsourcing on Riemannian Manifold

Recently, crowdsourcing has attracted substantial research interest due to its efficiency in collecting labels for machine learning and computer vision tasks. This paper proposes a Rieman-nian manifold optimization algorithm, ROLA (Robust Low-rank Approximation), to aggregate the labels from a novel perspective. Specifically, a novel low-rank approximation model is proposed to capture underlying correlation among annotators meanwhile identify annotator-specific noise. More significantly, ROLA defines the label noise in crowdsourcing as annotator-specific noise, which can be well regularized by l2,1-norm. The proposed ROLA can improve the aggregation performance when compared with state-of-the-art crowdsourcing methods

When in doubt ask the crowd : leveraging collective intelligence for improving event detection and machine learning

[no abstract

PREM: Prestige Network Enhanced Developer-Task Matching for Crowdsourced Software Development

Many software organizations are turning to employ crowdsourcing to augment their software production. For current practice of crowdsourcing, it is common to see a mass number of tasks posted on software crowdsourcing platforms, with little guidance for task selection. Considering that crowd developers may vary greatly in expertise, inappropriate developer-task matching will harm the quality of the deliverables. It is also not time-efficient for developers to discover their most appropriate tasks from vast open call requests. We propose an approach called PREM, aiming to appropriately match between developers and tasks. PREM automatically learns from the developers’ historical task data. In addition to task preference, PREM considers the competition nature of crowdsourcing by constructing developers’ prestige network. This differs our approach from previous developer recommendation methods that are based on task and/or individual features. Experiments are conducted on 3 TopCoder datasets with 9,191 tasks in total. Our experimental results show that reasonable accuracies are achievable (63%, 46%, 36% for the 3 datasets respectively, when matching 5 developers to each task) and the constructed prestige network can help improve the matching results