2,304 research outputs found
Bandit-Based Task Assignment for Heterogeneous Crowdsourcing
We consider a task assignment problem in crowdsourcing, which is aimed at
collecting as many reliable labels as possible within a limited budget. A
challenge in this scenario is how to cope with the diversity of tasks and the
task-dependent reliability of workers, e.g., a worker may be good at
recognizing the name of sports teams, but not be familiar with cosmetics
brands. We refer to this practical setting as heterogeneous crowdsourcing. In
this paper, we propose a contextual bandit formulation for task assignment in
heterogeneous crowdsourcing, which is able to deal with the
exploration-exploitation trade-off in worker selection. We also theoretically
investigate the regret bounds for the proposed method, and demonstrate its
practical usefulness experimentally
BUOCA: Budget-Optimized Crowd Worker Allocation
Due to concerns about human error in crowdsourcing, it is standard practice to collect labels for the same data point from multiple internet workers. We here show that the resulting budget can be used more effectively with a flexible worker assignment strategy that asks fewer workers to analyze easy-to-label data and more workers to analyze data that requires extra scrutiny. Our main contribution is to show how the allocations of the number of workers to a task can be computed optimally based on task features alone, without using worker profiles. Our target tasks are delineating cells in microscopy images and analyzing the sentiment toward the 2016 U.S. presidential candidates in tweets. We first propose an algorithm that computes budget-optimized crowd worker allocation (BUOCA). We next train a machine learning system (BUOCA-ML) that predicts an optimal number of crowd workers needed to maximize the accuracy of the labeling. We show that the computed allocation can yield large savings in the crowdsourcing budget (up to 49 percent points) while maintaining labeling accuracy. Finally, we envisage a human-machine system for performing budget-optimized data analysis at a scale beyond the feasibility of crowdsourcing.First author draf
BUOCA: Budget-Optimized Crowd Worker Allocation
Due to concerns about human error in crowdsourcing, it is standard practice
to collect labels for the same data point from multiple internet workers. We
here show that the resulting budget can be used more effectively with a
flexible worker assignment strategy that asks fewer workers to analyze
easy-to-label data and more workers to analyze data that requires extra
scrutiny. Our main contribution is to show how the allocations of the number of
workers to a task can be computed optimally based on task features alone,
without using worker profiles. Our target tasks are delineating cells in
microscopy images and analyzing the sentiment toward the 2016 U.S. presidential
candidates in tweets. We first propose an algorithm that computes
budget-optimized crowd worker allocation (BUOCA). We next train a machine
learning system (BUOCA-ML) that predicts an optimal number of crowd workers
needed to maximize the accuracy of the labeling. We show that the computed
allocation can yield large savings in the crowdsourcing budget (up to 49
percent points) while maintaining labeling accuracy. Finally, we envisage a
human-machine system for performing budget-optimized data analysis at a scale
beyond the feasibility of crowdsourcing
- …