Analytic Methods for Optimizing Realtime Crowdsourcing

Realtime crowdsourcing research has demonstrated that it is possible to recruit paid crowds within seconds by managing a small, fast-reacting worker pool. Realtime crowds enable crowd-powered systems that respond at interactive speeds: for example, cameras, robots and instant opinion polls. So far, these techniques have mainly been proof-of-concept prototypes: research has not yet attempted to understand how they might work at large scale or optimize their cost/performance trade-offs. In this paper, we use queueing theory to analyze the retainer model for realtime crowdsourcing, in particular its expected wait time and cost to requesters. We provide an algorithm that allows requesters to minimize their cost subject to performance requirements. We then propose and analyze three techniques to improve performance: push notifications, shared retainer pools, and precruitment, which involves recalling retainer workers before a task actually arrives. An experimental validation finds that precruited workers begin a task 500 milliseconds after it is posted, delivering results below the one-second cognitive threshold for an end-user to stay in flow.Comment: Presented at Collective Intelligence conference, 201

Realtime crowdsourcing with payment of idle workers in the Retainer Model

The realtime applications of crowdsourcing are a very promising topic, due to its high potentialities, for example in marketing, security or telecommunication applications. Realtime crowdsourcing ensures that solutions to a given problem are obtained in the shortest possible time using collective intelligence. In order to be ready to carry out any requested task in realtime, crowdworkers must be available at any time. Here we focus on the payment of crowdworkers and on the trade-off between the expected waiting time for a task to be carried out and the number of workers in the pool that should not become too large otherwise the total cost increases. In particular we consider the, so called, Retainer Model in which crowdworkers are paid in order to be ready to carry out any requested task in realtime. The Retainer Model considers an expected total cost which takes into account both the amount paid to a crowdworker to be in idle-state and the loss when the task is not completed in realtime. After checking the existence of a minimum cost we characterize the optimal number of crowdworkers, and suggest a practical and quick way to obtain it. Moreover, we analyse the sensitivity of the optimal number of crowdworkers with respect to different task intensities

Optimization in Knowledge-Intensive Crowdsourcing

We present SmartCrowd, a framework for optimizing collaborative knowledge-intensive crowdsourcing. SmartCrowd distinguishes itself by accounting for human factors in the process of assigning tasks to workers. Human factors designate workers' expertise in different skills, their expected minimum wage, and their availability. In SmartCrowd, we formulate task assignment as an optimization problem, and rely on pre-indexing workers and maintaining the indexes adaptively, in such a way that the task assignment process gets optimized both qualitatively, and computation time-wise. We present rigorous theoretical analyses of the optimization problem and propose optimal and approximation algorithms. We finally perform extensive performance and quality experiments using real and synthetic data to demonstrate that adaptive indexing in SmartCrowd is necessary to achieve efficient high quality task assignment.Comment: 12 page

Realtime crowdsourcing with payment of idle workers in the Retainer Model

The realtime applications of crowdsourcing are a very promising topic, due to its high potentialities, for example in marketing, security or telecommunication applications. Realtime crowdsourcing ensures that solutions to a given problem are obtained in the shortest possible time using collective intelligence. In order to be ready to carry out any requested task in realtime, crowdworkers must be available at any time. Here we focus on the payment of crowdworkers and on the trade-off between the expected waiting time for a task to be carried out and the number of workers in the pool that should not become too large otherwise the total cost increases. In particular we consider the, so called, Retainer Model in which crowdworkers are paid in order to be ready to carry out any requested task in realtime. The Retainer Model considers an expected total cost which takes into account both the amount paid to a crowdworker to be in idle-state and the loss when the task is not completed in realtime. After checking the existence of a minimum cost we characterize the optimal number of crowdworkers, and suggest a practical and quick way to obtain it. Moreover, we analyse the sensitivity of the optimal number of crowdworkers with respect to different task intensities