An Efficient and Truthful Pricing Mechanism for Team Formation in Crowdsourcing Markets

In a crowdsourcing market, a requester is looking to form a team of workers to perform a complex task that requires a variety of skills. Candidate workers advertise their certified skills and bid prices for their participation. We design four incentive mechanisms for selecting workers to form a valid team (that can complete the task) and determining each individual worker's payment. We examine profitability, individual rationality, computational efficiency, and truthfulness for each of the four mechanisms. Our analysis shows that TruTeam, one of the four mechanisms, is superior to the others, particularly due to its computational efficiency and truthfulness. Our extensive simulations confirm the analysis and demonstrate that TruTeam is an efficient and truthful pricing mechanism for team formation in crowdsourcing markets.Comment: 6 pages, 3 figures, IEEE ICC 201

Incentivizing the Workers for Truth Discovery in Crowdsourcing with Copiers

Crowdsourcing has become an efficient paradigm for performing large scale tasks. Truth discovery and incentive mechanism are fundamentally important for the crowdsourcing system. Many truth discovery methods and incentive mechanisms for crowdsourcing have been proposed. However, most of them cannot be applied to deal with the crowdsourcing with copiers. To address the issue, we formulate the problem of maximizing the social welfare such that all tasks can be completed with the least confidence for truth discovery. We design an incentive mechanism consisting truth discovery stage and reverse auction stage. In truth discovery stage, we estimate the truth for each task based on both the dependence and accuracy of workers. In reverse auction stage, we design a greedy algorithm to select the winners and determine the payment. Through both rigorous theoretical analysis and extensive simulations, we demonstrate that the proposed mechanisms achieve computational efficiency, individual rationality, truthfulness, and guaranteed approximation. Moreover, our truth discovery method shows prominent advantage in terms of precision when there are copiers in the crowdsourcing systems.Comment: 12 pages, 8 figure

Bayesian Budget Feasibility with Posted Pricing

We consider the problem of budget feasible mechanism design proposed by Singer (2010), but in a Bayesian setting. A principal has a public value for hiring a subset of the agents and a budget, while the agents have private costs for being hired. We consider both additive and submodular value functions of the principal. We show that there are simple, practical, ex post budget balanced posted pricing mechanisms that approximate the value obtained by the Bayesian optimal mechanism that is budget balanced only in expectation. A main motivating application for this work is the crowdsourcing large projects, e.g., on Mechanical Turk, where workers are drawn from a large population and posted pricing is standard. Our analysis methods relate to contention resolution schemes in submodular optimization of Vondrak et al. (2011) and the correlation gap analysis of Yan (2011)

General Privacy-Preserving Verifiable Incentive Mechanism for Crowdsourcing Markets

In crowdsourcing markets, there are two different type jobs, i.e. homogeneous jobs and heterogeneous jobs, which need to be allocated to workers. Incentive mechanisms are essential to attract extensive user participating for achieving good service quality, especially under a given budget constraint condition. To this end, recently, Singer et al. propose a novel class of auction mechanisms for determining near-optimal prices of tasks for crowdsourcing markets constrained by the given budget. Their mechanisms are very useful to motivate extensive user to truthfully participate in crowdsourcing markets. Although they are so important, there still exist many security and privacy challenges in real-life environments. In this paper, we present a general privacy-preserving verifiable incentive mechanism for crowdsourcing markets with the budget constraint, not only to exploit how to protect the bids and assignments' privacy, and the chosen winners' privacy in crowdsourcing markets with homogeneous jobs and heterogeneous jobs and identity privacy from users, but also to make the verifiable payment between the platform and users for crowdsourcing applications. Results show that our general privacy-preserving verifiable incentive mechanisms achieve the same results as the generic one without privacy preservation.Comment: This paper has been withdrawn by the author due to a crucial sign error in equation 1 and Figure

Rating Protocol Design for Extortion and Cooperation in the Crowdsourcing Contest Dilemma

Crowdsourcing has emerged as a paradigm for leveraging human intelligence and activity to solve a wide range of tasks. However, strategic workers will find enticement in their self-interest to free-ride and attack in a crowdsourcing contest dilemma game. Hence, incentive mechanisms are of great importance to overcome the inefficiency of the socially undesirable equilibrium. Existing incentive mechanisms are not effective in providing incentives for cooperation in crowdsourcing competitions due to the following features: heterogeneous workers compete against each other in a crowdsourcing platform with imperfect monitoring. In this paper, we take these features into consideration, and develop a novel game-theoretic design of rating protocols, which integrates binary rating labels with differential pricing to maximize the requester's utility, by extorting selfish workers and enforcing cooperation among them. By quantifying necessary and sufficient conditions for the sustainable social norm, we formulate the problem of maximizing the revenue of the requester among all sustainable rating protocols, provide design guidelines for optimal rating protocols, and design a low-complexity algorithm to select optimal design parameters which are related to differential punishments and pricing schemes. Simulation results demonstrate how intrinsic parameters impact on design parameters, as well as the performance gain of the proposed rating protocol.Comment: 13 pages, 21 figure

Beyond AMT: An Analysis of Crowd Work Platforms

While Amazon's Mechanical Turk (AMT) helped launch the paid crowd work industry eight years ago, many new vendors now offer a range of alternative models. Despite this, little crowd work research has explored other platforms. Such near-exclusive focus risks letting AMT's particular vagaries and limitations overly shape our understanding of crowd work and the research questions and directions being pursued. To address this, we present a cross-platform content analysis of seven crowd work platforms. We begin by reviewing how AMT assumptions and limitations have influenced prior research. Next, we formulate key criteria for characterizing and differentiating crowd work platforms. Our analysis of platforms contrasts them with AMT, informing both methodology of use and directions for future research. Our cross-platform analysis represents the only such study by researchers for researchers, intended to further enrich the diversity of research on crowd work and accelerate progress

Simple and Efficient Budget Feasible Mechanisms for Monotone Submodular Valuations

We study the problem of a budget limited buyer who wants to buy a set of items, each from a different seller, to maximize her value. The budget feasible mechanism design problem aims to design a mechanism which incentivizes the sellers to truthfully report their cost, and maximizes the buyer's value while guaranteeing that the total payment does not exceed her budget. Such budget feasible mechanisms can model a buyer in a crowdsourcing market interested in recruiting a set of workers (sellers) to accomplish a task for her. This budget feasible mechanism design problem was introduced by Singer in 2010. There have been a number of improvements on the approximation guarantee of such mechanisms since then. We consider the general case where the buyer's valuation is a monotone submodular function. We offer two general frameworks for simple mechanisms, and by combining these frameworks, we significantly improve on the best known results for this problem, while also simplifying the analysis. For example, we improve the approximation guarantee for the general monotone submodular case from 7.91 to 5; and for the case of large markets (where each individual item has negligible value) from 3 to 2.58. More generally, given an $r$ approximation algorithm for the optimization problem (ignoring incentives), our mechanism is a $r+1$ approximation mechanism for large markets, an improvement from $2r^2$. We also provide a similar parameterized mechanism without the large market assumption, where we achieve a $4r+1$ approximation guarantee

Optimal Prizes for All-Pay Contests in Heterogeneous Crowdsourcing

Incentives are key to the success of crowdsourcing which heavily depends on the level of user participation. This paper designs an incentive mechanism to motivate a heterogeneous crowd of users to actively participate in crowdsourcing campaigns. We cast the problem in a new, asymmetric all-pay contest model with incomplete information, where an arbitrary n of users exert irrevocable effort to compete for a prize tuple. The prize tuple is an array of prize functions as opposed to a single constant prize typically used by conventional contests. We design an optimal contest that (a) induces the maximum profit---total user effort minus the prize payout---for the crowdsourcer, and (b) ensures users to strictly have the incentive to participate. In stark contrast to intuition and prior related work, our mechanism induces an equilibrium in which heterogeneous users behave independently of one another as if they were in a homogeneous setting. This newly discovered property, which we coin as strategy autonomy (SA), is of practical significance: it (a) reduces computational and storage complexity by n-fold for each user, (b) increases the crowdsourcer's revenue by counteracting an effort reservation effect existing in asymmetric contests, and (c) neutralizes the (almost universal) law of diminishing marginal returns (DMR). Through an extensive numerical case study, we demonstrate and scrutinize the superior profitability of our mechanism, as well as draw insights into the SA property.Comment: 9 pages, 4 figures. IEEE MASS 201

On Cost-Effective Incentive Mechanisms in Microtask Crowdsourcing

While microtask crowdsourcing provides a new way to solve large volumes of small tasks at a much lower price compared with traditional in-house solutions, it suffers from quality problems due to the lack of incentives. On the other hand, providing incentives for microtask crowdsourcing is challenging since verifying the quality of submitted solutions is so expensive that will negate the advantage of microtask crowdsourcing. We study cost-effective incentive mechanisms for microtask crowdsourcing in this paper. In particular, we consider a model with strategic workers, where the primary objective of a worker is to maximize his own utility. Based on this model, we analyze two basic mechanisms widely adopted in existing microtask crowdsourcing applications and show that, to obtain high quality solutions from workers, their costs are constrained by some lower bounds. We then propose a cost-effective mechanism that employs quality-aware worker training as a tool to stimulate workers to provide high quality solutions. We prove theoretically that the proposed mechanism, when properly designed, can obtain high quality solutions with an arbitrarily low cost. Beyond its theoretical guarantees, we further demonstrate the effectiveness of our proposed mechanisms through a set of behavioral experiments

A Budget Feasible Peer Graded Mechanism For IoT-Based Crowdsourcing

We develop and extend a line of recent works on the design of mechanisms for heterogeneous tasks assignment problem in 'crowdsourcing'. The budgeted market we consider consists of multiple task requesters and multiple IoT devices as task executers; where each task requester is endowed with a single distinct task along with the publicly known budget. Also, each IoT device has valuations as the cost for executing the tasks and quality, which are private. Given such scenario, the objective is to select a subset of IoT devices for each task, such that the total payment made is within the allotted quota of the budget while attaining a threshold quality. For the purpose of determining the unknown quality of the IoT devices, we have utilized the concept of peer grading. In this paper, we have carefully crafted a truthful budget feasible mechanism; namely TUBE-TAP for the problem under investigation that also allows us to have the true information about the quality of the IoT devices. The simulations are performed in order to measure the efficacy of our proposed mechanism.Comment: In Version 2, errors are fixe