A robust consistency model of crowd workers in text labeling tasks

Crowdsourcing is a popular human-based model to acquire labeled data. Despite its ability to generate huge amounts of labelled data at moderate costs, it is susceptible to low quality labels. This can happen through unintentional or intentional errors by the crowd workers. Consistency is an important attribute of reliability. It is a practical metric that evaluates a crowd workers' reliability based on their ability to conform to themselves by yielding the same output when repeatedly given a particular input. Consistency has not yet been sufficiently explored in the literature. In this work, we propose a novel consistency model based on the pairwise comparisons method. We apply this model on unpaid workers. We measure the workers' consistency on tasks of labeling political text-based claims and study the effects of different duplicate task characteristics on their consistency. Our results show that the proposed model outperforms the current state-of-the-art models in terms of accuracy. This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0

In Their Shoes: A Structured Analysis of Job Demands, Resources, Work Experiences, and Platform Commitment of Crowdworkers in China

Despite the growing interest in crowdsourcing, this new labor model has recently received severe criticism. The most important point of this criticism is that crowdworkers are often underpaid and overworked. This severely affects job satisfaction and productivity. Although there is a growing body of evidence exploring the work experiences of crowdworkers in various countries, there have been a very limited number of studies to the best of our knowledge exploring the work experiences of Chinese crowdworkers. In this paper we aim to address this gap. Based on a framework of well-established approaches, namely the Job Demands-Resources model, the Work Design Questionnaire, the Oldenburg Burnout Inventory, the Utrecht Work Engagement Scale, and the Organizational Commitment Questionnaire, we systematically study the work experiences of 289 crowdworkers who work for ZBJ.com - the most popular Chinese crowdsourcing platform. Our study examines these crowdworker experiences along four dimensions: (1) crowdsourcing job demands, (2) job resources available to the workers, (3) crowdwork experiences, and (4) platform commitment. Our results indicate significant differences across the four dimensions based on crowdworkers\u27 gender, education, income, job nature, and health condition. Further, they illustrate that different crowdworkers have different needs and threshold of demands and resources and that this plays a significant role in terms of moderating the crowdwork experience and platform commitment. Overall, our study sheds light to the work experiences of the Chinese crowdworkers and at the same time contributes to furthering understandings related to the work experiences of crowdworkers

A Statistical Analysis of the Aggregation of Crowdsourced Labels

Crowdsourcing, due to its inexpensive and timely nature, has become a popular method of collecting data that is difficult for computers to generate. We focus on using this method of human computation to gather labels for classification tasks, to be used for machine learning. However, data gathered this way may be of varying quality, ranging from spam to perfect. We aim to maintain the cost-effective property of crowdsourcing, while also obtaining quality results. Towards a solution, we have multiple workers label the same problem instance, aggregating the responses into one label afterwards. We study what aggregation method to use, and what guarantees we can provide on its estimates. Different crowdsourcing models call for different techniques – we outline and organize various directions taken in the literature, and focus on the Dawid-Skene model. In this setting each instance has a true label, workers are independent, and the performance of each individual is assumed to be uniform over all instances, in the sense that she has an inherent skill that governs the probability with which she labels correctly. Her skill is unknown to us. Aggregation methods aim to find the true label of each task based solely on the labels the workers reported. We measure the performance of these methods by the probability with which the estimates they output match the true label. In practice, a popular procedure is to run the EM algorithm to find estimates of the skills and labels. However, this method is not directly guaranteed to perform well in our measure. We collect and evaluate theoretical results that bound the error of various aggregation methods, including specific variants of EM. Finally, we prove a guarantee on the error suffered by the maximum likelihood estimator, the global optima of the function that EM aims to numerically optimize

Social informatics

5th International Conference, SocInfo 2013, Kyoto, Japan, November 25-27, 2013, Proceedings</p