Weaving risk identification into crowdsourcing lifecycle

© 2015 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license. Crowdsourcing enables companies and individuals as well to tap into the versatile knowledge, creativity, and talent of a large population of crowd contributors. Yet, crowdsourcing can expose companies to a myriad of risks that can have drastic impact on the profitability and competitive position. This paper presents a Risk Breakdown Structure (RBS) of crowdsourcing projects that spans the entire project\u27s lifecycle. The paper first reports on a lifecycle model that captures the main phases of a crowdsourcing project. It then identifies the risk factors associated with each phase of the crowdsourcing lifecycle and discusses the impact of these identified risk factors on the crowdsourcing company. The proposed RBS calls for the need to pay close attention to risk monitoring during each phase of the crowdsourcing lifecycle

A Glimpse Far into the Future: Understanding Long-term Crowd Worker Quality

Microtask crowdsourcing is increasingly critical to the creation of extremely large datasets. As a result, crowd workers spend weeks or months repeating the exact same tasks, making it necessary to understand their behavior over these long periods of time. We utilize three large, longitudinal datasets of nine million annotations collected from Amazon Mechanical Turk to examine claims that workers fatigue or satisfice over these long periods, producing lower quality work. We find that, contrary to these claims, workers are extremely stable in their quality over the entire period. To understand whether workers set their quality based on the task's requirements for acceptance, we then perform an experiment where we vary the required quality for a large crowdsourcing task. Workers did not adjust their quality based on the acceptance threshold: workers who were above the threshold continued working at their usual quality level, and workers below the threshold self-selected themselves out of the task. Capitalizing on this consistency, we demonstrate that it is possible to predict workers' long-term quality using just a glimpse of their quality on the first five tasks.Comment: 10 pages, 11 figures, accepted CSCW 201

An Empirical Analysis of User Participation on Crowdsourcing Platform: A Two-sided Network Market Perspective

Crowdsourcing has recently emerged as a new platform for matching the demand and supply between professionals and businesses who seek external expertise for business task execution. Driven by the unique features of the two-sided crowdsourcing markets (such as auction-style competition on quality by professionals), this study seeks to examine how the dynamics of the two-sided crowdsourcing platform affect customers and professionals’ strategic behaviors and market outcomes. Using longitudinal transaction data from a crowdsourcing websites, we plan to empirically examine how the participation of professionals and customers, task reward and task completion rate are affected by the characteristics of the professionals such as distribution of the winning professionals and their reputation. The results of our study are expected to contribute to the growing literature on crowdsourcing and provide important insights on the design and assessment of the sustainability and profitability of the crowdsourcing business model

Theoretical Underpinnings and Practical Challenges of Crowdsourcing as a Mechanism for Academic Study

Researchers in a variety of fields are increasingly adopting crowdsourcing as a reliable instrument for performing tasks that are either complex for humans and computer algorithms. As a result, new forms of collective intelligence have emerged from the study of massive crowd-machine interactions in scientific work settings as a field for which there is no known theory or model able to explain how it really works. Such type of crowd work uses an open participation model that keeps the scientific activity (including datasets, methods, guidelines, and analysis results) widely available and mostly independent from institutions, which distinguishes crowd science from other crowd-assisted types of participation. In this paper, we build on the practical challenges of crowd-AI supported research and propose a conceptual framework for addressing the socio-technical aspects of crowd science from a CSCW viewpoint. Our study reinforces a manifested lack of systematic and empirical research of the symbiotic relation of AI with human computation and crowd computing in scientific endeavors

A Work-Systems Approach to Classifying Risks in Crowdfunding Platforms: An Exploratory Analysis

Crowdfunding has attracted much attention in the last few years because it has opened up new pathways for projects to obtain financing from individuals who are non-professional investors via the Internet. While risk occupies a central role in crowdfunding, this notion has been an unexplored area in the information systems literature. To close this gap, we contribute to the literature by identifying the main risks in crowdfunding platforms. Using the Work Systems Risk Framework, we analyze main risks in three equity crowdfunding platforms: Crowdfunder, AngelList and Seedrs. Our findings indicate that operational risk, project management risk, cognitive skill risk, IP risk, quality risk, legal risk and vendor relationship risk factors to be important to crowdfunding platforms. Findings from this study are relevant to platform owners and regulators in assessing the risks of crowdfunding platforms

FROM MULTIPLE POLYGONS TO SINGLE GEOMETRY: OPTIMIZATION OF POLYGON INTEGRATION FOR CROWDSOURCED DATA

Paid crowdsourcing is a popular approach for creating training data in machine learning, but output quality can suffer from various drawbacks, such as noisy data. One solution is to obtain multiple acquisitions of the same dataset and perform integration steps, which can be challenging for geometries such as polygons. In this paper, we propose a raster-based polygon integration approach for the use of crowdsourced data, providing a solution for integrating multiple geometric shapes into single geometries. We analyze the effects of the choice of the integration threshold parameter for different sample sizes on the quality measures intersection over union (IoU) and Hausdorff distance, and provide a recommendation for its optimal selection based on empirical analysis. Additionally, further possibilities to improve integration results are explored, i.e., methods of filtering data before integration by outlier detection

Don’t Get Lost in the Crowd: Best Practices for Using Amazon’s Mechanical Turk in Behavioral Research

The use of Amazon’s Mechanical Turk (MTurk) to conduct academic research has steadily grown since its inception in 2005. The ability to control every aspect of a study, from sampling to collection, is extremely appealing to researchers. Unfortunately, the additional control offered through MTurk can also lead to poor data quality if researchers are not careful. Despite research on various aspects of data quality, participant compensation, and participant demographics, the academic literature still lacks a practical guide to the effective use of settings and features in MTurk for survey and experimental research. Therefore, the purpose of this tutorial is to provide researchers with a recommended set of best practices to follow before, during, and after collecting data via MTurk to ensure that responses are of the highest possible quality. We also recommend that editors and reviewers place more emphasis on the collection methods employed by researchers, rather than assume that all samples collected using a given online platform are of equal quality. We also recommend that editors and reviewers place more emphasis on the collection methods employed by researchers, rather than assuming that all samples collected using a given online platform are of equal quality