1,366 research outputs found

    Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques and Assurance Actions

    Get PDF
    Crowdsourcing enables one to leverage on the intelligence and wisdom of potentially large groups of individuals toward solving problems. Common problems approached with crowdsourcing are labeling images, translating or transcribing text, providing opinions or ideas, and similar - all tasks that computers are not good at or where they may even fail altogether. The introduction of humans into computations and/or everyday work, however, also poses critical, novel challenges in terms of quality control, as the crowd is typically composed of people with unknown and very diverse abilities, skills, interests, personal objectives and technological resources. This survey studies quality in the context of crowdsourcing along several dimensions, so as to define and characterize it and to understand the current state of the art. Specifically, this survey derives a quality model for crowdsourcing tasks, identifies the methods and techniques that can be used to assess the attributes of the model, and the actions and strategies that help prevent and mitigate quality problems. An analysis of how these features are supported by the state of the art further identifies open issues and informs an outlook on hot future research directions.Comment: 40 pages main paper, 5 pages appendi

    Crowdsourcing in Computer Vision

    Full text link
    Computer vision systems require large amounts of manually annotated data to properly learn challenging visual concepts. Crowdsourcing platforms offer an inexpensive method to capture human knowledge and understanding, for a vast number of visual perception tasks. In this survey, we describe the types of annotations computer vision researchers have collected using crowdsourcing, and how they have ensured that this data is of high quality while annotation effort is minimized. We begin by discussing data collection on both classic (e.g., object recognition) and recent (e.g., visual story-telling) vision tasks. We then summarize key design decisions for creating effective data collection interfaces and workflows, and present strategies for intelligently selecting the most important data instances to annotate. Finally, we conclude with some thoughts on the future of crowdsourcing in computer vision.Comment: A 69-page meta review of the field, Foundations and Trends in Computer Graphics and Vision, 201

    A Conceptual Probabilistic Framework for Annotation Aggregation of Citizen Science Data

    Get PDF
    Over the last decade, hundreds of thousands of volunteers have contributed to science by collecting or analyzing data. This public participation in science, also known as citizen science, has contributed to significant discoveries and led to publications in major scientific journals. However, little attention has been paid to data quality issues. In this work we argue that being able to determine the accuracy of data obtained by crowdsourcing is a fundamental question and we point out that, for many real-life scenarios, mathematical tools and processes for the evaluation of data quality are missing. We propose a probabilistic methodology for the evaluation of the accuracy of labeling data obtained by crowdsourcing in citizen science. The methodology builds on an abstract probabilistic graphical model formalism, which is shown to generalize some already existing label aggregation models. We show how to make practical use of the methodology through a comparison of data obtained from different citizen science communities analyzing the earthquake that took place in Albania in 2019

    Ranked by Truth Metrics: A New Communication Method Approach, on Crowd-Sourced Fact-Checking Platforms for Journalistic and Social Media Content

    Get PDF
    Fake news, misinformation, and non-true stories create a definite threat to the world's public sphere. Fake news contaminates democracy by blurring the sight and the vision, or by altering the beliefs of citizens on simple everyday matters but also on significant matters such as vaccination, politics, social issues, or public health. Lots of efforts have been conducted in order to tackle the phenomenon. Fact-checking platforms consist of a major step in this issue. Certain cases of fact-checking platforms worldwide seem to work properly and fulfill their strategic goals, although functional and other issues might emerge. This study comes to take the fact-checking platform evolution one step beyond by proposing a new communication model for fake news detection and busting. The proposed model's blueprint is based on the Greek "Ellinika Hoaxes" fact-checking platform with some critical reinforcements: More extensive use of crowdsourcing strategies for detecting and busting non-true stories with the aid of AI chatbots in order not only to bust non-true stories but also to rank news outlets, writers, social media personas and journalists for their credibility. This way, serious news outlets, journalists, and media professionals can build their trust and be ranked for the credibility of their services for a more trustful and democratic public sphere. 
    corecore