Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques and Assurance Actions

Crowdsourcing enables one to leverage on the intelligence and wisdom of potentially large groups of individuals toward solving problems. Common problems approached with crowdsourcing are labeling images, translating or transcribing text, providing opinions or ideas, and similar - all tasks that computers are not good at or where they may even fail altogether. The introduction of humans into computations and/or everyday work, however, also poses critical, novel challenges in terms of quality control, as the crowd is typically composed of people with unknown and very diverse abilities, skills, interests, personal objectives and technological resources. This survey studies quality in the context of crowdsourcing along several dimensions, so as to define and characterize it and to understand the current state of the art. Specifically, this survey derives a quality model for crowdsourcing tasks, identifies the methods and techniques that can be used to assess the attributes of the model, and the actions and strategies that help prevent and mitigate quality problems. An analysis of how these features are supported by the state of the art further identifies open issues and informs an outlook on hot future research directions.Comment: 40 pages main paper, 5 pages appendi

Do the Selfish Mimic Cooperators? Experimental Evidence from Finitely-Repeated Labor Markets

Experimental studies have consistently shown that cooperative outcomes can emerge even in finitely repeated games. Such outcomes are justified by existing reputation building models, which suggest that cooperative outcomes can be sustained if some subjects have other-regarding preferences. While the existence of other-regarding preferences is typically used to justify experimental outcomes, we are unaware of empirical studies that explicitly examine the interaction between cooperators (those with other-regarding preferences) and selfish subjects in sustaining cooperation. In this paper, we classify subjects as either selfish or cooperative using simple social preference games and then test for behavioral differences between the two types in a finitely-repeated labor market with unenforceable worker effort. Theory predicts, and our data confirms, that (1) selfish players mimic the actions of cooperators when trading partners can track the individual reputation of past partners and (2) selfish and cooperative types act differently when individual reputations cannot be tracked.contracts, relational contracts, implicit contracts, market interaction, experimental economics, repeated transaction, social preferences, reputation, firm latitude, finitely-repeated games

Beyond AMT: An Analysis of Crowd Work Platforms

While many competitor platforms to Amazon’s Mechanical Turk (AMT) now exist, little research has considered them. Such near-exclusive focus on AMT risks its particular vagaries and limitations overly shaping our understanding of crowd work and our field’s research questions and directions. To address this, we present a qualitative content analysis of seven alternative platforms. After organizing prior AMT studies around a set of key problem types encountered, we define our process for inducing categories for qualitative assessment of platforms. We then contrast the key problem types with AMT vs. platform features from content analysis, informing both methodology of use and directions for future research. Our cross-platform analysis represents the only such study by researchers for researchers, intended to enrich diversity of research on crowd work and accelerate progress.ye

College and University Ranking Systems: Global Perspectives and American Challenges

Examines how higher education ranking systems function, how other countries use ranking systems, and the impact of college rankings in the United States on student access, choice, and opportunity

Work Precarity and Gig Literacies in Online Freelancing

Many workers have been drawn to the gig economy by the promise of flexible, autonomous work, but scholars have highlighted how independent working arrangements also come with the drawbacks of precarity. Digital platforms appear to provide an alternative to certain aspects of precarity by helping workers find work consistently and securely. However, these platforms also introduce their own demands and constraints. Drawing on 20 interviews with online freelancers, 19 interviews with corresponding clients and a first-hand walkthrough of the Upwork platform, we identify critical literacies (what we call gig literacies), which are emerging around online freelancing. We find that gig workers must adapt their skills and work strategies in order to leverage platforms creatively and productively, and as a component of their ‘personal holding environment’. This involves not only using the resources provided by the platform effectively, but also negotiating or working around its imposed structures and control mechanisms

The Law of Employee Data: Privacy, Property, Governance

The availability of data related to the employment relationship has ballooned into an unruly mass of performance metrics, personal characteristics, biometric recordings, and creative output. The law governing this collection of information has been awkwardly split between privacy regulations and intellectual property rights, with employees generally losing on both ends. This Article rejects a binary approach that either carves out private spaces ineffectually or renders data into isolated pieces of ownership. Instead, the law should implement a hybrid system that provides workers with continuing input and control without blocking efforts at joint production. In addition, employers should have fiduciary responsibilities in managing employee data, and workers should have collective governance rights over the data’s collection and use

The Law of Employee Data: Privacy, Property, Governance

The availability of data related to the employment relationship has ballooned into an unruly mass of personal characteristics, performance metrics, biometric recordings, and creative output. The law governing this collection of information has been awkwardly split between privacy regulations and intellectual property rights, with employees generally losing on both ends. This Article rejects a binary approach that either carves out private spaces ineffectually or renders data into isolated pieces of ownership. Instead, the law should implement a hybrid system that provides workers with continuing input and control without blocking efforts at joint production. In addition, employers should have fiduciary responsibilities in managing employee data, and workers should have collective governance rights over the data’s collection and use