Crowdsourcing and Scholarly Culture: Understanding Expertise in an Age of Popularism

The increasing volume of digital material available to the humanities creates clear potential for crowdsourcing. However, tasks in the digital humanities typically do not satisfy the standard requirement for decomposition into microtasks each of which must require little expertise on behalf of the worker and little context of the broader task. Instead, humanities tasks require scholarly knowledge to perform and even where sub-tasks can be extracted, these often involve broader context of the document or corpus from which they are extracted. That is the tasks are macrotasks, resisting simple decomposition. Building on a case study from musicology, the In Concert project, we will explore both the barriers to crowdsourcing in the creation of digital corpora and also examples where elements of automatic processing or less-expert work are possible in a broader matrix that also includes expert microtasks and macrotasks. Crucially we will see that the macrotask–microtask distinction is nuanced: it is often possible to create a partial decomposition into less-expert microtasks with residual expert macrotasks, and crucially do this in ways that preserve scholarly values

Crowdsourcing and Scholarly Culture: Understanding Expertise in an Age of Popularism

The increasing volume of digital material available to the humanities creates clear potential for crowdsourcing. However, tasks in the digital humanities typically do not satisfy the standard requirement for decomposition into microtasks each of which must require little expertise on behalf of the worker and little context of the broader task. Instead, humanities tasks require scholarly knowledge to perform and even where sub-tasks can be extracted, these often involve broader context of the document or corpus from which they are extracted. That is the tasks are macrotasks, resisting simple decomposition. Building on a case study from musicology, the In Concert project, we will explore both the barriers to crowdsourcing in the creation of digital corpora and also examples where elements of automatic processing or less-expert work are possible in a broader matrix that also includes expert microtasks and macrotasks. Crucially we will see that the macrotask–microtask distinction is nuanced: it is often possible to create a partial decomposition into less-expert microtasks with residual expert macrotasks, and crucially do this in ways that preserve scholarly values

An Examination of the Work Practices of Crowdfarms

Crowdsourcing is a new value creation business model. Annual revenue of the Chinese market alone is hundreds of millions of dollars, yet few studies have focused on the practices of the Chinese crowdsourcing workforce, and those that do mainly focus on solo crowdworkers. We have extended our study of solo crowdworker practices to include crowdfarms, a relatively new entry to the gig economy: small companies that carry out crowdwork as a key part of their business. We report here on interviews of people who work in 53 crowdfarms. We describe how crowdfarms procure jobs, carry out macrotasks and microtasks, manage their reputation, and employ different management practices to motivate crowdworkers and customers

Who wants to cooperate-and why? Attitude and perception of crowd workers in online labor markets

Existing literature and studies predominantly focus on how crowdsource workers individually complete tasks and projects. Our study examines crowdsource workers' willingness to work collaboratively. We report results from a survey of 122 workers on a leading online labor platform (Upwork) to examine crowd workers' behavioral preferences for collaboration and explore several antecedents of cooperative behaviors. We then test if actual cooperative behavior matches with workers' behavioral preferences through an incentivized social dilemma experiment. We find that respondents cooperate at a higher rate (85%) than reported in previous comparable studies (between 50-75%). This high rate of cooperation is likely explained by an ingroup bias. Using a sequential mediation model we demonstrate the importance of a sense of shared expectations and accountability for cooperation. We contribute to a better understanding of the potential for collaborative work in crowdsourcing by accessing if and what social factors and collective culture exist among crowd workers. We discuss the implications of our results for platform designers by highlighting the importance of platform features that promote shared expectations and improve accountability. Overall, contrary to existing literature and predictions, our results suggest that crowd workers display traits that are more consistent with belonging to a coherent group with a shared collective culture, rather than being anonymous actors in a transaction-based market

Crowdsourcing Coordination: A Review and Research Agenda for Crowdsourcing Coordination Used for Macro-tasks

Crowdsourcing has become a widely accepted approach to leveraging the skills and expertise of others to accomplish work. Despite the potential of crowdsourcing to tackle complex problems, it has often been used to address simple micro-tasks. To tackle more complex macro-tasks, more attention is needed to better comprehend crowd coordination. Crowd coordination is defined as the synchronization of crowd workers in an attempt to direct and align their efforts in pursuit of a shared goal. The goal of this chapter is to advance our understanding of crowd coordination to tackle complex macro-tasks. To accomplish this, we have three objectives. First, we review popular theories of coordination. Second, we examine the current approaches to crowd coordination in the HCI and CSCW literature. Finally, the chapter identifies shortcomings in the literature and proposes a research agenda directed at advancing our understanding of crowd coordination needed to address complex macro-tasks.National Science Foundation grant CHS-1617820Peer Reviewedhttps://deepblue.lib.umich.edu/bitstream/2027.42/150620/1/Kim and Robert 2019 Preprint Chapter 2.pdfDescription of Kim and Robert 2019 Preprint Chapter 2.pdf : Preprint Versio

Self-Organizing Teams in Online Work Settings

As the volume and complexity of distributed online work increases, the collaboration among people who have never worked together in the past is becoming increasingly necessary. Recent research has proposed algorithms to maximize the performance of such teams by grouping workers according to a set of predefined decision criteria. This approach micro-manages workers, who have no say in the team formation process. Depriving users of control over who they will work with stifles creativity, causes psychological discomfort and results in less-than-optimal collaboration results. In this work, we propose an alternative model, called Self-Organizing Teams (SOTs), which relies on the crowd of online workers itself to organize into effective teams. Supported but not guided by an algorithm, SOTs are a new human-centered computational structure, which enables participants to control, correct and guide the output of their collaboration as a collective. Experimental results, comparing SOTs to two benchmarks that do not offer user agency over the collaboration, reveal that participants in the SOTs condition produce results of higher quality and report higher teamwork satisfaction. We also find that, similarly to machine learning-based self-organization, human SOTs exhibit emergent collective properties, including the presence of an objective function and the tendency to form more distinct clusters of compatible teammates

A Task-Interdependency Model of Complex Collaboration Towards Human-Centered Crowd Work

Models of crowdsourcing and human computation often assume that individuals independently carry out small, modular tasks. However, while these models have successfully shown how crowds can accomplish significant objectives, they can inadvertently advance a less than human view of crowd workers and fail to capture the unique human capacity for complex collaborative work. We present a model centered on interdependencies -- a phenomenon well understood to be at the core of collaboration -- that allows one to formally reason about diverse challenges to complex collaboration. Our model represents tasks as an interdependent collection of subtasks, formalized as a task graph. We use it to explain challenges to scaling complex collaborative work, underscore the importance of expert workers, reveal critical factors for learning on the job, and explore the relationship between coordination intensity and occupational wages. Using data from O*NET and the Bureau of Labor Statistics, we introduce an index of occupational coordination intensity to validate our theoretical predictions. We present preliminary evidence that occupations with greater coordination intensity are less exposed to displacement by AI, and discuss opportunities for models that emphasize the collaborative capacities of human workers, bridge models of crowd work and traditional work, and promote AI in roles augmenting human collaboration