Considering Human Aspects on Strategies for Designing and Managing Distributed Human Computation

A human computation system can be viewed as a distributed system in which the processors are humans, called workers. Such systems harness the cognitive power of a group of workers connected to the Internet to execute relatively simple tasks, whose solutions, once grouped, solve a problem that systems equipped with only machines could not solve satisfactorily. Examples of such systems are Amazon Mechanical Turk and the Zooniverse platform. A human computation application comprises a group of tasks, each of them can be performed by one worker. Tasks might have dependencies among each other. In this study, we propose a theoretical framework to analyze such type of application from a distributed systems point of view. Our framework is established on three dimensions that represent different perspectives in which human computation applications can be approached: quality-of-service requirements, design and management strategies, and human aspects. By using this framework, we review human computation in the perspective of programmers seeking to improve the design of human computation applications and managers seeking to increase the effectiveness of human computation infrastructures in running such applications. In doing so, besides integrating and organizing what has been done in this direction, we also put into perspective the fact that the human aspects of the workers in such systems introduce new challenges in terms of, for example, task assignment, dependency management, and fault prevention and tolerance. We discuss how they are related to distributed systems and other areas of knowledge.Comment: 3 figures, 1 tabl

Reputation Agent: Prompting Fair Reviews in Gig Markets

Our study presents a new tool, Reputation Agent, to promote fairer reviews from requesters (employers or customers) on gig markets. Unfair reviews, created when requesters consider factors outside of a worker's control, are known to plague gig workers and can result in lost job opportunities and even termination from the marketplace. Our tool leverages machine learning to implement an intelligent interface that: (1) uses deep learning to automatically detect when an individual has included unfair factors into her review (factors outside the worker's control per the policies of the market); and (2) prompts the individual to reconsider her review if she has incorporated unfair factors. To study the effectiveness of Reputation Agent, we conducted a controlled experiment over different gig markets. Our experiment illustrates that across markets, Reputation Agent, in contrast with traditional approaches, motivates requesters to review gig workers' performance more fairly. We discuss how tools that bring more transparency to employers about the policies of a gig market can help build empathy thus resulting in reasoned discussions around potential injustices towards workers generated by these interfaces. Our vision is that with tools that promote truth and transparency we can bring fairer treatment to gig workers.Comment: 12 pages, 5 figures, The Web Conference 2020, ACM WWW 202

Innovation is created by humans, not by systems: an exploration of user involvement in living labs: user motivation versus lead user criteria

The past few years companies have become more interested in involving users during the production process of their products. On the other hand, a group of users started to innovate on their own. Users also became interested in becoming part of the production processes themselves. Certain users experience certain needs earlier than others and they enjoy finding solutions for these needs. They are called Lead Users (von Hippel, 2005). Living Labs are one possibility for users to realize this interest to innovate. iLab.o, the Living Lab division of iMinds, has been organizing Living Lab research since 2009. To get a better view on the motivations of this panel, we analyzed the behavior of the involved users from September 2009 to December 2013. We tried to detect Lead Users, but it is not obvious to define people as Lead Users because of the different used definitions. Instead, we divided this panel into three types of users based on the intensity of their involvement: passive, sleeping and active users. A small group of users is extremely active and are been defined as “alpha users”. Based on interviews with these alpha users in November and December 2013, a better view on their motivations to keep participating in Living Lab research was made. In this paper we focus on the participation of these different user types in one research phase type within Living Lab research, more specifically co-creation sessions. By means of a comparative case study, we tried to get a better understanding of the behavior of the different user types. It became clear that in order to keep the panel involved it is important to focus on community building

Organizational Learning with Crowdsourcing: The Revelatory Case of LEGO

Extant organizational learning theory conceptualizes organizational learning as an internal, member-based process, sometimes supported by, yet often independent of, IT. Recently, however, several organizations have begun to involve non-members systematically in their learning by using crowdsourcing, a form of open innovation enabled by state-of-the-art IT. We examine the phenomenon of IT-enabled organizational learning with crowdsourcing in a longitudinal revelatory case study of one such organization, LEGO (2010-14). We studied the LEGO Cuusoo crowdsourcing platform’s secret test in Japan, its widely recognized global launch, and its success in generating top-selling LEGO models. Based on an analysis of how crowdsourcing contributes to the organizational learning at LEGO, we propose the “ambient organizational learning” framework. The framework accommodates both traditional, member-based organizational learning and IT-enabled, non-member-based organizational learning with crowdsourcing

Aplikacije utemeljene na mnoštvu i društveni izazovi

The spread of mobile technology and ubiquitous connectivity have opened great possibilities for the implementation of applications that leverage data generated by normal users’ interactions on the web. As a consequence, there is a growing interest in crowd-based applications, namely those programs that involve people in a participatory or opportunistic way. In many cases, data can be gathered automatically without user intervention and, in some cases, even without their explicit knowledge. The possibility to elude a users’ awareness fosters concerns regarding the potential risks hidden inside crowd-based applications. These applications might compromise the privacy of citizens, whilst data collected by them might be used to manipulate people’s opinions. The governance of technology is a controversial area, and there is a wide array of different positions on the matter. There are those who dogmatically argue the positive value of technology, while others interpret the ongoing digital advancements as a dystopian menace. This article focuses on crowd-based applications, highlighting some societal challenges and risks that they may present. Technology runs so fast that it is challenging to keep pace with the changes brought by the digital revolution. However, an effort is required to extend the depth of digital knowledge of citizens and involve them in the use of the new technologies, and in this endeavor, greater knowledge is an essential step in any critical process.Širenje mobilne tehnologije i sveprisutna povezanost otvorili su velike mogućnosti za upotrebu aplikacija koje iskorištavaju podatke generirane normalnim interakcijama korisnika na webu. Kao posljedica toga, sve je veći interes za aplikacije utemeljene na mnoštvu (engl. crowd-based applications), za one programe koji uključuju ljude na participativni ili oportunistički način. U mnogim se slučajevima podaci mogu prikupljati automatski, bez djelovanja korisnika, a u nekim slučajevima čak i bez njihova izričitog znanja. Mogućnost izbjegavanja svjesnosti korisnika potiče zabrinutosti u vezi s potencijalnim rizicima koji su skriveni u aplikacijama utemeljenim na mnoštvu. Te aplikacije mogu ugroziti privatnost građana, dok bi se prikupljeni podaci mogli koristiti za manipuliranje stavovima ljudi. Upravljanje tehnologijom kontroverzno je područje i o tom pitanju postoji mnoštvo različitih stajališta. Neki dogmatski zastupaju pozitivne vrijednosti tehnologije, dok drugi digitalni napredak tumače kao distopijsku prijetnju. Rad se usredotočuje na aplikacije utemeljene na mnoštvu, ističući neke društvene izazove i rizike koje mogu predstavljati. Tehnologija napreduje tako brzo da je izazovno biti u tijeku s promjenama koje je donijela digitalna revolucija. No, potrebno je pokušati produbiti digitalno znanje građana i uključiti ih u upotrebu novih tehnologija, a u tom je poduhvatu veće znanje temeljni korak u svakom kritičnom procesu

Supporting Answerers with Feedback in Social Q&A

Prior research has examined the use of Social Question and Answer (Q&A) websites for answer and help seeking. However, the potential for these websites to support domain learning has not yet been realized. Helping users write effective answers can be beneficial for subject area learning for both answerers and the recipients of answers. In this study, we examine the utility of crowdsourced, criteria-based feedback for answerers on a student-centered Q&A website, Brainly.com. In an experiment with 55 users, we compared perceptions of the current rating system against two feedback designs with explicit criteria (Appropriate, Understandable, and Generalizable). Contrary to our hypotheses, answerers disagreed with and rejected the criteria-based feedback. Although the criteria aligned with answerers' goals, and crowdsourced ratings were found to be objectively accurate, the norms and expectations for answers on Brainly conflicted with our design. We conclude with implications for the design of feedback in social Q&A.Comment: Published in Proceedings of the Fifth Annual ACM Conference on Learning at Scale, Article No. 10, London, United Kingdom. June 26 - 28, 201

Co-producing Knowledge Online

Knowledge production today relies increasingly on exchanges between groups of people who connect through the Internet. This can happen in many forms that include, for example, consulting and amending Wikipedia entries, engaging in Twitter conversations about a certain topic, or developing research software by building on existing code released under a license that allows free sharing, modification and reuse. Other kinds of collaborative research are enabled by more bespoke websites built for specific institutions or groups, such as the Smithsonian Transcription Centre, which was created to involve interested volunpeers (volunteers who are viewed as peers) in the digitisation of collections that support multiple research agendas. The British Library has also recently embraced a similar goal, setting up the LibCrowds platform, while adventure seekers can connect to GlobalXplorer and inspect satellite images to identify signs of looting and assist with understanding the current state of preservation of archaeology-rich landscapes worldwide. For nature lovers, Snapshot Serengeti offers the possibility to ‘observe animals in the wild’ and help to answer questions about the ways in which competing species coexist. All of these processes have become possible thanks to the wide diffusion of the Internet, and the emergence of online public spaces from an interactive and interconnected World Wide Web. This kind of web has enabled new practices of data and information generation, sharing and aggregation, but, arguably, the collaborative production (and consumption) of knowledge is sometimes so deeply embedded in our personal and professional lives that we do not always pause to reflect on its nature and deeper implications. 1 The aim of this review is to bring attention to these issues by addressing a number of questions relating to online research collaborations established between stakeholders within and beyond the academy. How can collaborative research be strategically and effectively designed online? What are its roots and traditions? What values can it generate for participants? What effects does it have on those excluded? And what are its consequences in epistemological and ethical terms