The hidden cost of using Amazon Mechanical Turk for research

In this study, we investigate the attentiveness exhibited by participants sourced through Amazon Mechanical Turk (MTurk), thereby discovering a significant level of inattentiveness amongst the platform’s top crowd workers (those classified as ‘Master’, with an ‘Approval Rate’ of 98% or more, and a ‘Number of HITS approved’ value of 1,000 or more). A total of 564 individuals from the United States participated in our experiment. They were asked to read a vignette outlining one of four hypothetical technology products and then complete a related survey. Three forms of attention check (logic, honesty, and time) were used to assess attentiveness. Through this experiment we determined that a total of 126 (22.3%) participants failed at least one of the three forms of attention check, with most (94) failing the honesty check – followed by the logic check (31), and the time check (27). Thus, we established that significant levels of inattentiveness exist even among the most elite MTurk workers. The study concludes by reaffirming the need for multiple forms of carefully crafted attention checks, irrespective of whether participant quality is presumed to be high according to MTurk criteria such as ‘Master’, ‘Approval Rate’, and ‘Number of HITS approved’. Furthermore, we propose that researchers adjust their proposals to account for the effort and costs required to address participant inattentiveness

Analyzing the capabilities of crowdsourcing services for text summarization

This paper presents a detailed analysis of the use of crowdsourcing services for the Text Summarization task in the context of the tourist domain. In particular, our aim is to retrieve relevant information about a place or an object pictured in an image in order to provide a short summary which will be of great help for a tourist. For tackling this task, we proposed a broad set of experiments using crowdsourcing services that could be useful as a reference for others who want to rely also on crowdsourcing. From the analysis carried out through our experimental setup and the results obtained, we can conclude that although crowdsourcing services were not good to simply gather gold-standard summaries (i.e., from the results obtained for experiments 1, 2 and 4), the encouraging results obtained in the third and sixth experiments motivate us to strongly believe that they can be successfully employed for finding some patterns of behaviour humans have when generating summaries, and for validating and checking other tasks. Furthermore, this analysis serves as a guideline for the types of experiments that might or might not work when using crowdsourcing in the context of text summarization.This work was supported by the EU-funded TRIPOD project (IST-FP6-045335) and by the Spanish Government through the FPU program and the projects TIN2009-14659-C03-01, TSI 020312-2009-44, and TIN2009-13391-C04-01; and by Conselleria d’Educació–Generalitat Valenciana (grant no. PROMETEO/2009/119 and grant no. ACOMP/2010/286)

Toward the Optimized Crowdsourcing Strategy for OCR Post-Correction

Digitization of historical documents is a challenging task in many digital humanities projects. A popular approach for digitization is to scan the documents into images, and then convert images into text using Optical Character Recognition (OCR) algorithms. However, the outcome of OCR processing of historical documents is usually inaccurate and requires post-processing error correction. This study investigates how crowdsourcing can be utilized to correct OCR errors in historical text collections, and which crowdsourcing methodology is the most effective in different scenarios and for various research objectives. A series of experiments with different micro-task's structures and text lengths was conducted with 753 workers on the Amazon's Mechanical Turk platform. The workers had to fix OCR errors in a selected historical text. To analyze the results, new accuracy and efficiency measures have been devised. The analysis suggests that in terms of accuracy, the optimal text length is medium (paragraph-size) and the optimal structure of the experiment is two-phase with a scanned image. In terms of efficiency, the best results were obtained when using longer text in the single-stage structure with no image. The study provides practical recommendations to researchers on how to build the optimal crowdsourcing task for OCR post-correction. The developed methodology can also be utilized to create golden standard historical texts for automatic OCR post-correction. This is the first attempt to systematically investigate the influence of various factors on crowdsourcing-based OCR post-correction and propose an optimal strategy for this process.Comment: 25 pages, 12 figures, 1 tabl

“Unsung” business strategy: Exploring crowd sourcing practices among Malaysian SMEs

The emerging of Internet of Things (IoTs) and changing global environment lead the demand for every organization to change and sustain their business. Due to this forces, the organization has no choice and forced to adapt with new Information and Communication Technology (ICT) or Technology Web 2.0 such as social media in order to strengthen their business strategy and for the business survival. Based on previous literature reviews, numerous strategies adopted by the organization as the mechanism to improve and sustain in their business performance. Amongst the business strategies are Lean Manufacturing (AlManei, Salonitis & Zu, 2017); Total Quality Management (TQM) (Sahoo & Yadav, 2018); Cloud Computing (Hassan, 2017); New Product Development (Abu, Mansor & Nasir, 2017); Human Capital (Muda & Che Abdul Rahman, 2016); Kaizen (Maarof & Mahmud, 2016) and many more. Nevertheless, the recent “unsung” business strategy or a new business strategy not being realize among the business owners is known as crowd sourcing practices. The phenomenon of crowd sourcing describes a new form of outsourcing tasks or value creation activities and function by tapping from external parties such as an individual, group, freelancer or any undefined group of people or crowds that are possessed heterogeneous of backgrounds, skill and knowledge, intelligence ideas to perform the organization tasks (Durward & Blohm, 2017). Crowd sourcing practices was introduced by Jeff Howe (2006) in the Wired Magazine articles. It refers to new phenomenon and as open business model in order to obtain diverse web-based solution from unknown individual or group of people to earn money by completing the projects or tasks (Howe, 2006; Liu et al., 2016)

Control de calidad en sistemas crowdsourcing: un mapeo sistemático

The web 2.0 have been easing the implementation of aneffective model denominated crowdsourcing to find solutions forgiven situations. This model pretend to take the advantages,capacities and proficiency of a group of people (from few ones tothousands of peoples) to solve some tasks. The crowdsourcing isnot a new concept, but thanks to information and communicationstechnologies (ICT) it is evolving in the same way as the new ICThave been develop and applied. Currently, the crowdsourcing isusing in data acquiring process and information generation,establishing a new perspective in the knowledge managementfield. The present work shows the results of a systematic mappingdid to literature review, with the purpose to recognize or learnabout several proposed mechanisms to ensure quality resultswithin crowdsourcing systems and research opportunities aroundthis topic.En el ámbito de la Web 2.0 se ha facilitado la implementación de un modelo efectivo en la solución de problemas denominado crowdsourcing que pretende aprovechar las capacidades y competencias de un grupo grande de individuos para resolver una determinada tarea.  El crowdsourcing no es un concepto nuevo, pero gracias a las nuevas tecnologías de la información y la comunicación está evolucionando a medida que se emplea en nuevas investigaciones y aplicaciones. Actualmente, el crowdsourcing es utilizado, entre otros, en procesos de adquisición y tratamiento de información estableciendo un nuevo enfoque en el campo de la gestión del conocimiento. Este artículo presenta los resultados de un mapeo sistemático realizado a la literatura con el propósito de conocer los diferentes mecanismos propuestos para asegurar la calidad de los resultados en los sistemas crowdsourcing y las oportunidades de investigación se encuentran en torno a este tema.

Control de calidad en sistemas crowdsourcing: un mapeo sistemático

The web 2.0 have been easing the implementation of aneffective model denominated crowdsourcing to find solutions forgiven situations. This model pretend to take the advantages,capacities and proficiency of a group of people (from few ones tothousands of peoples) to solve some tasks. The crowdsourcing isnot a new concept, but thanks to information and communicationstechnologies (ICT) it is evolving in the same way as the new ICThave been develop and applied. Currently, the crowdsourcing isusing in data acquiring process and information generation,establishing a new perspective in the knowledge managementfield. The present work shows the results of a systematic mappingdid to literature review, with the purpose to recognize or learnabout several proposed mechanisms to ensure quality resultswithin crowdsourcing systems and research opportunities aroundthis topic.En el ámbito de la Web 2.0 se ha facilitado la implementación de un modelo efectivo en la solución de problemas denominado crowdsourcing que pretende aprovechar las capacidades y competencias de un grupo grande de individuos para resolver una determinada tarea.  El crowdsourcing no es un concepto nuevo, pero gracias a las nuevas tecnologías de la información y la comunicación está evolucionando a medida que se emplea en nuevas investigaciones y aplicaciones. Actualmente, el crowdsourcing es utilizado, entre otros, en procesos de adquisición y tratamiento de información estableciendo un nuevo enfoque en el campo de la gestión del conocimiento. Este artículo presenta los resultados de un mapeo sistemático realizado a la literatura con el propósito de conocer los diferentes mecanismos propuestos para asegurar la calidad de los resultados en los sistemas crowdsourcing y las oportunidades de investigación se encuentran en torno a este tema.

The Use of Online Panel Data in Management Research: A Review and Recommendations

Management scholars have long depended on convenience samples to conduct research involving human participants. However, the past decade has seen an emergence of a new convenience sample: online panels and online panel participants. The data these participants provide—online panel data (OPD)—has been embraced by many management scholars owing to the numerous benefits it provides over “traditional” convenience samples. Despite those advantages, OPD has not been warmly received by all. Currently, there is a divide in the field over the appropriateness of OPD in management scholarship. Our review takes aim at the divide with the goal of providing a common understanding of OPD and its utility and providing recommendations regarding when and how to use OPD and how and where to publish it. To accomplish these goals, we inventoried and reviewed OPD use across 13 management journals spanning 2006 to 2017. Our search resulted in 804 OPD-based studies across 439 articles. Notably, our search also identified 26 online panel platforms (“brokers”) used to connect researchers with online panel participants. Importantly, we offer specific guidance to authors, reviewers, and editors, having implications for both micro and macro management scholars

Insufficient Effort Responding in Experimental Philosophy

Providing valid responses to a self-report survey requires cognitive effort. Subjects engaging in insufficient effort responding (IER) are unwilling to take this effort. Compared to psychologists, experimental philosophers so far seem to have paid less attention to IER. This paper is an attempt to begin to alleviate this shortcoming. First, I explain IER’s nature, prevalence and negative effects in self-report surveys in general. Second, I argue that IER might also affect experimental philosophy studies. Third, I develop recommendations as to how experimental philosophers should (and should not) try to prevent IER. Fourth, I develop recommendations as to how experimental philosophers should (and should not) try to detect IER. Fifth, I sketch how experimental philosophers ought to proceed once a subject has been identified as an insufficient effort responder. And finally, I report the results of an online survey that addresses experimental philosophers’ current knowledge, consideration and assessment of IER

Are people more averse to microbe-sharing contact with ethnic outgroup members? A registered report

Intergroup biases are widespread across cultures and time. The current study tests an existing hypothesis that has been proposed to explain such biases: the mind has evolved to interpret outgroup membership as a cue to pathogen threat. In this registered report, we test a core feature of this hypothesis. Adapting methods from earlier work, we examine (1) whether people are less comfortable with microbe-sharing contact with an ethnic outgroup member than an ethnic ingroup member, and (2) whether this difference is exacerbated by additional visual cues to a target's infectiousness. Using Chinese (N = 1533) and British (N = 1371) samples recruited from the online platforms WJX and Prolific, we assessed contact comfort with targets who were either East Asian or White and who were either modified to have symptoms of infection or unmodified (or, for exploratory purposes, modified to wear facemasks). Contact comfort was lower for targets modified to have symptoms of infection. However, we detected no differences in contact comfort with ethnic-ingroup targets versus ethnic-outgroup targets. These results do not support the hypothesis that people interpret ethnic outgroup membership alone as a cue to infection risk

Making health social: Effects of health PSA videos on social media

In the digital age, social networking sites are an essential tool for health educators to promote and spread awareness about health issues that may be plaguing society. Internet communications, such as social media, offer tremendous opportunities for modifying health because it allows people of all demographics to access health information. The present study examined the effectiveness of health public service announcement (PSAs) videos on social media and examined how the heuristics of social media “likes” may play a role in how social media users perceive these health messages. To assess how health messages on social media are perceived, this thesis used an experimental design (N = 272). This study could not find statistical support for the hypotheses that subjects were likely to show more favorable attitudes, self-efficacious behaviors, perceived norms, or intentions when exposed to health messages that displayed a high amount of social media “likes.” Results indicated, however, that attitudes (F(1, 270) = 162.38, p \u3c .001) perceptions of self-efficacy (F(1, 270) = 347.69, p \u3c .001), and perceived social norms (F(1,270) = 139.25, p \u3c .001) were related to intentions to exercise, as predicted by the Theory of Planned Behavior. The results of this study indicate that health messages in the context of social media are extremely complex, and deserve future attention in literature