Crowdsourcing data through online marketplaces such as

Amazon Mechanical Turk poses new challenges about how consumer

research should be designed, conducted and analysed.

Additionally, it raises questions about the validity of the participants

and the information they provide. As protocols for crowdsourcing

data are still being worked out, we have developed a

few guidelines that will benefit those using such platforms for

research purposes

Paolacci, G. (Gabriele)

Paolacci, Gabriele

EUR Research Repository

ROTTERDAM SCHOOL OF MANAGEMENT, ERASMUS UNIVERSITY The effects of guanxi on employee voiceChris Murray talks with Tina Davidson How big should a young firm think?By Giuseppe Criaco Crowdsourced  consumer data:  how do we make  sure it’s good?By Gabriele Paolacci  Innovation is a  team sportBy Dirk Deichmann The whys and whens of intra-team power strugglesBy Lisanne van Bunderen2nd Quarter 2018Management Knowledge         RSM DISCOVERYwww.rsm.nl/discovery RSM - a force for positive change10   |  2nd Quarter 2018Management KnowledgeRSM DISCOVERYsourced data is no exception. Some fear that the people filling out the ma-terials might not be giving them their full attention. Others worry that peo-ple who taking many surveys may no longer be naïve respondents, possibly compromising the results. Finally, how do you know the person filling out the survey is who he says he is? After all, as the old New Yorker cartoon put it, on the internet, nobody knows you’re a dog. Getting it rightTo address these worries, my col-league Joe Goodman of the Ohio State University and I undertook a review of the evidence underlying them, and came up with some guidelines for sur-vey and experimental researchers to harness the benefits of online pools and avoid their drawbacks.  Some of these problems are difficult to prevent, but you can steer away from most by adopting a few strategies. You’ll find more if you read our paper, but:• Avoid asking for a specific quality un-less it’s a pre-sortable category, such as geography. Without knowing in advance what you want, respond-ents won’t be tempted to lie about themselves simply to get the job.• Require participants to formally en-rol before you show them the study. Requiring enrolment prevents pre-viewing the study (which can com-promise its validity) and raises the time costs required by participants. In addition, increasing the effort demanded before the survey de-lected from MTurk workers, making it perhaps the most represented pool of participants in the history of my discipline. In the Journal of Consumer Research, one of the major journals in my area of specialism, 43 per cent of behavioural studies in the June 2015-April 2016 volumes were con-ducted using MTurk. MTurk and analogous platforms such as Prolific have enabled my col-leagues and me to collect samples much more quickly and more cheaply than through traditional alternatives (e.g, university participant pools), and those of us who looked into the qual-ity of the resulting data found it to be comparable to such alternatives.But every powerful new tool cre-ates a new set of risks, and crowd-When Amazon launched Mechanical Turk (MTurk) in 2005, executives touted it as a way to augment artificial intelli-gence with the old-fashioned human va-riety. Organisations would post details about a small task that needed to be completed, such as writing product de-scriptions or identifying performers on music CDs, and then people searching the MTurk site would browse the jobs available and start working on those they were qualified for and thought sounded interesting or lucrative.But as is often the case with an in-novation, one of MTurk’s most popu-lar applications seems to have caught Amazon by surprise: social science research. In consumer behaviour re-search alone, over 15,000 studies have been published based on evidence col-Crowdsourcing data through online marketplaces such as Amazon Mechanical Turk poses new challenges about how con-sumer research should be designed, conducted and analysed. Additionally, it raises questions about the validity of the partici-pants and the information they provide. As protocols for crowd-sourcing data are still being worked out, we have developed a few guidelines that will benefit those using such platforms for research purposes. Crowdsourced consumer data:how do we make sure it’s good?By Gabriele Paolacci“Crowdsourcing websites like MTurk  make survey and experimental  investigations more efficient.”2nd Quarter 2018   |   11www.rsm.nl/discovery Crowdsourcing websites like MTurk make survey and experimental investi-gations more efficient. When used con-scientiously, crowdsourcing can also help improve consumer science by ena-bling more numerous and informative studies and increasing participant and researcher diversity. However, online research and crowdsourcing in par-ticular have their own set of risks, and researchers need to design studies in ways that mitigate them. The paper, Crowdsourcing consumer re-search, written by Joseph K. Goodman and Gabriele Paolacci, is published in the Journal of Consumer Research, Volume 44, Issue 1, 1 June 2017, p196–210. https://doi.org/10.1093/jcr/ucx047Gabriele Paolacci is Associate Professor of Marketing, Department of Marketing Management, Rotterdam School of Management.  EMAIL  gpaolacci@rsm.nlcreases the attractiveness of quit-ting halfway through.• If you are worried about “profes-sional participants”, consider us-ing a third-party crowdsourcing support service such as Turk Prime that can help you recruit people with somewhat less experience tak-ing psychological surveys.• Pay a reasonable rate. Though qual-ity seems to be relatively independ-ent of pay rate, you may compro-mise your individual reputation and the future attractiveness of partici-pating in your studies and in MTurk in general. (Besides, there are ob-vious ethical reasons not to exploit those who work for you, right?) • For similar reasons, resist block-ing any MTurk worker. This can get them knocked off the site (also hurt-ing your own reputation in the end).Other issuesOther issues may take more time to work out. Crowdsourced data has been around for roughly a decade now but protocols for its use are still being worked out. Our work has suggested some guidelines for handling it better, but there are other issues that remain. For instance, we need time to know the distinctive qualities of the crowd behind the data. For example, one survey has found that American MTurk workers tend to score higher on reporting if they needed to think about an answer (need for cognition) and higher on civics questions. They tend to be younger and better edu-cated than the general run of people. They are also unusual in that they are slightly more introverted, show great-er levels of social anxiety, and have slightly lower self-esteem than the general population. This should serve to remind us that absent more sophis-ticated recruitment tools, we should al-ways treat crowdsourced samples as non-representative.Also, the technology itself still has plenty of room for improvement. For example, though third parties can help, these sites don’t have a good way yet to handle interaction between partici-pants. Similarly, easier tools to share projects and data across research-ers would be helpful. By enabling re-searchers to collect larger samples, crowdsourcing is already contributing to making consumer research better, but more can be done to facilitate more open collaboration between scientists.Crowdsourced consumer data:how do we make sure it’s good?By Gabriele Paolacci

English

Crowdsourced consumer data: how do we make sure it's good?

Erasmus University Digital Repository

https://repub.eur.nl/pub/108776/p10-11-GabrielePaolacci.pdf

Crowdsourced consumer data: how do we make sure it's good?

Abstract

Similar works

Full text

Available Versions

EUR Research Repository

Erasmus University Digital Repository

EUR Research Repository