Amazon Mechanical Turk (MTurk) is an online marketplace for labor recruitment that has become a popular platform for data collection. In particular, MTurk can be a valuable tool for Extension professionals. As an example, MTurk workers can provide feedback, write reviews, or give input on a website design. In this article we discuss the many uses of MTurk for Extension professionals and provide best practices for its use

Brar, Pooja

Dworkin, Jodi

Hessel, Heather

Rudi, Jessie

English

Clemson University: TigerPrints

Journal of Extension Volume 54 Number 6 Article 5 12-1-2016 MTurk 101: An Introduction to Amazon Mechanical Turk for Extension Professionals Jodi Dworkin University of Minnesota, jdworkin@umn.edu Pooja Brar University of Minnesota, brarx011@umn.edu Heather Hessel University of Minnesota, hhessel@umn.edu Jessie Rudi University of Minnesota, conne262@umn.edu Recommended Citation Dworkin, J., Brar, P., Hessel, H., & Rudi, J. (2016). MTurk 101: An Introduction to Amazon Mechanical Turk for Extension Professionals. Journal of Extension, 54(6), Article 5. https://tigerprints.clemson.edu/joe/vol54/iss6/5 This Tools of the Trade is brought to you for free and open access by TigerPrints. It has been accepted for inclusion in Journal of Extension by an authorized editor of TigerPrints. For more information, please contact kokeefe@clemson.edu. December 2016Volume 54Number 6Article # 6TOT5Tools of the TradeMTurk 101: An Introduction to Amazon Mechanical Turk forExtension ProfessionalsAbstractAmazon Mechanical Turk (MTurk) is an online marketplace for labor recruitment that has become a popular platformfor data collection. In particular, MTurk can be a valuable tool for Extension professionals. As an example, MTurkworkers can provide feedback, write reviews, or give input on a website design. In this article we discuss the manyuses of MTurk for Extension professionals and provide best practices for its use.   Internet-based surveys are a useful and low-cost method Extension professionals can use to recruit participantsfor research. Archer (2003) lists advantages and disadvantages of conducting surveys on the web andrecommends the use of this environment for Extension surveys. Amazon Mechanical Turk (MTurk)(www.mturk.com) is an online marketplace for labor recruitment that has become a popular platform forcollecting data online for social science research. Individuals or organizations looking for "workers" can register as"requesters" and post jobs or human intelligence tasks (HITs) for registered workers to complete. Common tasksinclude completing online surveys, categorizing photographs or other data, and providing feedback on websites ordocuments. Workers are paid upon successful task completion, with compensation ranging from pennies toseveral dollars. Research has shown MTurk to be cost effective for obtaining large, diverse samples forquestionnaire research (Buhrmester, Kwang, & Gosling, 2011; Dworkin, Hessel, Rudi, & Gliske, 2015).Studies have found that MTurk samples are more diverse in age, geographic location, and race/ethnicity thangeneral Internet samples (Buhrmester et al., 2011; Goodman, Cryder, & Cheema, 2013). Amazon.com (2016)reports employing more than 500,000 MTurk workers in more than 190 countries. Tasks can be completed in 13languages quickly and accurately (Pavlick, Post, Irvine, Kachaev, & Callison-Burch, 2014). Amazon typically paysinternational workers with gift cards but in 2009 began offering workers in India the option of being paid inIndian rupees, greatly increasing the percentage of Indian workers (Ipeirotis, 2010).Jodi DworkinAssociate ProfessorDepartment of FamilySocial Sciencejdworkin@umn.eduPooja BrarDoctoral StudentDepartment of FamilySocial Sciencebrarx011@umn.eduHeather HesselDoctoral StudentDepartment of FamilySocial Sciencehhessel@umn.eduJessie RudiResearch AssociateInstitute forTranslational Researchin Children's MentalHealth andDepartment of FamilySocial Scienceconne262@umn.eduUniversity ofMinnesotaSaint Paul, MinnesotaData QualitySome researchers are concerned about the quality of data from MTurk workers. However, a number ofresearchers have replicated studies to compare online and offline respondents and found that data quality wasnot significantly different between MTurk and offline and laboratory experiments (Horton, Rand, & Zeckhauser,2011; Paolacci, Chandler, & Ipeirotis, 2010; Suri & Watts, 2011). MTurk workers provided high-quality data thatmet or exceeded psychometric standards associated with research conducted in face-to-face settings(Buhrmester et al., 2011).Uses of MTurk by Extension ProfessionalsPossible applications of MTurk within Extension are wide ranging. Specifically, MTurk can be used for research inthe following ways:Researchers can collect data from national and international samples about individual behaviors andperceptions. A HIT can link workers to a website hosting an online survey (e.g., SurveyMonkey). MTurk alsoprovides its own online survey tool.The platform allows researchers to collect unique data in ways that previously could be done only in face-to-face settings, such as by simulating group interactions or synchronous tasks.Researchers can create virtual "waiting rooms" to hold workers while others accept a HIT and then proceedwith the experiment by having all workers act simultaneously (Mason & Suri, 2012).Researchers can conduct in-depth qualitative interviews with geographically diverse participants, homeboundparticipants, participants without college degrees, and minority groups (Williamson, 2014).Participants can be recruited online for in-person data collection.MTurk workers can provide a variety of services, includingwriting reviews, descriptions, and blog entries for websites;editing, transcribing, or rating the accuracy of search engine results;giving feedback about videos and photos, advertisements, recruitment flyers, and program descriptions;providing feedback about whether people or materials in photos are relatable and culturally appropriate; andtagging photos or videos with keywords that can be used by practitioners to recruit a particular audience.MTurk has many benefits over other methods of data collection:It provides access to a worldwide workforce of people who can perform tasks in multiple languages.The platform has workers who are available on-demand 24 hours a day, 7 days a week. The cost of developingan online recruitment space and recruiting participants is much lower than hiring staff or working with aTools of Trade MTurk 101: An Introduction to Amazon Mechanical Turk for Extension Professionals JOE 54(6)© 2016 Extension Journal Inc. 1consulting firm.It provides templates for designing tasks. Using the MTurk HTML editor, requesters can incorporate images,figures, and videos into tasks.It provides access to a more diverse sample than may be available in person or through other onlinerecruitment strategies.It can be used to obtain feedback from a particular group (e.g., parents of young children) about a website'sformat, design, or content. For example, workers could complete a usability task by exploring a website andanswering a short survey that provides feedback about their experience.Suggestions and RecommendationsBelow are some guidelines to consider when using MTurk. See Table 1 for suggested resources and the appendixfor step-by-step guidance for using MTurk.Prior to setting up a task, consider the type of participant you want to recruit (e.g., single parents, non-college-going youth).Become an MTurk worker yourself to become familiar with the worker experience, including with how to getpaid and how to peruse the various HITs that are available.Test a HIT in the MTurk Sandbox before publishing it to workers; it is a good idea to ensure that surveys workwell with a target population and are thoroughly pilot tested before recruiting begins (Monroe & Adams, 2012).The MTurk Sandbox simulates the MTurk environment without any payment involved; it is solely for testingHITs. You can enter your task into the sandbox (just as you would in the actual MTurk website) and chooseparameters for how many workers can pilot test the survey to check for errors and identify difficulties with thesurvey experience (Johnson & Borden, 2012).Take advantage of MTurk's tools for sample selection by specifying worker requirements and inclusion andexclusion criteria for who can see HITs. For example, you can choose to show HITs to only experienced MTurkworkers (those who have completed a certain number of HITs successfully), MTurk workers with highreputations (those with high completion ratings), or MTurk workers in particular countries. You can alsoexclude workers who have already completed the HIT, thereby avoiding duplicate responses.Incorporate attention checks into your survey. Attention checks are questions that are not related to yourstudy but allow you to determine whether respondents are paying attention (e.g., "Select 'Strongly Disagree'for this question").Table 1.Resources Supporting MTurk UseURL Descriptionhttps://requester.mturk.com/ Official requester site of MTurk—used to create an account and aTools of Trade MTurk 101: An Introduction to Amazon Mechanical Turk for Extension Professionals JOE 54(6)© 2016 Extension Journal Inc. 2HIThttps://blog.mturk.com/ Official MTurk bloghttp://MTurk.mit.edu/tutorial/ Site created in 2014 by MorrisAlper, an undergraduate studentat Massachusetts Institute ofTechnology, to show the step-by-step process for setting up a HITon MTurkhttps://www.youtube.com/watch?v=Eq_8ETBWkfk YouTube video that demonstratesthe process of uploading a surveyto MTurkhttp://www.turkernation.com/ Online forum for MTurk workers touse to discuss various aspects ofworking on MTurk; used by someresearchers as a forum for talkingabout their research andconnecting with workershttp://turkopticon.ucsd.edu/ Website created by workers toconnect with and support eachother by reviewing requestershttp://top.sagepub.com/content/39/4/245.short Article written in 2012 by Dan R.Johnson and Lauren A. Bordenthat includes a brief user's guidefor MTurk (getting started,designing a HIT/study, publishinga HIT/study, and managing aHIT/study)http://rsw.sagepub.com/content/26/4/441.abstract Article written in 2015 by ChitatChan and Michael J. Holosko thatprovides a brief overview of MTurkand discusses possible uses bysocial work professionalshttps://workersandbox.mturk.com/mturk/welcome Developer sandbox that allows forpractice in creating andcompleting HITsNote. MTurk = Amazon Mechanical Turk. HIT = human intelligence task.MTurk is useful to Extension professionals who want to learn more about the communities they work with. Byproviding access to individuals from diverse backgrounds who are able to provide quality data, MTurk supportsboth research (e.g., data collection) and outreach (e.g., receiving feedback on documents).Tools of Trade MTurk 101: An Introduction to Amazon Mechanical Turk for Extension Professionals JOE 54(6)© 2016 Extension Journal Inc. 3ReferencesAmazon.com. (2016). Service highlights. Retrieved from https://requester.mturk.com/tourArcher, T. M. (2003). Web-based surveys. Journal of Extension, 41(2) Article 4TOT6. Available at:http://www.joe.org/joe/2003august/tt6.phpBuhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's Mechanical Turk a new source of inexpensive, yethigh-quality, data? Perspectives on Psychological Science, 6(1), 3–5.Dworkin, J., Hessel, H., Rudi, J., & Gliske, K. (2015). A comparison of three online recruitment strategies forengaging parents. Manuscript submitted for publication.Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data collection in a flat world: The strengths andweaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26(3), 213–224.Horton, J. J., Rand, D. G., & Zeckhauser, R. J. (2011). The online laboratory: Conducting experiments in a reallabor market. Experimental Economics, 14(3), 399–425.Ipeirotis, P. G. (2010, March). Demographics of Mechanical Turk. NYU Working Paper No. CEDER-10-01.Retrieved from http://ssrn.com/abstract=1585030Johnson, D. R., & Borden, L. A. (2012). Participants at your fingertips: Using Amazon's Mechanical Turk toincrease study-faculty collaborative research. Teaching of Psychology, 39(4), 245–251.Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon's Mechanical Turk. Behavior ResearchMethods, 44(1), 1–23.Monroe, M. C., & Adams, D. C. (2012). Increasing response rates to web-based surveys. Journal of Extension,50(6) Article 6TOT7. Available at: http://www.joe.org/joe/2012december/tt7.phpPaolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgmentand Decision Making, 5(5), 411–419.Pavlick, E., Post, M., Irvine, A., Kachaev, D., & Callison-Burch, C. (2014). The language demographics of AmazonMechanical Turk. Transactions of the Association for Computational Linguistics, 2, 79–92.Suri, S., & Watts, D. J. (2011). Cooperation and contagion in web-based, networked public goods experiments.PLoS One, 6(3), e16836.Williamson, V. (2014, August 10). On the ethics of crowd-sourced research. Retrieved fromhttp://scholar.harvard.edu/files/williamson/files/mturk_ps_081014.pdfAppendixGuidance for Using MTurkA. How to create an account with Amazon MTurkCreate a requester account with Amazon MTurk by going tohttps://www.mturk.com/mturk/welcomeTools of Trade MTurk 101: An Introduction to Amazon Mechanical Turk for Extension Professionals JOE 54(6)© 2016 Extension Journal Inc. 4Add money to your Amazon Payments account to purchase prepaid HITsand to calculate the amount you want to pay per worker (plus the fee toAmazon MTurk).Keep in mind that Amazon charges a higher fee for more experiencedworkers (i.e., workers identified as Master workers within MTurk).B. How to create a high-quality HITCreate tasks that are simple and concise:It is recommended that you give clear and easy-to-read instructions.It is recommended that you break larger tasks into smaller parts.Provide an example of acceptable work, if possible.Either create a survey on your organization's or institution's platform (e.g.,Qualtrics or SurveyMonkey) and link it to MTurk for recruitment, or useMTurk's online survey tool.C. How to test the HIT in the sandboxLog on as a requester.Click the Developer tab and find the Sandbox.Enter the requester Sandbox to create practice HITs:First click Create.Then click Survey Link to connect your survey to the MTurk platform.Describe the HIT.Provide a concise description of the task.Provide an appropriate list of keywords to attract desired workers.Enter the following information about payment, number of tasks, andtiming:Payment amount.Number of completed tasks desired.Tools of Trade MTurk 101: An Introduction to Amazon Mechanical Turk for Extension Professionals JOE 54(6)© 2016 Extension Journal Inc. 5Amount of time workers have to complete tasks.Length of time you want your HIT to be advertised.General time line for approving the task and paying workers.Click Advanced Options to specify worker requirements.Click Customize Worker Requirements.You must consider whether you need Master workers.You can specify the location (country) of workers.You must consider what kind of task approval rating is acceptable.Show your HIT to workers who are qualified to complete it.Create the design layout.Click the Design Layout tab.Type instructions for the survey.Activate your survey by clicking Distribute Survey.Copy and paste the survey link on the MTurk platform.Provide a survey completion code for workers to enter in MTurk toconfirm survey completion.Preview your HIT.Click the Preview tab.Examine the HIT.Click the Finish tab.Test your HIT by using the worker account in the Sandbox.Have people complete the practice task.Get a sense of the time it takes to complete the HIT.Tools of Trade MTurk 101: An Introduction to Amazon Mechanical Turk for Extension Professionals JOE 54(6)© 2016 Extension Journal Inc. 6Make any necessary corrections or adjustments.D. Reasons to visit online worker communities (optional; see Table 1)Gain insight into workers by visiting online worker communities.Introduce yourself and your HIT to recruit motivated participants.E. Publish the HIT on MTurkPut money in the Prepaid HIT Balance account on Amazon.Create and publish HITs.Watch and monitor as HITs are completed.Approve payments within the specified time.Copyright © by Extension Journal, Inc. ISSN 1077-5315. Articles appearing in the Journal become the property ofthe Journal. Single copies of articles may be reproduced in electronic or print form for use in educational ortraining activities. Inclusion of articles in other publications, electronic sources, or systematic large-scaledistribution may be done only with prior electronic or written permission of the Journal Editorial Office, joe-ed@joe.org.If you have difficulties viewing or printing this page, please contact JOE Technical SupportTools of Trade MTurk 101: An Introduction to Amazon Mechanical Turk for Extension Professionals JOE 54(6)© 2016 Extension Journal Inc. 7

MTurk 101: An Introduction to Amazon Mechanical Turk for Extension Professionals

https://tigerprints.clemson.edu/cgi/viewcontent.cgi?article=2106&context=joe

MTurk 101: An Introduction to Amazon Mechanical Turk for Extension Professionals

Abstract

Similar works

Full text

Available Versions

Clemson University: TigerPrints