Tahaei, Mohammad

Vaniea, Kami E

English

Edinburgh Research Explorer

     Edinburgh Research Explorer                                      Lessons Learned from Recruiting Participants with ProgrammingSkills for Empirical Privacy and Security StudiesCitation for published version:Tahaei, M & Vaniea, KE 2022, 'Lessons Learned from Recruiting Participants with Programming Skills forEmpirical Privacy and Security Studies', Paper presented at 1st International Workshop on RecruitingParticipants for Empirical Software Engineering, 17/05/22 - 17/05/22.Link:Link to publication record in Edinburgh Research ExplorerDocument Version:Peer reviewed versionGeneral rightsCopyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)and / or other copyright owners and it is a condition of accessing these publications that users recognise andabide by the legal requirements associated with these rights.Take down policyThe University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorercontent complies with UK legislation. If you believe that the public display of this file breaches copyright pleasecontact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately andinvestigate your claim.Download date: 13. Jan. 2023Lessons Learned From Recruiting Participants WithProgramming Skills for Empirical Privacy and Security StudiesMohammad Tahaeimohammad.tahaei@bristol.ac.ukDepartment of Computer ScienceUniversity of BristolUnited KingdomKami Vanieakami.vaniea@ed.ac.ukSchool of InformaticsUniversity of EdinburghUnited KingdomABSTRACTIn the past decade the usable privacy and security research com-munity has extended its research to expert users such as softwaredevelopers who have been shown to face challenges when workingwith privacy and security technologies. However, it is often chal-lenging to run empirical studies with developers because they canbe hard to recruit for interviews or surveys. Researchers have suc-cessfully used social media, crowdsourcing platforms, and computerscience students for recruiting, but recruitment is still a significantpain point in many studies. This short paper reflects on our experi-ence recruiting developers, particularly for empirical privacy andsecurity research, for multiple studies from 2019 to 2022.CCS CONCEPTS• Security and privacy→ Software and application security;Human and societal aspects of security and privacy;Usabilityin security and privacy; •Human-centered computing→Hu-man computer interaction (HCI); HCI design and evaluationmethods; • Software and its engineering;KEYWORDSrecruitment, developers, crowdsourcing, usable privacy and secu-rity, empirical software engineeringACM Reference Format:Mohammad Tahaei and Kami Vaniea. 2022. Lessons Learned From Recruit-ing Participants With Programming Skills for Empirical Privacy and Se-curity Studies. In Proceedings of 1st International Workshop on RecruitingParticipants for Empirical Software Engineering, co-located with the 44th In-ternational Conference on Software Engineering (RoPES - ICSE 2022). ACM,New York, NY, USA, 3 pages.1 AUTHORS’ EXPERIENCESStarting in 2019, we have recruited participants with a backgroundin privacy, security, and programming for multiple studies (Table 1).The experience level of participants has ranged from novice toexperienced participants in large tech companies. Methodologiesused are also varied and include interviews, tasked-based surveys,and artifact analysis (e.g., developers’ posts on Stack Overflow).2 LESSONS LEARNED FROM RECRUITMENTTrade-Offs Between Researchers’ Time, Participants’ Prior Experi-ence, and Number of Participants. We find that social media such© 2022 Authors retain copyright. This paper is published under a Creative CommonsAttribution 4.0 International License.RoPES - ICSE 2022, May 21–29, 2022, Pittsburgh, PA, USAas Twitter and LinkedIn may help recruit experienced participants.However, it comes with a trade-off that it requires several monthsand extra effort to search for potential candidates, answer theirquestions, and results in only a few participants. In one study, wespent several months recruiting on multiple social media channels,which resulted in 12 experienced participants [2].Virtual vs. In-Person Studies. While virtual studies allowed usto recruit participants worldwide, they also created a barrier forrunning lab studies and observing participants. Virtual meetingsare easy to cancel too. For example, several participants canceledtheir appointments [2], but we did not have such experiences within-person meetings [3], encouraging them to commit to showingup. Therefore, we suggest virtual studies consider recruiting moreparticipants in the case of a high no-show rate.Computer Science Students as a Go-To Solution. Computer sciencestudents were easy to recruit. In an interview study intended tounderstand students’ security mindsets, we recruited 20 studentsin less than a month. Scheduling the interviews was also easierbecause everyone was located on the same campus, making it easyto find and attend the session [3].Consider Using Crowdsourcing Platforms. Crowdsourcing plat-forms can provide a large pool of potential participants. In a studydesigned to understand opportunities for recruiting participantswith programming skills [5], we found that Appen is not designedfor recruiting this type of participant; Clickworker may be helpful,but its interfaces are not designed for surveys and recruitment;and MTurk provided few workers with actual programming skills.Prolific is the only platform that might result in a decent numberof participants who pass the programming questions. We also findthat computer science students are a reasonable group for findingparticipants with programming skills.Make Use of Publicly Accessible Forums’ Data. Aquasi-recruitmentmethod for understanding developers is to analyze their artifacts.We have done two studies with Stack Overflow’s data, and thismethod allows for the study of a whole community and gives in-sights into developers’ practices on a large scale. We encouragemore studies with developer forums, potentially across multipleforums, and combine those studies with interviews and surveys toproduce a strong theory and study design plan before recruitingdevelopers, which is more time-consuming.Check Terms and Policies of Platforms. While we have recruitedparticipants by harvesting emails from GitHub, we believe that themethod is not sustainable long term. It is not encouraged by theplatform’s terms of service and privacy policy to harvest emails forRoPES - ICSE 2022, May 21–29, 2022, Pittsburgh, PA, USA Tahaei and VanieaTable 1: A summary of our studies from 2019 to 2022.Publication Type of participants Recruitment channel To study . . . Method N Year[3] Computer science students Local University mailing list Security mindsets of participants Interview, qual. 20 2019[7] A range of developers Stack Overflow Developers privacy challenges and how they conceptualize privacy Artifact, qual. & quant. 315 2020[6] A range of developers Prolific and GitHub emails Usefulness of security notifications in static analysis tools Survey, qual. & quant. 132 2020[2] Experts in privacy Social media, word of mouth How experts champion for privacy in software teams Interview, qual. 12 2020[1] A range of developers Prolific, GitHub emails, and word of mouth Impact of interfaces and choices on developers’ privacy decisions Survey, qual. & quant. 400 2021[4] A range of developers Stack Overflow Developers’ privacy-related advice Artifact, qual. & quant. 119 2022[5]Participants who claimedto have programming skillsFour crowdsourcing platformsand a CS mailing listHow recruitment channels compare forrecruiting participants with programming skillsSurvey, quant. 613 2022research purposes (unfortunately, we did not know this at the timeof running our prior project [6]). Therefore, we do not suggest fu-ture researchers use developers’ emails from software developmentplatforms because, on top of breaching terms of services, it maytarnish researchers’ reputation among the developers’ community(particularly the open-source community). More broadly, we sug-gest researchers carefully read the terms and policies of platformsthey intend to use to ensure they will not violate the terms.3 EXPERIENCEWITH REVIEWERSNumber of Participants. Our primary publication venues havebeen CHI, SOUPS, and PETS. We received at least one reviewercomment about the number of participants in all of our submissions,suggesting that the community might not have a standard for therequired number of participants for an interview or a survey study.While concepts like saturation for qualitative studies and power forquantitative studies exist, we believe that the community still lacksa review of standard practices and suggestions for best practices.Future research on the published papers with a specific focus onrecruitment and the number of participants may provide insightsinto the standard and best practices of the community.Participants’ Levels of Experience. In all reviews, we also receivedcomments about participants’ experience. When conducting sur-veys or other remote research, we can rarely verify a participant’sactual work status; and therefore rely on self-reported professionsand years of experience in software development, among otherdemographic questions. We made the first effort to understand thedifferences between five recruitment channels to provide ground-ing for future research [5]. For example, we find that while partici-pants may think they have programming skills, in particular fromcrowdsourcing platforms, they can still fail simple programming-related questions such as the appropriate value of a Boolean vari-able (True/False). Therefore, we strongly suggest future researchersuse a screening survey on crowdsourcing platforms to ensure therequired programming understanding of all participants.Vague Meaning of Programming Experience. We note that theproblem may not be participants misrepresenting their experi-ence level. Instead, it may be that questions like “do you haveprogramming skills?” (used by Prolific) or similar generic questionsmight have a range of participant interpretations. Participants whohave only had one generic programming course or watched on-line videos may consider themselves as having programming skills.Researchers need to be aware of the type of developer experiencethey expect, and for work where participants need to do more com-plex tasks like test a library, a clear definition of “programming”is needed. We suggest designing a screening questionnaire ratherthan relying on general questions or self-reported demographics.4 WHAT DOES THE COMMUNITY NEED?Standardized Screening Questions. We believe a standard set ofquestions that can help researchers understand the level of expe-rience developers have is needed to push forward the researchin empirical software engineering. Such a set of questions mayhave programming, demographics, and other related topics thatresearchers can use to find a suitable sample for their studies. Wedo not think that all studies require professional, experienced de-velopers. Some studies that test the usability of a prototype andcompare it with prior technologies may well be tested with othergroups of participants who may not be experts but can provideinsights into the design process.Collaboration Between Academia and Industry. A potentially sus-tainable approach to moving research forward is having softwarecompanies work alongside the research community in empiricalsoftware engineering. While a large technology company fundedone author’s Ph.D., they did not get support from the company tolocate and recruit developers for studies. Perhaps explicit supportfrom these companies in the contract and having a dedicated con-tact point for assisting Ph.D. students in recruiting developers mayfacilitate such studies. We believe that two of the primary benefi-ciaries of research from empirical software engineering are largesoftware companies and the open-source community. Therefore,until they commit themselves to help researchers find participantsand test new technologies with them, the research community maycontinue to find it challenging to move the field forward.ACKNOWLEDGMENTSWe thank all of our collaborators in all publications that helped usgain these experiences.REFERENCES[1] Mohammad Tahaei, Alisa Frik, and Kami Vaniea. 2021. Deciding on PersonalizedAds: Nudging Developers About User Privacy. In SOUPS ’21. https://www.usenix.org/conference/soups2021/presentation/tahaei[2] Mohammad Tahaei, Alisa Frik, and Kami Vaniea. 2021. Privacy Champions inSoftware Teams: Understanding Their Motivations, Strategies, and Challenges. InCHI ’21. https://doi.org/10.1145/3411764.3445768[3] Mohammad Tahaei, Adam Jenkins, Kami Vaniea, and Maria K. Wolters. 2020. “IDon’t Know Too Much About It”: On the Security Mindsets of Computer ScienceLessons Learned From Recruiting Participants With Programming RoPES - ICSE 2022, May 21–29, 2022, Pittsburgh, PA, USAStudents. In Socio-Technical Aspects in Security and Trust. https://www.springer.com/book/9783030559571[4] Mohammad Tahaei, Tianshi Li, and Kami Vaniea. 2022. Understanding Privacy-Related Advice on Stack Overflow. In PETS ’22. https://doi.org/10.2478/popets-2022-0032[5] Mohammad Tahaei and Kami Vaniea. 2022. Recruiting Participants With Program-ming Skills: A Comparison of Four Crowdsourcing Platforms and a CS StudentMailing List. In CHI ’22. https://doi.org/10.1145/3491102.3501957[6] Mohammad Tahaei, Kami Vaniea, Beznosov Konstantin, and Maria K. Wolters.2021. Security Notifications in Static Analysis Tools: Developers’ Attitudes, Com-prehension, and Ability to Act on Them. In CHI ’21. https://doi.org/10.1145/3411764.3445616[7] Mohammad Tahaei, Kami Vaniea, and Naomi Saphra. 2020. Understanding Privacy-Related Questions on Stack Overflow. In CHI ’20. https://doi.org/10.1145/3313831.3376768

https://www.research.ed.ac.uk/files/319935965/tahaei2022ropes.pdf

Lessons Learned from Recruiting Participants with Programming Skills for Empirical Privacy and Security Studies

Abstract

Similar works

Full text

Available Versions

Edinburgh Research Explorer