Who is in the sample? An analysis of real and surrogate users as participants in user study research in the information technology fields

Background Constructing a sample of real users as participants in user studies is considered by most researchers to be vital for the validity, usefulness, and applicability of research findings. However, how often user studies reported in information technology academic literature sample real users or surrogate users is unknown. Therefore, it is uncertain whether or not the use of surrogate users in place of real users is a widespread problem within user study practice. Objective To determine how often user studies reported in peer-reviewed information technology literature sample real users or surrogate users as participants. Method We analyzed 725 user studies reported in 628 peer-reviewed articles published from 2013 through 2021 in 233 unique conference and journal outlets, retrieved from the ACM Digital Library, IEEE Xplore, and Web of Science archives. To study the sample selection choices, we categorized each study as generic (i.e., users are from the general population) or targeted (i.e., users are from a specific subpopulation), and the sampled study participants as real users (i.e., from the study population) or surrogate users (i.e., other than real users). Results Our analysis of all 725 user studies shows that roughly two-thirds (75.4%) sampled real users. However, of the targeted studies, only around half (58.4%) sampled real users. Of the targeted studies sampling surrogate users, the majority (69.7%) used students, around one-in-four (23.6%) sampled through crowdsourcing, and the remaining 6.7% of studies used researchers or did not specify who the participants were. Conclusions Key findings are as follows: (a) the state of sampling real users in information technology research has substantial room for improvement for targeted studies; (b) researchers often do not explicitly characterize their study participants in adequate detail, which is probably the most disconcerting finding; and (c) suggestions are provided for recruiting real users, which may be challenging for researchers. Implications The results imply a need for standard guidelines for reporting the types of users sampled for a user study. We provide a template for reporting user study sampling with examples.© 2022 Salminen et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, reproduction and adaptation in any medium and for any purpose provided that it is properly attributed. For attribution, the original author(s), title, publication source (PeerJ Computer Science) and either DOI or URL of the article must be cited.fi=vertaisarvioitu|en=peerReviewed

Who is in the sample? An analysis of real and surrogate users as participants in user study research in the information technology fields

Background: Constructing a sample of real users as participants in user studies is considered by most researchers to be vital for the validity, usefulness, and applicability of research findings. However, how often user studies reported in information technology academic literature sample real users or surrogate users is unknown. Therefore, it is uncertain whether or not the use of surrogate users in place of real users is a widespread problem within user study practice.Objective: To determine how often user studies reported in peer-reviewed information technology literature sample real users or surrogate users as participants.Method: We analyzed 725 user studies reported in 628 peer-reviewed articles published from 2013 through 2021 in 233 unique conference and journal outlets, retrieved from the ACM Digital Library, IEEE Xplore, and Web of Science archives. To study the sample selection choices, we categorized each study as generic (i.e., users are from the general population) or targeted (i.e., users are from a specific subpopulation), and the sampled study participants as real users (i.e., from the study population) or surrogate users (i.e., other than real users).Results: Our analysis of all 725 user studies shows that roughly two-thirds (75.4%) sampled real users. However, of the targeted studies, only around half (58.4%) sampled real users. Of the targeted studies sampling surrogate users, the majority (69.7%) used students, around one-in-four (23.6%) sampled through crowdsourcing, and the remaining 6.7% of studies used researchers or did not specify who the participants were.Conclusions: Key findings are as follows: (a) the state of sampling real users in information technology research has substantial room for improvement for targeted studies; (b) researchers often do not explicitly characterize their study participants in adequate detail, which is probably the most disconcerting finding; and (c) suggestions are provided for recruiting real users, which may be challenging for researchers.Implications: The results imply a need for standard guidelines for reporting the types of users sampled for a user study. We provide a template for reporting user study sampling with examples.</p

Understanding and designing technologies for everyday financial collaboration

Perhaps enticed by the promise of reduced marginal costs per customer and other “operational efficiencies”, the financial industry seems to take for granted that introducing technology into their services delivers convenience and makes it easier for people to manage their money. The overwhelmingly positive discourse that surrounds financial technologies portrays them as the inevitable next step in the evolution of money, and as driving consumer empowerment by reducing costs and improving quality of service. Research, however, has linked those very same technologies to new and existing forms of financial exclusion. This raises the question of how we can design financial technologies that promote access and fairness. In this thesis, I take on this question by casting a critical lens over the design of financial technologies through experiences of financial difficulty and financial third party access. I conducted qualitative studies with a team inside the banking industry tasked with servicing customers deemed “vulnerable”; and with a group of people who live under the “double trouble” (Topor et al., 2016) of mental illness and financial difficulty. The latter trialled a new financial third party access digital service for 3 months. These varied perspectives on financial difficulty and third party access reveal the unintended consequences of introducing technology into our interactions with money, and the theories and assumptions concealed in the design of existing financial technologies. Based on the insights of these studies, and a synthesis of the literature on the nature of money, this thesis contributes alternative paradigms that may help us design financial technologies differently. Such technologies would reflect an understanding of money as a social relation, and of our finances as a collaborative endeavour. Rather than focusing on efficiency, resource optimisation and asset protection, they would encourage flexibility, complementarity, reflection, appropriation, positive forms of security, collaboration and participation. By designing financial technologies under different theoretical premises and with different priorities, we may promote access, fairness and democratic oversight in financial service provision, particularly for those experiencing financial difficulty