Correcting Misconceptions About Gamification of Assessment: More Than SJTs and Badges

Describing the current state of gamification, Chamorro-Premuzic, Winsborough, Sherman, and Hogan () provide a troubling contradiction: They offer examples of a broad spectrum of gamification interventions, but they then summarize the entirety of gamification as “the digital equivalent of situational judgment tests.” This mischaracterization grossly oversimplifies a rapidly growing area of research and practice both within and outside of industrial–organizational (I-O) psychology. We agree that situational judgment tests (SJTs) can be considered a type of gamified assessment, and gamification provides a toolkit to make SJTs even more gameful. However, the term gamification refers to a much broader and potentially more impactful set of tools than just SJTs, which are incremental, versatile, and especially valuable to practitioners in an era moving toward business-to-consumer (B2C) assessment models. In this commentary, we contend that gamification is commonly misunderstood and misapplied by I-O psychologists, and our goals are to remedy such misconceptions and to provide a research agenda designed to improve both the science and the practice surrounding gamification of human resource processes

Pay for Performance, Satisfaction and Retention in Longitudinal Crowdsourced Research

In the social and cognitive sciences, crowdsourcing provides up to half of all research participants. Despite this popularity, researchers typically do not conceptualize participants accurately, as gig-economy worker-participants. Applying theories of employee motivation and the psychological contract between employees and employers, we hypothesized that pay and pay raises would drive worker-participant satisfaction, performance, and retention in a longitudinal study. In an experiment hiring 359 Amazon Mechanical Turk Workers, we found that initial pay, relative increase of pay over time, and overall pay did not have substantial influence on subsequent performance. However, pay significantly predicted participants\u27 perceived choice, justice perceptions, and attrition. Given this, we conclude that worker-participants are particularly vulnerable to exploitation, having relatively low power to negotiate pay. Results of this study suggest that researchers wishing to crowdsource research participants using MTurk might not face practical dangers such as decreased performance as a result of lower pay, but they must recognize an ethical obligation to treat Workers fairly

HIT Me Baby, One More Time: Optimal Worker Incentive Strategies for Human Intelligence Tasks on Amazon Mechanical Turk

The use of crowdsourced human intelligence tasks (HITs) is increasing in research and business, with Amazon Mechanical Turk (MTurk) being one of the most popular platforms connecting workers and requestors. Despite this popularity, data on optimal strategies to minimize attrition and maximize data quality is lacking. This study assesses the effects of different payment strategies on worker participation, speed, effort, and retention in HITs. MTurk workers were recruited to participate in a multi-wave study on personality, values, and behavior. Multiple payment strategies were developed with total payment for both phases ranging from $1.00 to$10.00 across experimental conditions. Results and implications will be discussed

Correcting Misconceptions About Gamification of Assessment: More Than SJTs and Badges

Describing the current state of gamification, Chamorro-Premuzic, Winsborough, Sherman, and Hogan () provide a troubling contradiction: They offer examples of a broad spectrum of gamification interventions, but they then summarize the entirety of gamification as “the digital equivalent of situational judgment tests.” This mischaracterization grossly oversimplifies a rapidly growing area of research and practice both within and outside of industrial–organizational (I-O) psychology. We agree that situational judgment tests (SJTs) can be considered a type of gamified assessment, and gamification provides a toolkit to make SJTs even more gameful. However, the term gamification refers to a much broader and potentially more impactful set of tools than just SJTs, which are incremental, versatile, and especially valuable to practitioners in an era moving toward business-to-consumer (B2C) assessment models. In this commentary, we contend that gamification is commonly misunderstood and misapplied by I-O psychologists, and our goals are to remedy such misconceptions and to provide a research agenda designed to improve both the science and the practice surrounding gamification of human resource processes

Theory-driven Game-based Assessment of General Cognitive Ability: Design Theory, Measurement, Prediction of Performance, and Test Fairness

Games, which can be defined as an externally structured, goal-directed type of play, are increasingly being used in high-stakes testing contexts to measure targeted constructs for use in the selection and promotion of employees. Despite this increasing popularity, little is known about how theory-driven game-based assessments (GBA), those designed to reflect a targeted construct, should be designed, or their potential for achieving their simultaneous goals of positive reactions and high-quality psychometric measurement. In the present research, we develop a theory of GBA design by integrating game design and development theory from human-computer interaction with psychometric theory. Next, we test measurement characteristics, prediction of performance, fairness, and reactions of a GBA designed according to this theory to measure latent general intelligence (g). Using an academic sample with GPA data (N=633), we demonstrate convergence between latent GBA performance and g ( = .97). Adding an organizational sample with supervisory ratings of job performance (N=49), we show GBA prediction of both GPA (r=.16) and supervisory ratings (r=.29). We also show incremental prediction of GPA using unit-weighted composites of the g test battery beyond that of the g-GBA battery but not the reverse. We also show the presence of similar adverse impact for both the traditional test battery and GBA but the absence of differential prediction of criteria. Reactions were more positive across all measures for the g-GBA compared to the traditional test battery. Overall, results support GBA design theory as a promising foundation from which to build high quality theory-driven GBAs

Defining Gameful Experience as a Psychological State Caused by Gameplay: Replacing the Term 'Gamefulness' with Three Distinct Constructs

The final publication is available at Elsevier via https://doi.org/10.1016/j.ijhcs.2018.08.003. © 2019. This manuscript version is made available under the CC-BY-NC-ND 4.0 license http://creativecommons.org/licenses/by-nc-nd/4.0/Background and Aim Gamefulness is commonly cited as the primary goal of gamification, a family of approaches employed in education, business, healthcare, government, and elsewhere. However, gamefulness is defined imprecisely across the literature. To address this, we present a theory of gamefulness that splits gamefulness into more specific constructs and outlines their effects in a process model. Method We integrate extant literature from psychology, human-computer interaction, and other fields to define gameful design, systems, and experiences. Most critically, we argue that gameful experience is the core focal construct of this theory and define it as an interactive state occurring when a person perceives non-trivial achievable goals created externally, is motivated to pursue them under an arbitrary set of behavioral rules, and evaluates that motivation as voluntary. Results We present six resulting propositions: (1) gameful systems lead to gameful experiences, (2) gameful systems impact psychological characteristics, (3) effective gameful design leads to gameful systems, (4) effective gameful systems lead to behavioral change, (5) appropriate behavioral change causes the distal outcomes gamification designers target, and (6) individual differences moderate the effectiveness of gameful systems. Conclusion Gameful experience theory provides researchers with a unified foundation to study gamification from any social scientific lens