Disentangling Item and Testing Effects in Inoculation Research on Online Misinformation: Solomon Revisited

Online misinformation is a pervasive global problem. In response, psychologists have recently explored the theory of psychological inoculation: If people are preemptively exposed to a weakened version of a misinformation technique, they can build up cognitive resistance. This study addresses two unanswered methodological questions about a widely adopted online “fake news” inoculation game, Bad News. First, research in this area has often looked at pre- and post-intervention difference scores for the same items, which may imply that any observed effects are specific to the survey items themselves (item effects). Second, it is possible that using a pretest influences the outcome variable of interest, or that the pretest may interact with the intervention (testing effects). We investigate both item and testing effects in two online studies (total N = 2,159) using the Bad News game. For the item effect, we examine if inoculation effects are still observed when different items are used in the pre- and posttest. To examine the testing effect, we use a Solomon’s Three Group Design. We find that inoculation interventions are somewhat influenced by item effects, and not by testing effects. We show that inoculation interventions are effective at improving people’s ability to spot misinformation techniques and that the Bad News game does not make people more skeptical of real news. We discuss the larger relevance of these findings for evaluating real-world psychological interventions

Disentangling Item and Testing Effects in Inoculation Research on Online Misinformation: Solomon Revisited

Online misinformation is a pervasive global problem. In response, psychologists have recently explored the theory of psychological inoculation: If people are preemptively exposed to a weakened version of a misinformation technique, they can build up cognitive resistance. This study addresses two unanswered methodological questions about a widely adopted online “fake news” inoculation game, Bad News. First, research in this area has often looked at pre- and post-intervention difference scores for the same items, which may imply that any observed effects are specific to the survey items themselves (item effects). Second, it is possible that using a pretest influences the outcome variable of interest, or that the pretest may interact with the intervention (testing effects). We investigate both item and testing effects in two online studies (total N = 2,159) using the Bad News game. For the item effect, we examine if inoculation effects are still observed when different items are used in the pre- and posttest. To examine the testing effect, we use a Solomon’s Three Group Design. We find that inoculation interventions are somewhat influenced by item effects, and not by testing effects. We show that inoculation interventions are effective at improving people’s ability to spot misinformation techniques and that the Bad News game does not make people more skeptical of real news. We discuss the larger relevance of these findings for evaluating real-world psychological interventions

Misinformation interventions decay rapidly without an immediate posttest

In recent years, many kinds of interventions have been developed that seek to reduce susceptibility to misinformation. In two preregistered longitudinal studies (N1 = 503, N2 = 673), we leverage two previously validated “inoculation” interventions (a video and a game) to address two important questions in misinformation interventions research: (1) whether displaying additional stimuli (such as videos unrelated to misinformation) alongside an intervention interferes with its effectiveness, and (2) whether administering an immediate posttest (in the form of a social media post evaluation task after the intervention) plays a role in the longevity of the intervention. We find no evidence that other stimuli interfere with intervention efficacy, but strong evidence that immediate posttests strengthen the learnings from the intervention. In study 1, we find that 48 h after watching a video, participants who received an immediate posttest continued to be significantly better at discerning untrustworthy social media posts from neutral ones than the control group (d = 0.416, p = .007), whereas participants who only received a posttest 48 h later showed no differences with a control (d = 0.010, p = .854). In study 2, we observe highly similar results for a gamified intervention, and provide evidence for a causal mechanism: immediate posttests help strengthen people's memory of the lessons learned in the intervention. We argue that the active rehearsal and application of relevant information are therefore requirements for the longevity of learning‐based misinformation interventions, which has substantial implications for their scalability

The Long-Term Effectiveness of Inoculation Against Misinformation: An Integrated Theory of Memory, Threat, and Motivation

For over 60 years, inoculation theory has been a key framework to understand resistance to persuasion, yet many critical questions have remained unanswered. This dissertation aims to provide a theoretical and empirical understanding of how resistance to persuasion effects decay over time. In the context of resistance to persuasion by misinformation, I offer 10 empirical experiments that shed new light on this question, including several methodological innovations. In Chapter 2, I propose a new model that integrates memory theories with motivation theories on inoculation. In Chapters 3–6, I evaluate the long-term effectiveness of inoculation in message-based, gamified, and video-based inoculation interventions, unveiling the underlying mechanisms of decay. In Chapter 7, I address methodological issues, including the effects of repeated testing, and unstandardised items, and the development of a new misinformation susceptibility test. In summary, this thesis advances our understanding of the mechanisms of decay in resistance to persuasion, and sheds light on the role of and interplay between memory and motivation. The new memory-motivation model brings a significant advancement to the field, as it taps into the memory literature of forgetting—a domain in cognitive psychology—to shed new light on a concept in social psychology, and enables a new approach to modelling the longevity of inoculation effects. In addition, I offer novel insights into limitations with current methodological paradigms, and demonstrate how new standardised measurement tools can be developed to more accurately map inoculation effects in future research. Finally, I discuss how the findings of this dissertation can inform not only inoculation scholarship, but also intervention designers, evaluators, and policy makers, on how to address the problem of misinformation, and demonstrate how to extend the long-term effects of inoculation in applied interventions.United Kingdom Economic and Social Research Council (Cambridge Doctoral Training Partnership Award); Cambridge Commonwealth, European & International Trust (Cambridge European Scholarship

Misleading But Not Fake: Measuring the Difference Between Manipulativeness Discernment and Veracity Discernment Using Psychometrically Validated Tests

Misinformation continues to pose a substantial societal problem, but the measurement of misinformation susceptibility has often been done using non-validated tests. Furthermore, research shows that misleading content (implied misinformation) is much more common than outright false content (explicit misinformation). However, there is very little research on the predictors of belief in implied misinformation, and it is unknown if susceptibility to direct and implied misinformation are psychologically similar. To address these questions, we ran three preregistered studies (N 1 = 487, N 2 = 547, N 3 = 490) in which we developed and validated the 24-item and 12-item Manipulative Online Content Recognition Inventory (MOCRI), a test that measures a person’s ability to distinguish between misleading and neutral content. This test substantially outperforms other known predictors of misinformation susceptibility in terms of its predictive value for people’s ability to correctly identify many kinds of misleading content. We also show that susceptibility to misleading and false content are psychologically different from one another, although they are related. Finally, we show that people who score high on the MOCRI are much better than low MOCRI performers at discerning manipulative from non-manipulative statements (i.e., they have better “discernment”), but that this ability does not necessarily translate to better discernment in the quality of their sharing decisions, or their willingness to reply to manipulative vs. non-manipulative messages. Instead, people who are more resilient to being manipulated are less likely to share and respond to both manipulative and non-manipulative content

Long-term effectiveness of inoculation against misinformation: Three longitudinal experiments.

This study investigates the long-term effectiveness of active psychological inoculation as a means to build resistance against misinformation. Using 3 longitudinal experiments (2 preregistered), we tested the effectiveness of Bad News, a real-world intervention in which participants develop resistance against misinformation through exposure to weakened doses of misinformation techniques. In 3 experiments (NExp1 = 151, NExp2 = 194, NExp3 = 170), participants played either Bad News (inoculation group) or Tetris (gamified control group) and rated the reliability of news headlines that either used a misinformation technique or not. We found that participants rate fake news as significantly less reliable after the intervention. In Experiment 1, we assessed participants at regular intervals to explore the longevity of this effect and found that the inoculation effect remains stable for at least 3 months. In Experiment 2, we sought to replicate these findings without regular testing and found significant decay over a 2-month time period so that the long-term inoculation effect was no longer significant. In Experiment 3, we replicated the inoculation effect and investigated whether long-term effects could be due to item-response memorization or the fake-to-real ratio of items presented, but found that this is not the case. We discuss implications for inoculation theory and psychological research on misinformation. (PsycInfo Database Record (c) 2021 APA, all rights reserved)