Bartoš, F.

Harris, A.J.L.

Maier, M.

Shanks, D.R.

Stanley, T.D.

Wagenmakers, E.-J.

English

International Migration, Integration and Social Cohesion online publications

UvA-DARE is a service provided by the library of the University of Amsterdam (https://dare.uva.nl)UvA-DARE (Digital Academic Repository)No evidence for nudging after adjusting for publication biasMaier, M.; Bartoš, F.; Stanley, T.D.; Shanks, D.R.; Harris, A.J.L.; Wagenmakers, E.-J.DOI10.1073/pnas.2200300119Publication date2022Document VersionFinal published versionPublished inProceedings of the National Academy of Sciences of the United States of AmericaLicenseCC BYLink to publicationCitation for published version (APA):Maier, M., Bartoš, F., Stanley, T. D., Shanks, D. R., Harris, A. J. L., & Wagenmakers, E-J.(2022). No evidence for nudging after adjusting for publication bias. Proceedings of theNational Academy of Sciences of the United States of America, 119(31), [e2200300119].https://doi.org/10.1073/pnas.2200300119General rightsIt is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s)and/or copyright holder(s), other than for strictly personal, individual use, unless the work is under an opencontent license (like Creative Commons).Disclaimer/Complaints regulationsIf you believe that digital publication of certain material infringes any of your rights or (privacy) interests, pleaselet the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the materialinaccessible and/or remove it from the website. Please Ask the Library: https://uba.uva.nl/en/contact, or a letterto: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. Youwill be contacted as soon as possible.Download date:10 Mar 2023No evidence for nudging after adjusting for publication biasMaximilian Maiera,1,2 , Frantisek Bartosb,1 , T. D. Stanleyc,d, David R. Shanksa , Adam J. L. Harrisa, andEric-Jan WagenmakersbThaler and Sunstein’s “nudge” (1) has spawned a revolutionin behavioral science research. Despite its popularity, the“nudge approach” has been criticized for having a “limitedevidence base” (e.g., ref. 2). Mertens et al. (3) seek toaddress that limitation with a timely and comprehensivemetaanalysis. Mertens et al.’s headline finding is that“choice architecture [nudging] is an effective and widelyapplicable behavior change tool” (p. 8). We propose theirfinding of “moderate publication bias” (p. 1) is the realheadline; when this publication bias is appropriately cor-rected for, no evidence for the effectiveness of nudgesremains (Fig. 1).Mertens et al. (3) find significant publication bias,through Egger regression. Their sensitivity analysis (4)indicates that the true effect size could be as low asd = 0.08 (if publication bias is severe). Mertens et al.argue that severe publication bias is only partially sup-ported by the funnel plot and proceed largely withouttaking publication bias into account in their subsequentanalyses. However, the reported Egger coefficient (b = 2.10) is“severe” (5).A newly proposed bias correction technique, robust Bayes-ian metaanalysis (RoBMA) (6), avoids an all-or-none debateover whether or not publication bias is “severe.” RoBMAsimultaneously applies 1) selection models that estimate rela-tive publication probabilities (7) and 2) models of the relation-ship between effect sizes and SEs [i.e., Precision Effect Testand Precision Effect Estimate with Standard Error (6, 8, 9)].Multimodel inference is then guided mostly by those modelsthat predict the observed data best (6, 9, 10). RoBMA makesmultimodel inferences about the presence or absence of aneffect, heterogeneity, and publication bias (6, 9).Table 1 compares the unadjusted results to the publica-tion bias–adjusted results.* Since publication bias–correctedthree-level selection models are computationally intracta-ble, we analyzed the data in two ways: 1) ignoring the three-level structure (column 2) and 2) using only the mostprecise estimate from studies with multiple results (column3). Strikingly, there is an absence of evidence for an overalleffect and evidence against an effect in the “information”and “assistance” intervention categories, whereas the evi-dence is undecided for “structure” interventions. Whenusing only the most precise estimates, we further find evi-dence against an effect in most of the domains, apart from“other,” “food,” and “prosocial” (the evidence is indecisive)and weak evidence for the overall effect.† However, all inter-vention categories and domains apart from “finance” showevidence for heterogeneity, which implies that some nudgesmight be effective, even when there is evidence against the−0.4 −0.2 0.0 0.2 0.4Cohen's dCombined 0.04 [0.00, 0.14]BF01 = 0.95Interventioncategory:InformationStructureAssistance0.00 [0.00, 0.00]BF01 = 33.840.12 [0.00, 0.43]BF01 = 1.120.01 [0.00, 0.07]BF01 = 9.05Domain:HealthFoodEnvironmentFinancePro−socialOther0.01 [0.00, 0.10]BF01 = 8.980.02 [−0.09, 0.32]BF01 = 5.160.01 [−0.18, 0.25]BF01 = 4.410.00 [0.00, 0.00]BF01 = 41.230.00 [0.00, 0.05]BF01 = 11.930.08 [0.00, 0.33]BF01 = 1.38Fig. 1. RoBMAPSMA model-averaged posterior mean effect size estimateswith 95% credible intervals and Bayes factors for the absence of the effectfor the combined sample or split by either the domain or intervention cate-gory (ignoring the clustering of SEs). BF01 quantifies evidence for the nullhypothesis. BF01 larger than one corresponds to evidence in favor of thenull hypothesis, and BF01 lower than one corresponds to evidence in favorof the alternative hypothesis (evidence for the alternative hypothesis canbe obtained by reciprocating the Bayes factor; BF10 = 1/BF01). As a rule ofthumb, Bayes factors between 3 and 10 indicate moderate evidence, andBayes factors larger than 10 indicate strong evidence.Author affiliations: aDepartment of Experimental Psychology, University College London,London WC1H 0AP, United Kingdom; bDepartment of Psychological Methods, Universityof Amsterdam, Amsterdam, 1018 WS, The Netherlands; cDeakin Laboratory for the Meta-Analysis of Research, Deakin University, Burwood, VIC 3125, Australia; and dDepartmentof Economics, Deakin University, Burwood, VIC 3125, AustraliaAuthor contributions: M.M. and F.B. designed research; M.M. and F.B. performedresearch; M.M. and F.B. analyzed data; and M.M., F.B., T.D.S., D.R.S., A.J.L.H., and E.-J.W.wrote the paper.The authors declare no competing interest.Copyright © 2022 the Author(s). Published by PNAS. This article is distributed underCreative Commons Attribution License 4.0 (CC BY).1M.M. and F.B. contributed equally to this work.2To whom correspondence may be addressed. Email: m.maier@ucl.ac.uk.Published July 19, 2022.*Our analysis is based on the corrected dataset in ref. 12.†We also reanalyzed the data by including only models of selection for statistical signifi-cance, confirming our results.PNAS 2022 Vol. 119 No. 31 e2200300119 https://doi.org/10.1073/pnas.2200300119 1 of 2LETTERDownloaded from https://www.pnas.org by Universiteit van Amsterdam on December 13, 2022 from IP address 146.50.157.192.mean effect. Finally, we find strong evidence for publicationbias across all subdomains (BFpb > 10), apart from food,when using only the most precise estimates (BFpb = 2.49).We conclude that the “nudge” literature analyzed inref. 3 is characterized by severe publication bias. Contraryto Mertens et al. (3), our Bayesian analysis indicates that,after correcting for this bias, no evidence remains thatnudges are effective as tools for behaviour change.Data Availability. Data and analysis script are available in ref. 11.ACKNOWLEDGMENTS. We thank Mertens et al. for sharing well-documenteddata and code.1. R. H. Thaler, C. R. Sunstein, Nudge: Improving Decisions about Health, Wealth, and Happiness (Yale University Press, 2008).2. Y. Lin, M. Osman, R. Ashcroft, Nudge: Concept, effectiveness, and ethics. Basic Appl. Soc. Psych. 39, 293–306 (2017).3. S. Mertens, M. Herberz, U. J. J. Hahnel, T. Brosch, The effectiveness of nudging: A meta-analysis of choice architecture interventions across behavioral domains. Proc. Natl. Acad. Sci. U.S.A. 119, e2107346118(2022).4. J. L. Vevea, C. M. Woods, Publication bias in research synthesis: Sensitivity analysis using a priori weight functions. Psychol. Methods 10, 428–443 (2005).5. C. Doucouliagos, T. D. Stanley, Theory competition and selectivity: Are all economic facts greatly exaggerated? J. Econ. Surv. 27, 316–339 (2013).6. M. Maier, F. Bartos, E. J. Wagenmakers, Robust Bayesian meta-analysis: Addressing publication bias with model-averaging. Psychol. Methods, 10.1037/met0000405 (2022).7. J. L. Vevea, L. V. Hedges, A general linear model for estimating effect size in the presence of publication bias. Psychometrika 60, 419–435 (1995).8. T. D. Stanley, H. Doucouliagos, Meta-regression approximations to reduce publication selection bias. Res. Synth. Methods 5, 60–78 (2014).9. F. Bartos, M. Maier, E.-J. Wagenmakers, H. Doucouliagos, T. D. Stanley, No need to choose: Model-averaging across complementary publication bias adjustment methods. Evidence Synthesis Methods, in press.10. J. A. Hoeting, D. Madigan, A. E. Raftery, C. T. Volinsky, Bayesian model averaging: A tutorial. Stat. Sci. 14, 382–417 (1999).11. S. Mertens, M. Herberz, U. J. J. Hahnel, T. Brosch, mhhb_nma_data_corrected.csv. Open Science Framework. https://osf.io/ubt9a/. Accessed 3 May 2022.12. M. Maier et al., Code and data for analyses in “No evidence for nudging after adjusting for publication bias.” Open Science Framework. https://osf.io/svz6e/. Deposited 6 January 2022.Table 1. Comparison of unadjusted and adjusted effect size estimates for all studies and for subsets ofstudies based on different categories or domainsRandom effects RoBMAPSMA RoBMAPSMA (precise)Combined 0.43 [0.38, 0.48] 0.04 [0.00, 0.14] 0.11 [0.00, 0.24]t(333) = 16.51 BF01 = 0.95 BF01 = 0.31Intervention categoryInformation 0.25 [0.19, 0.30] 0.00 [0.00, 0.00] 0.00 [0.00, 0.07]t(88) = 8.79 BF01 = 33.84 BF01 = 10.57Structure 0.58 [0.50, 0.66] 0.12 [0.00, 0.43] 0.23 [0.00, 0.49]t(186) = 13.93 BF01 = 1.12 BF01 = 0.33Assistance 0.22 [0.15, 0.29] 0.01 [0.00, 0.07] 0.01 [0.00, 0.12]t(65) = 6.42 BF01 = 9.05 BF01 = 8.00DomainHealth 0.31 [0.22, 0.39] 0.01 [0.00, 0.10] 0.02 [0.00, 0.19]t(64) = 7.03 BF01 = 8.98 BF01 = 3.53Food 0.66 [0.52, 0.81] 0.02 [0.09, 0.32] 0.27 [0.00, 0.64]t(81) = 9.01 BF01 = 5.16 BF01 = 0.55Environment 0.48 [0.37, 0.58] 0.01 [0.18, 0.25] 0.00 [0.44, 0.34]t(56) = 9.16 BF01 = 4.41 BF01 = 3.05Finance 0.23 [0.15, 0.31] 0.00 [0.00, 0.00] 0.00 [0.00, 0.00]t(34) = 6.08 BF01 = 41.23 BF01 = 30.95Prosocial 0.32 [0.22, 0.42] 0.00 [0.00, 0.05] 0.05 [0.00, 0.27]t(38) = 6.36 BF01 = 11.93 BF01 = 1.89Other 0.40 [0.29, 0.50] 0.08 [0.00, 0.33] 0.04 [0.22, 0.40]t(55) = 7.66 BF01 = 1.38 BF01 = 2.45First column: Random effects metaanalysis estimates with 95% CI based on clustered SEs, all P values < 0.001. Second andthird columns: RoBMAPSMA model-averaged posterior mean effect size estimates with 95% credible intervals and Bayesfactor for the presence of the effect ignoring the clustering of SEs or using the most precise estimates (precise). Resultsdiffer slightly from the moderator analysis presented in the article because we analyzed each subfield separately toallow 1) testing for the presence of the effect in each category/domain in the Bayesian framework, and 2) publicationbias to operate differently in different subdomains.2 of 2 https://doi.org/10.1073/pnas.2200300119 pnas.orgDownloaded from https://www.pnas.org by Universiteit van Amsterdam on December 13, 2022 from IP address 146.50.157.192.

No evidence for nudging after adjusting for publication bias

https://pure.uva.nl/ws/files/100872641/pnas.2200300119.pdf

No evidence for nudging after adjusting for publication bias

Abstract

Similar works

Full text

Available Versions

International Migration, Integration and Social Cohesion online publications