Linde, Maximilian

Tendeiro, Jorge

van Ravenzwaaij, Don

Wagenmakers, Eric-Jan

English

University of Groningen

   University of GroningenPractical Implications of Equating Equivalence TestsLinde, Maximilian; Tendeiro, Jorge; Wagenmakers, Eric-Jan; van Ravenzwaaij, DonPublished in:Psychological MethodsIMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.Document VersionEarly version, also known as pre-printPublication date:2022Link to publication in University of Groningen/UMCG research databaseCitation for published version (APA):Linde, M., Tendeiro, J., Wagenmakers, E-J., & van Ravenzwaaij, D. (Accepted/In press). PracticalImplications of Equating Equivalence Tests: Reply to Campbell and Gustafson (2022). PsychologicalMethods.CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).The publication may also be distributed here under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license.More information can be found on the University of Groningen website: https://www.rug.nl/library/open-access/self-archiving-pure/taverne-amendment.Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.Download date: 15-02-2023REPLY TO CAMPBELL AND GUSTAFSON (2022) 1Practical Implications of Equating Equivalence Tests: Reply to Campbell andGustafson (2022)Maximilian Linde1, Jorge N. Tendeiro2, Eric-Jan Wagenmakers3, andDon van Ravenzwaaij11Unit of Psychometrics and Statistics, Department of Psychology, Faculty of Behaviouraland Social Sciences, University of Groningen, Groningen, The Netherlands2Office of Research and Academia-Government-Community Collaboration, Education andResearch Center for Artificial Intelligence and Data Innovation, Hiroshima University,Hiroshima, Japan3Unit of Psychological Methods, Department of Psychology, Faculty of Social andBehavioural Sciences, University of Amsterdam, Amsterdam, The NetherlandsREPLY TO CAMPBELL AND GUSTAFSON (2022) 2Author NoteMaximilian Linde https://orcid.org/0000-0001-8421-090XJorge N. Tendeiro https://orcid.org/0000-0003-1660-3642Eric-Jan Wagenmakers https://orcid.org/0000-0003-1596-1034Don van Ravenzwaaij https://orcid.org/0000-0002-5030-4091This research was supported by a Dutch scientific organization VIDI fellowship grantawarded to Don van Ravenzwaaij (016.Vidi.188.001), a Dutch scientific organization VICIfellowship grant awarded to Eric-Jan Wagenmakers (016.Vici.170.083), and a JapaneseJSPS KAKENHI grant awarded to Jorge N. Tendeiro (21K20211). Correspondenceconcerning this article should be addressed to: Maximilian Linde, University of Groningen,Department of Psychology, Grote Kruisstraat 2/1, Heymans Building, room 217, 9712 TSGroningen, The Netherlands, Phone: (+31) 50 363 2702, E-mail: m.linde@rug.nlThe analysis code is available at https://osf.io/rvs9g/.REPLY TO CAMPBELL AND GUSTAFSON (2022) 3AbstractLinde et al. (2021) compared the “two one-sided tests”, the “highest density interval –region of practical equivalence”, and the “interval Bayes factor” approaches to establishingequivalence in terms of power and Type I error rate using typical decision thresholds. Theyfound that the interval Bayes factor approach exhibited a higher power but also a higherType I error rate than the other approaches. In response, Campbell and Gustafson (2022)showed that the performances of the three approaches can approximate one another whenthey are calibrated to have the same Type I error rate. In this article, we argue that theseresults have little bearing on how these approaches are used in practice; a concrete exampleis used to highlight this important point.Keywords: two one-sided tests, highest density interval, region of practicalequivalence, interval Bayes factor, optimal test, calibrationREPLY TO CAMPBELL AND GUSTAFSON (2022) 4Practical Implications of Equating Equivalence Tests: Reply to Campbell andGustafson (2022)In a recent simulation study (Linde et al., 2021, henceforth LTSWvR), we comparedthe performances of three approaches to establishing equivalence between two groups.Specifically, we examined the statistical power and Type I error rate of the followingmethods: (1) the “two one-sided tests” (TOST; Hodges & Lehmann, 1954; Schuirmann,1987; Westlake, 1976); (2) the “highest density interval – region of practical equivalence”(HDI-ROPE; Kruschke, 2011, 2015, 2018); and (3) the “interval Bayes factor” (BF; Morey& Rouder, 2011; van Ravenzwaaij et al., 2019; see also Linde & van Ravenzwaaij, 2019).LTSWvR found that when using typical parameters (i.e., α = .05 for TOST, 95% HDI forHDI-ROPE, and BF thresholds of 3 and 10 for BF) the BF approach exhibited a higherstatistical power to detect equivalence but also a higher Type I error rate than the TOSTand HDI-ROPE approaches. This difference in operating characteristics was most evidentfor relatively small sample sizes (i.e., n = 50 and n = 100 per group) and narrowequivalence intervals (i.e., m = .1 and m = .2); in these cases, the TOST and HDI-ROPEapproaches were unable to conclude equivalence at all, and application of these procedureswould therefore result in a foregone conclusion.In response, Campbell and Gustafson (2022, henceforth CG) demonstrated that theHDI-ROPE, the BF, and the so-called “optimal test” (OT; Möllenhoff et al., 2022;Romano, 2005) procedures for establishing equivalence “can be reverse-engineered from theother (at least approximately)” so that they exhibit almost identical performancecharacteristics (CG agree with LTSWvR where TOST is concerned). However, thecalibration results of CG often required the adoption of extreme and highly unorthodoxvalues, such as HDIs with coverage close to 0% instead of the typical 95% or BF thresholdshigher than 196. In our assessment, no researcher would ever apply these procedures withsuch threshold values, and therefore we argue that the results of CG have limited impacton statistical practice. Although we believe that CG provide a valuable contribution andREPLY TO CAMPBELL AND GUSTAFSON (2022) 5important qualification to the findings of LTSWvR, we are worried that the readership isleft with the impression that it does not matter in practice which approach to establishingequivalence is chosen.To demonstrate why the choice of methodology for establishing equivalence mattersin practice, we reanalyzed a series of real studies (Galak et al., 2012) that attempted toreplicate experiments 8 and 9 of Bem (2011). In his original experiments, Bem (2011)claimed to have found consistent evidence for the existence of precognition, the idea thatfuture events can retroactively influence current experiences and processes. We specificallychose the replications of Galak et al. (2012) for our reanalyses because (1) we can safelyassume that the null hypothesis, claiming the absence of an effect, is in fact true (unless weare willing to turn over major parts of the current scientific framework), which is necessarysince we want to determine whether the approaches to establishing equivalence makecorrect decisions; and (2) Galak et al. (2012) provide summary statistics in their Table 1that we can use for the reanalyses.1 We reanalyzed experiments 1−5, 6 (test-beforepractice), and 7; we did not reanalyze the original experiments of Bem (2011) andexperiment 6 (practice-before-test) because in the latter case an effect was expected. Therelevant summary statistics of Table 1 in Galak et al. (2012) are reproduced in Table 1.Some alterations to the procedures described in LTSWvR were necessary: Firstly, inorder to conform to the study designs in Galak et al. (2012), we adjusted the TOST,HDI-ROPE, and BF procedures to work for one-sample designs. Secondly, CG agree withLTSWvR that TOST is less suitable for relatively small sample sizes and therefore suggestto use OT instead.2 This led us to include OT in addition to the TOST, HDI-ROPE, and1 Table 1 of Galak et al. (2012) claims to report standard deviations but the reported t-values suggest thatthe values that are indicated as standard deviations are actually standard errors.2 It is noteworthy that when the sample mean (in case of the one-sample design) or the difference insample means (in case of the two-sample design) is exactly 0, OT provides a p-value of exactly 0,irrespective of the sample size and the equivalence margin used. This is undesirable as the p-value shouldnot be so extreme in case we only have, say, N = 5 cases. Mathematically speaking, a sample mean or aREPLY TO CAMPBELL AND GUSTAFSON (2022) 6Table 1Summary statistics from Table 1of Galak et al. (2012).Study N M SE1 112 −1.21 1.012 158 0.00 0.773 124 1.17 0.924 109 1.59 0.905 211 −0.49 0.696 (TBPa) 106 −0.29 0.887 2469 −0.05 0.22Note. aTBP = test-before-practice.BF approaches. For OT, we adapted the code in the Appendix of CG. Similar to LTSWvR,we chose α = .05 for TOST and OT, 95% HDI for HDI-ROPE, and BFthr = 10 for BF.Moreover, we used a standardized equivalence margin of m = .1 and a Cauchy prior scaleof r = 1/√2 for HDI-ROPE and BF. All code for our analyses can be examined athttps://osf.io/rvs9g/.All-or-none DecisionsTable 2 shows the results of our reanalyses. Both TOST and HDI-ROPE only makethe correct decision of concluding equivalence in study 7. OT declares equivalence instudies 2 and 7. Lastly, BF concludes equivalence in studies 2, 5, 6, and 7. Thus, BF makesthe most correct decisions.Following the logic of CG, we can calibrate the four approaches, so that they makesample mean difference of exactly 0 should never occur, but in practice researchers round their numbers tofinite decimals and as such this test may well produce extremely low p-values even for very little data.REPLY TO CAMPBELL AND GUSTAFSON (2022) 7the same decisions. Specifically, if we would have used a Bayes factor threshold between32.528 (cf. study 5) and 39.911 (cf. study 2), BF would have made the same twoequivalence conclusions as OT. Similarly, if we would have used a significance levelanywhere between α = .214 (cf. study 5) and α = .544 (cf. study 3), OT would have madethe same four equivalence decisions as BF. However, equating TOST, OT, HDI-ROPE, andBF through this kind of calibration is cumbersome (it requires sifting through all theanalysis results and making post-hoc corrections), ad-hoc (different calibration levels foreach data set), and defeats the purpose of conducting statistical inference in the first place(thresholds should be set based on substantive reasons or convention).Bayes Factors Quantify EvidenceA Bayes factor does not require a decision threshold. That is, the Bayes factorquantifies evidence; it compares the probability of the data under H0 to the probability ofthe data under H1. In other words, it quantifies the relative predictive performance of therival hypotheses, and thereby yields the extent to which researchers should update theirbeliefs after seeing the data. In contrast, a p-value resulting from TOST and OT does notrelate to the probability of the data under two hypotheses, but instead focuses solely onthe probability of the observed data (or data more extreme) under H0.In our reanalyses of the Galak et al. (2012) replication experiments, the obtainedBayes factors always provide evidence towards H0 (i.e., all BF01 > 3). For example, forstudies 1 and 3 we obtained BF01 = 8.205 and BF01 = 8.062, respectively. Even though wecannot conclude equivalence in an all-or-none decision for these Bayes factors (because theevidence falls below the stipulated threshold of 10), they still provide moderate evidence infavor of equivalence. On the other hand, when the result of TOST or OT is anon-significant p-value, we cannot conclude anything because it is impossible to disentanglewhether the non-significant p-value resulted from (1) equivalence at the population levelcombined with insufficient statistical power; (2) non-equivalence at the population level(Bakan, 1966; Keysers et al., 2020).REPLY TO CAMPBELL AND GUSTAFSON (2022) 8ConclusionWe agree with CG that OT, HDI-ROPE, and BF can be calibrated to have thesame operating characteristics in terms of statistical power and Type I error rate. However,this calibration requires the use of extreme, highly unorthodox settings; consequently, wedo not believe the calibration to be helpful for pragmatic researchers. Our reanalysis of theGalak et al. (2012) replication experiments confirms the practical advantages of the BFapproach for equivalence testing.REPLY TO CAMPBELL AND GUSTAFSON (2022) 9Table 2Comparison of TOST, OT, HDI-ROPE, and BF for 7replication experiments from Galak et al. (2012).MethodStudy TOST OT HDI-ROPE BF(p-value) (p-value) (max |HDI|) (BF01)1 .555 .544 0.293 8.2052 .105 < .001 0.154 39.9113 .563 .554 0.285 8.0624 .764 .763 0.351 3.3485 .229 .214 0.182 32.5286 (TBPa) .243 .155 0.218 21.6647 < .001 < .001 0.044 > 1000Note. Cells with green background indicate equivalencedecisions. For TOST and OT, equivalence is concluded ifp < α = .05; for HDI-ROPE, equivalence is concluded ifmax |HDI| < m = 0.1; for BF, equivalence is concluded ifBF01 > BFthr = 10.aTBP = test-before-practice.REPLY TO CAMPBELL AND GUSTAFSON (2022) 10ReferencesBakan, D. (1966). The test of significance in psychological research. Psychological Bulletin,66 (6), 423–437. https://doi.org/10.1037/h0020412Bem, D. J. (2011). Feeling the future: Experimental evidence for anomalous retroactiveinfluences on cognition and affect. Journal of Personality and Social Psychology,100 (3), 407–425. https://doi.org/10.1037/a0021524Campbell, H., & Gustafson, P. (2022). re:Linde et al. (2021): The Bayes factor, HDI-ROPEand frequentist equivalence tests can all be reverse engineered – almost exactly –from one another.Galak, J., LeBoeuf, R. A., Nelson, L. D., & Simmons, J. P. (2012). Correcting the past:Failures to replicate psi. Journal of Personality and Social Psychology, 103 (6),933–948. https://doi.org/10.1037/a0029709Hodges, J. L., & Lehmann, E. L. (1954). Testing the approximate validity of statisticalhypotheses. Journal of the Royal Statistical Society. Series B (Methodological),16 (2), 261–268.Keysers, C., Gazzola, V., & Wagenmakers, E.-J. (2020). Using Bayes factor hypothesistesting in neuroscience to establish evidence of absence. Nature Neuroscience, 23,788–799. https://doi.org/10.1038/s41593-020-0660-4Kruschke, J. K. (2011). Bayesian assessment of null values via parameter estimation andmodel comparison. Perspectives on Psychological Science, 6 (3), 299–312.https://doi.org/10.1177/1745691611406925Kruschke, J. K. (2015). Doing Bayesian data analysis: A tutorial with R, JAGS, and Stan(2nd). Academic Press.Kruschke, J. K. (2018). Rejecting or accepting parameter values in Bayesian estimation.Advances in Methods and Practices in Psychological Science, 1 (2), 270–280.https://doi.org/10.1177/2515245918771304REPLY TO CAMPBELL AND GUSTAFSON (2022) 11Linde, M., Tendeiro, J. N., Selker, R., Wagenmakers, E.-J., & van Ravenzwaaij, D. (2021).Decisions about equivalence: A comparison of tost, hdi-rope, and the Bayes factor.Psychological Methods, Advance online publication.https://doi.org/10.1037/met0000402Linde, M., & van Ravenzwaaij, D. (2019). baymedr: An R package for the calculation ofBayes factors for equivalence, non-inferiority, and superiority designs.Möllenhoff, K., Loingeville, F., Bertrand, J., Nguyen, T. T., Sharan, S., Zhao, L., Fang, L.,Sun, G., Grosser, S., Mentré, F., & Dette, H. (2022). Efficient model-basedbioequivalence testing. Biostatistics, 23 (1), 314–327.https://doi.org/10.1093/biostatistics/kxaa026Morey, R. D., & Rouder, J. N. (2011). Bayes factor approaches for testing interval nullhypotheses. Psychological Methods, 16 (4), 406–419.https://doi.org/10.1037/a0024377Romano, J. P. (2005). Optimal testing of equivalence hypotheses. The Annals of Statistics,33 (3), 1036–1047. https://doi.org/10.1214/009053605000000048Schuirmann, D. J. (1987). A comparison of the two one-sided tests procedure and thepower approach for assessing the equivalence of average bioavailability. Journal ofPharmacokinetics and Biopharmaceutics, 15 (6), 657–680.https://doi.org/10.1007/BF01068419van Ravenzwaaij, D., Monden, R., Tendeiro, J. N., & Ioannidis, J. P. A. (2019). Bayesfactors for superiority, non-inferiority, and equivalence designs. BMC MedicalResearch Methodology, 19 (1), 71. https://doi.org/10.1186/s12874-019-0699-7Westlake, W. J. (1976). Symmetrical confidence intervals for bioequivalence trials.Biometrics, 32 (4), 741–744. https://doi.org/10.2307/2529259

Practical Implications of Equating Equivalence Tests:Reply to Campbell and Gustafson (2022)

https://pure.rug.nl/ws/files/553355084/preprint.pdf

Practical Implications of Equating Equivalence Tests:Reply to Campbell and Gustafson (2022)

Abstract

Similar works

Full text

Available Versions

University of Groningen