Efficiency in sequential testing: Comparing the sequential probability ratio test and the sequential Bayes factor test

In a sequential hypothesis test, the analyst checks at multiple steps during data collection whether sufficient evidence has accrued to make a decision about the tested hypotheses. As soon as sufficient information has been obtained, data collection is terminated. Here, we compare two sequential hypothesis testing procedures that have recently been proposed for use in psychological research: Sequential Probability Ratio Test (SPRT; Psychological Methods, 25(2), 206–226, 2020) and the Sequential Bayes Factor Test (SBFT; Psychological Methods, 22(2), 322–339, 2017). We show that although the two methods have different philosophical roots, they share many similarities and can even be mathematically regarded as two instances of an overarching hypothesis testing framework. We demonstrate that the two methods use the same mechanisms for evidence monitoring and error control, and that differences in efficiency between the methods depend on the exact specification of the statistical models involved, as well as on the population truth. Our simulations indicate that when deciding on a sequential design within a unified sequential testing framework, researchers need to balance the needs of test efficiency, robustness against model misspecification, and appropriate uncertainty quantification. We provide guidance for navigating these design decisions based on individual preferences and simulation-based design analyses

Bayes factor design analysis: Planning for compelling evidence

A sizeable literature exists on the use of frequentist power analysis in the null-hypothesis significance testing (NHST) paradigm to facilitate the design of informative experiments. In contrast, there is almost no literature that discusses the design of experiments when Bayes factors (BFs) are used as a measure of evidence. Here we explore Bayes Factor Design Analysis (BFDA) as a useful tool to design studies for maximum efficiency and informativeness. We elaborate on three possible BF designs, (a) a fixed-n design, (b) an open-ended Sequential Bayes Factor (SBF) design, where researchers can test after each participant and can stop data collection whenever there is strong evidence for either (Formula presented.) or (Formula presented.), and (c) a modified SBF design that defines a maximal sample size where data collection is stopped regardless of the current state of evidence. We demonstrate how the properties of each design (i.e., expected strength of evidence, expected sample size, expected probability of misleading evidence, expected probability of weak evidence) can be evaluated using Monte Carlo simulations and equip researchers with the necessary information to compute their own Bayesian design analyses

A tutorial on Bayes Factor Design Analysis using an informed prior

Well-designed experiments are likely to yield compelling evidence with efficient sample sizes. Bayes Factor Design Analysis (BFDA) is a recently developed methodology that allows researchers to balance the informativeness and efficiency of their experiment (Schönbrodt & Wagenmakers, Psychonomic Bulletin & Review, 25(1), 128–142 2018). With BFDA, researchers can control the rate of misleading evidence but, in addition, they can plan for a target strength of evidence. BFDA can be applied to fixed-N and sequential designs. In this tutorial paper, we provide an introduction to BFDA and analyze how the use of informed prior distributions affects the results of the BFDA. We also present a user-friendly web-based BFDA application that allows researchers to conduct BFDAs with ease. Two practical examples highlight how researchers can use a BFDA to plan for informative and efficient research designs

A Bayesian bird's eye view of 'Replications of important results in social psychology'

We applied three Bayesian methods to reanalyze the preregistered contributions to the Social Psychology special issue “Replications of Important Results in Social Psychology' [1]. First, individual-experiment Bayesian parameter estimation revealed that for directed effect size measures, only 3 out of 44 central 95% credible intervals did not overlap with zero and fell in the expected direction. For undirected effect size measures, only 4 out of 59 credible intervals contained values greater than :10 (10% of variance explained), and only 19 intervals contained values larger than :05. Second, a Bayesian random-effects meta-analysis for all 38 t-tests showed that only one out of the 38 hierarchically estimated credible intervals did not overlap with zero and fell in the expected direction. Third, a Bayes factor hypothesis test was used to quantify the evidence for the null hypothesis against a default one-sided alternative. Only 7 out of 60 Bayes factors indicated non-anecdotal support in favor of the alternative hypothesis (BF10 > 3), whereas 51 Bayes factors indicated at least some support for the null hypothesis. We hope that future analyses of replication success will embrace a more inclusive statistical approach by adopting a wider range of complementary techniques

Why Condition-Based Regression Analysis (CRA) is Indeed a Valid Test of Self-Enhancement Effects: A Response to Krueger et al. (2017)

How can the consequences of self-enhancement (SE) be tested empirically? Traditional two-step approaches for investigating SE effects have been criticized for providing systematically biased results. Recently, we suggested condition-based regression analysis (CRA) as an approach that enables users to test SE effects while overcoming the shortcomings of previous methods. Krueger et al. (2017) reiterated the problems of previous two-step approaches and criticized the extent to which CRA could solve these problems. However, their critique was based on a misrepresentation of our approach: Whereas a key element of CRA is the requirement that the coefficients of a multiple regression model must meet two conditions, Krueger et al.’s argumentation referred to the test of only a single condition. As a consequence, their reasoning does not allow any conclusions to be drawn about the validity of our approach. In this paper, we clarify these misunderstandings and explain why CRA is a valid approach for investigating the consequences of SE.</p