14,970 research outputs found
Consecutive Sequential Probability Ratio Tests of Multiple Statistical Hypotheses
In this paper, we develop a simple approach for testing multiple statistical
hypotheses based on the observations of a number of probability ratios
enumerated consecutively with respect to the index of hypotheses. Explicit and
tight bounds for the probability of making wrong decisions are obtained for
choosing appropriate parameters for the proposed tests. In the special case of
testing two hypotheses, our tests reduce to Wald's sequential probability ratio
tests.Comment: 29 pages, no figure; The main results of this paper have appeared in
Proceedings of SPIE Conferences, Baltimore, Maryland, April 24-27, 201
Continuous Monitoring of A/B Tests without Pain: Optional Stopping in Bayesian Testing
A/B testing is one of the most successful applications of statistical theory
in modern Internet age. One problem of Null Hypothesis Statistical Testing
(NHST), the backbone of A/B testing methodology, is that experimenters are not
allowed to continuously monitor the result and make decision in real time. Many
people see this restriction as a setback against the trend in the technology
toward real time data analytics. Recently, Bayesian Hypothesis Testing, which
intuitively is more suitable for real time decision making, attracted growing
interest as an alternative to NHST. While corrections of NHST for the
continuous monitoring setting are well established in the existing literature
and known in A/B testing community, the debate over the issue of whether
continuous monitoring is a proper practice in Bayesian testing exists among
both academic researchers and general practitioners. In this paper, we formally
prove the validity of Bayesian testing with continuous monitoring when proper
stopping rules are used, and illustrate the theoretical results with concrete
simulation illustrations. We point out common bad practices where stopping
rules are not proper and also compare our methodology to NHST corrections.
General guidelines for researchers and practitioners are also provided
A Rejection Principle for Sequential Tests of Multiple Hypotheses Controlling Familywise Error Rates
We present a unifying approach to multiple testing procedures for sequential
(or streaming) data by giving sufficient conditions for a sequential multiple
testing procedure to control the familywise error rate (FWER), extending to the
sequential domain the work of Goeman and Solari (2010) who accomplished this
for fixed sample size procedures. Together we call these conditions the
"rejection principle for sequential tests," which we then apply to some
existing sequential multiple testing procedures to give simplified
understanding of their FWER control. Next the principle is applied to derive
two new sequential multiple testing procedures with provable FWER control, one
for testing hypotheses in order and another for closed testing. Examples of
these new procedures are given by applying them to a chromosome aberration data
set and to finding the maximum safe dose of a treatment
- …