Optimal testing of equivalence hypotheses

In this paper we consider the construction of optimal tests of equivalence hypotheses. Specifically, assume X_1,..., X_n are i.i.d. with distribution P_{\theta}, with \theta \in R^k. Let g(\theta) be some real-valued parameter of interest. The null hypothesis asserts g(\theta)\notin (a,b) versus the alternative g(\theta)\in (a,b). For example, such hypotheses occur in bioequivalence studies where one may wish to show two drugs, a brand name and a proposed generic version, have the same therapeutic effect. Little optimal theory is available for such testing problems, and it is the purpose of this paper to provide an asymptotic optimality theory. Thus, we provide asymptotic upper bounds for what is achievable, as well as asymptotically uniformly most powerful test constructions that attain the bounds. The asymptotic theory is based on Le Cam's notion of asymptotically normal experiments. In order to approximate a general problem by a limiting normal problem, a UMP equivalence test is obtained for testing the mean of a multivariate normal mean.Comment: Published at http://dx.doi.org/10.1214/009053605000000048 in the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org

Explicit nonparametric confidence intervals for the variance with guaranteed coverage

In this paper, we provide a method for constructing confidence intervals for the variance that exhibit guaranteed coverage probability for any sample size, uniformly over a wide class of probability distributions. In contrast, standard methods achieve guaranteed coverage only in the limit for a fixed distribution or for any sample size over a very restrictive (parametric) class of probability distributions. Of course, it is impossible to construct effective confidence intervals for the variance without some restriction, due to a result of Bahadur and Savage (1956). However, it is possible if the observations lie in a fixed compact set. We also consider the case of lower confidence bounds without any support restriction. Our method is based on the behavior of the variance over distributions that lie within a Kolmogorov-Smirnov confidence band for the underlying distribution. The method is a generalization of an idea of Anderson (1967), who considered only the case of the mean; it applies to very general parameters, and particularly the variance. While typically it is not clear how to compute these intervals explicitly, for the special case of the variance we provide an algorithm to do so. Asymptotically, the length of the intervals is of order n -1/2 in probability), so that, while providing guaranteed coverage, they are not overly conservative. A small simulation study examines the finite sample behavior of the proposed intervals

Exact and approximate stepdown methods for multiple hypothesis testing

Consider the problem of testing k hypotheses simultaneously. In this paper, we discuss finite and large sample theory of stepdown methods that provide control of the familywise error rate (FWE). In order to improve upon the Bonferroni method or Holm's (1979) stepdown method, Westfall and Young (1993) make eective use of resampling to construct stepdown methods that implicitly estimate the dependence structure of the test statistics. However, their methods depend on an assumption called subset pivotality. The goal of this paper is to construct general stepdown methods that do not require such an assumption. In order to accomplish this, we take a close look at what makes stepdown procedures work, and a key component is a monotonicity requirement of critical values. By imposing such monotonicity on estimated critical values (which is not an assumption on the model but an assumption on the method), it is demonstrated that the problem of constructing a valid multiple test procedure which controls the FWE can be reduced to the problem of contructing a single test which controls the usual probability of a Type 1 error. This reduction allows us to draw upon an enormous resampling literature as a general means of test contruction.Bootstrap, familywise error rate, multiple testing, permutation test, randomization test, stepdown procedure, subsampling

On stepdown control of the false discovery proportion

Consider the problem of testing multiple null hypotheses. A classical approach to dealing with the multiplicity problem is to restrict attention to procedures that control the familywise error rate ($FWER$), the probability of even one false rejection. However, if $s$ is large, control of the $FWER$ is so stringent that the ability of a procedure which controls the $FWER$ to detect false null hypotheses is limited. Consequently, it is desirable to consider other measures of error control. We will consider methods based on control of the false discovery proportion ($FDP$) defined by the number of false rejections divided by the total number of rejections (defined to be 0 if there are no rejections). The false discovery rate proposed by Benjamini and Hochberg (1995) controls $E(FDP)$. Here, we construct methods such that, for any $\gamma$ and $\alpha$, $P\{FDP>\gamma \}\le \alpha$. Based on $p$-values of individual tests, we consider stepdown procedures that control the $FDP$, without imposing dependence assumptions on the joint distribution of the $p$-values. A greatly improved version of a method given in Lehmann and Romano \citer10 is derived and generalized to provide a means by which any sequence of nondecreasing constants can be rescaled to ensure control of the $FDP$. We also provide a stepdown procedure that controls the $FDR$ under a dependence assumption.Comment: Published at http://dx.doi.org/10.1214/074921706000000383 in the IMS Lecture Notes--Monograph Series (http://www.imstat.org/publications/lecnotes.htm) by the Institute of Mathematical Statistics (http://www.imstat.org

Stepup procedures for control of generalizations of the familywise error rate

Consider the multiple testing problem of testing null hypotheses $H_1,...,H_s$. A classical approach to dealing with the multiplicity problem is to restrict attention to procedures that control the familywise error rate ($\mathit{FWER}$), the probability of even one false rejection. But if $s$ is large, control of the $\mathit{FWER}$ is so stringent that the ability of a procedure that controls the $\mathit{FWER}$ to detect false null hypotheses is limited. It is therefore desirable to consider other measures of error control. This article considers two generalizations of the $\mathit{FWER}$. The first is the $k-\mathit{FWER}$, in which one is willing to tolerate $k$ or more false rejections for some fixed $k\geq 1$. The second is based on the false discovery proportion ($\mathit{FDP}$), defined to be the number of false rejections divided by the total number of rejections (and defined to be 0 if there are no rejections). Benjamini and Hochberg [J. Roy. Statist. Soc. Ser. B 57 (1995) 289--300] proposed control of the false discovery rate ($\mathit{FDR}$), by which they meant that, for fixed $\alpha$, $E(\mathit{FDP})\leq\alpha$. Here, we consider control of the $\mathit{FDP}$ in the sense that, for fixed $\gamma$ and $\alpha$, $P\{\mathit{FDP}>\gamma\}\leq \alpha$. Beginning with any nondecreasing sequence of constants and $p$-values for the individual tests, we derive stepup procedures that control each of these two measures of error control without imposing any assumptions on the dependence structure of the $p$-values. We use our results to point out a few interesting connections with some closely related stepdown procedures. We then compare and contrast two $\mathit{FDP}$-controlling procedures obtained using our results with the stepup procedure for control of the $\mathit{FDR}$ of Benjamini and Yekutieli [Ann. Statist. 29 (2001) 1165--1188].Comment: Published at http://dx.doi.org/10.1214/009053606000000461 in the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org