229 research outputs found
Multiple Testing for Exploratory Research
Motivated by the practice of exploratory research, we formulate an approach
to multiple testing that reverses the conventional roles of the user and the
multiple testing procedure. Traditionally, the user chooses the error
criterion, and the procedure the resulting rejected set. Instead, we propose to
let the user choose the rejected set freely, and to let the multiple testing
procedure return a confidence statement on the number of false rejections
incurred. In our approach, such confidence statements are simultaneous for all
choices of the rejected set, so that post hoc selection of the rejected set
does not compromise their validity. The proposed reversal of roles requires
nothing more than a review of the familiar closed testing procedure, but with a
focus on the non-consonant rejections that this procedure makes. We suggest
several shortcuts to avoid the computational problems associated with closed
testing.Comment: Published in at http://dx.doi.org/10.1214/11-STS356 the Statistical
Science (http://www.imstat.org/sts/) by the Institute of Mathematical
Statistics (http://www.imstat.org
Simultaneous directional inference
We consider the problem of inference on the signs of parameters. We aim
to provide post-hoc confidence bounds on the number of positive and
negative (or non-positive) parameters. The guarantee is simultaneous, for all
subsets of parameters. Our suggestion is as follows: start by using the data to
select the direction of the hypothesis test for each parameter; then, adjust
the -values of the one-sided hypotheses for the selection, and use the
adjusted -values for simultaneous inference on the selected one-sided
hypotheses. The adjustment is straightforward assuming that the -values of
one-sided hypotheses have densities with monotone likelihood ratio, and are
mutually independent. We show that the bounds we provide are tighter (often by
a great margin) than existing alternatives, and that they can be obtained by at
most a polynomial time. We demonstrate the usefulness of our simultaneous
post-hoc bounds in the evaluation of treatment effects across studies or
subgroups. Specifically, we provide a tight lower bound on the number of
studies which are beneficial, as well as on the number of studies which are
harmful (or non-beneficial), and in addition conclude on the effect direction
of individual studies, while guaranteeing that the probability of at least one
wrong inference is at most 0.05.Comment: 59 pages, 11 figures, 7 table
Multi Split Conformal Prediction
Split conformal prediction is a computationally efficient method for
performing distribution-free predictive inference in regression. It involves,
however, a one-time random split of the data, and the result depends on the
particular split. To address this problem, we propose multi split conformal
prediction, a simple method based on Markov's inequality to aggregate single
split conformal prediction intervals across multiple splits.Comment: 12 pages, 1 figure, 2 tabl
Rejoinder to "Multiple Testing for Exploratory Research"
Rejoinder to "Multiple Testing for Exploratory Research" by J. J. Goeman, A.
Solari [arXiv:1208.2841].Comment: Published in at http://dx.doi.org/10.1214/11-STS356REJ the
Statistical Science (http://www.imstat.org/sts/) by the Institute of
Mathematical Statistics (http://www.imstat.org
Only Closed Testing Procedures are Admissible for Controlling False Discovery Proportions
We consider the class of all multiple testing methods controlling tail
probabilities of the false discovery proportion, either for one random set or
simultaneously for many such sets. This class encompasses methods controlling
familywise error rate, generalized familywise error rate, false discovery
exceedance, joint error rate, simultaneous control of all false discovery
proportions, and others, as well as seemingly unrelated methods such as gene
set testing in genomics and cluster inference methods in neuroimaging. We show
that all such methods are either equivalent to a closed testing method, or are
uniformly improved by one. Moreover, we show that a closed testing method is
admissible as a method controlling tail probabilities of false discovery
proportions if and only if all its local tests are admissible. This implies
that, when designing such methods, it is sufficient to restrict attention to
closed testing methods only. We demonstrate the practical usefulness of this
design principle by constructing a uniform improvement of a recently proposed
method
- …