61,436 research outputs found
How Strong is the Confirmation of a Hypothesis by Significant Data?
The aim of the article is to determine how much a hypothesis H is actually confirmed if it has successfully passed a classical significance test. Bayesians have already raised many serious objections against significance testing, but in doing so they have always had to rely on epistemic probabilities and a further Bayesian analysis, which are rejected by classical statisticians. Therefore, I will suggest a purely frequentist evaluation procedure for significance tests, that should also be accepted by a classical statistician. This procedure likewise indicates some additional problems of significance tests. Such tests generally offer only incremental support of a hypothesis, although an absolute confirmation is necessary, and they overestimate positive results for small effects, since the confirmation of H in these cases is often rather marginal. This phenomenon leads in specific cases, for example, in cases of ESP-hypotheses, such as precognition, too easily to a significant confirmation. I will propose a method of how to evaluate and supplement significance tests so that we can avoid their epistemic deficits
Second-generation p-values: improved rigor, reproducibility, & transparency in statistical analyses
Verifying that a statistically significant result is scientifically
meaningful is not only good scientific practice, it is a natural way to control
the Type I error rate. Here we introduce a novel extension of the p-value - a
second-generation p-value - that formally accounts for scientific relevance and
leverages this natural Type I Error control. The approach relies on a
pre-specified interval null hypothesis that represents the collection of effect
sizes that are scientifically uninteresting or are practically null. The
second-generation p-value is the proportion of data-supported hypotheses that
are also null hypotheses. As such, second-generation p-values indicate when the
data are compatible with null hypotheses, or with alternative hypotheses, or
when the data are inconclusive. Moreover, second-generation p-values provide a
proper scientific adjustment for multiple comparisons and reduce false
discovery rates. This is an advance for environments rich in data, where
traditional p-value adjustments are needlessly punitive. Second-generation
p-values promote transparency, rigor and reproducibility of scientific results
by a priori specifying which candidate hypotheses are practically meaningful
and by providing a more reliable statistical summary of when the data are
compatible with alternative or null hypotheses.Comment: 29 pages, 29 page Supplemen
Two simple sufficient conditions for FDR control
We show that the control of the false discovery rate (FDR) for a multiple
testing procedure is implied by two coupled simple sufficient conditions. The
first one, which we call ``self-consistency condition'', concerns the algorithm
itself, and the second, called ``dependency control condition'' is related to
the dependency assumptions on the -value family. Many standard multiple
testing procedures are self-consistent (e.g. step-up, step-down or step-up-down
procedures), and we prove that the dependency control condition can be
fulfilled when choosing correspondingly appropriate rejection functions, in
three classical types of dependency: independence, positive dependency (PRDS)
and unspecified dependency. As a consequence, we recover earlier results
through simple and unifying proofs while extending their scope to several
regards: weighted FDR, -value reweighting, new family of step-up procedures
under unspecified -value dependency and adaptive step-up procedures. We give
additional examples of other possible applications. This framework also allows
for defining and studying FDR control for multiple testing procedures over a
continuous, uncountable space of hypotheses.Comment: Published in at http://dx.doi.org/10.1214/08-EJS180 the Electronic
Journal of Statistics (http://www.i-journals.org/ejs/) by the Institute of
Mathematical Statistics (http://www.imstat.org
- …