The concept of multiplicity, conducting multiple statistical significance tests in one study, has pervaded primary research over the last 7 decades (Hochberg & Tamhane, 1987). This continued discussion was due to the fact that multiplicity increases the probability of committing a Type 1 error (i.e., deriving a false conclusion). Little attention has been paid, unfortunately, to multiplicity in meta-analysis (Tendal, Nuesch, Higgins, Juni, & Gotzsche, 2011) and calls have been made for meta-analysis methodologists to address this critical issue (Bender et al., 2008). As such, the purpose and significance of this project was to answer these calls by formally quantifying the multiplicity of statistical test in meta-analyses published within education and psychology literature, and to ameliorate the problem of multiplicity errors through the advancement of Type 1 error corrections for meta-analyses.
To accomplish this goal, this project screened all citations from Psychological Bulletin and Review of Educational Research from 1986-2011 for quantitative meta-analyses. From the citations that met inclusion criteria, 130 articles were randomly selected to code.
The results revealed an alarmingly high number of statistical tests used per study (μ = 70.82, σ = 94.2, M = 46.5). A major contributor to the number of statistical tests utilized was the number of independent syntheses; the average study conducted 12.72 independent syntheses (σ = 21.26, M = 5.0). A multiple regression model predicting the number of statistical tests used per study found that the date of publication, number of studies included in the review, and the number of independent syntheses per review all were linear predictors. A second phase of the project purposively selected four reviews to investigate the potential use of Type 1 error corrections. The results provided by the review authors were compared to the results using the statistical corrections. Using the statistical corrections, an average of 3.33 conclusions would require modification.
This project\u27s results indicated a community of researchers becoming more reliant on statistical significance testing while simultaneously ignoring the consequences of multiplicity. Failure to prevent further reliance on statistical significance testing in meta-analysis has the potential to prorogate the progress of cumulative science
