A call for cautious interpretation of meta-analytic reviews

Meta-analytic reviews collect available empirical studies on a specified domain and calculate the average effect of a factor. Educators as well as researchers exploring a new domain of inquiry may rely on the conclusions from meta-analytic reviews rather than reading multiple primary studies. This article calls for caution in this regard, because the outcome of a meta-analysis is determined by how effect sizes are calculated, how factors are defined, and how studies are selected for inclusion. Three recently published meta-analyses are re-examined to illustrate these issues. One illustrates the risk of conflating effect sizes from studies with different design features, another illustrates problems with delineating the variable of interest, with implications for cause-effect relations, and the third illustrates the challenge of determining the eligibility of candidate studies. Replication attempts yield outcomes that differ from the three original meta-analyses, suggesting that also conclusions drawn from meta-analyses need to be interpreted cautiously