Systematic differences between Cochrane and non-Cochrane meta-analyses on the same topic: a matched pair analysis

BACKGROUND: Meta-analyses conducted via the Cochrane Collaboration adhere to strict methodological and reporting standards aiming to minimize bias, maximize transparency/reproducibility, and improve the accuracy of summarized data. Whether this results in differences in the results reported by meta-analyses on the same topic conducted outside the Cochrane Collaboration is an open question. METHODS: We conducted a matched-pair analysis with individual meta-analyses as the unit of analysis, comparing Cochrane and non-Cochrane reviews. Using meta-analyses from the cardiovascular literature, we identified pairs that matched on intervention and outcome. The pairs were contrasted in terms of how frequently results disagreed between the Cochrane and non-Cochrane reviews, whether effect sizes and statistical precision differed systematically, and how these differences related to the frequency of secondary citations of those reviews. RESULTS: Our search yielded 40 matched pairs of reviews. The two sets were similar in terms of which was first to publication, how many studies were included, and average sample sizes. The paired reviews included a total of 344 individual clinical trials: 111 (32.3%) studies were included only in a Cochrane review, 104 (30.2%) only in a non-Cochrane review, and 129 (37.5%) in both. Stated another way, 62.5% of studies were only included in one or the other meta-analytic literature. Overall, 37.5% of pairs had discrepant results. The most common involved shifts in the width of 95% confidence intervals that would yield a different statistical interpretation of the significance of results (7 pairs). Additionally, 20% differed in the direction of the summary effect size (5 pairs) or reported greater than a 2-fold difference in its magnitude (3 pairs). Non-Cochrane reviews reported significantly higher effect sizes (P < 0.001) and lower precision (P < 0.001) than matched Cochrane reviews. Reviews reporting an effect size at least 2-fold greater than their matched pair were cited more frequently. CONCLUSIONS: Though results between topic-matched Cochrane and non-Cochrane reviews were quite similar, discrepant results were frequent, and the overlap of included studies was surprisingly low. Non-Cochrane reviews report larger effect sizes with lower precision than Cochrane reviews, indicating systematic differences, likely reflective of methodology, between the two types of reviews that could generate different interpretations of the interventions under question

Systematic differences between Cochrane and non-Cochrane meta-analyses on the same topic: a matched pair analysis

BACKGROUND: Meta-analyses conducted via the Cochrane Collaboration adhere to strict methodological and reporting standards aiming to minimize bias, maximize transparency/reproducibility, and improve the accuracy of summarized data. Whether this results in differences in the results reported by meta-analyses on the same topic conducted outside the Cochrane Collaboration is an open question. METHODS: We conducted a matched-pair analysis with individual meta-analyses as the unit of analysis, comparing Cochrane and non-Cochrane reviews. Using meta-analyses from the cardiovascular literature, we identified pairs that matched on intervention and outcome. The pairs were contrasted in terms of how frequently results disagreed between the Cochrane and non-Cochrane reviews, whether effect sizes and statistical precision differed systematically, and how these differences related to the frequency of secondary citations of those reviews. RESULTS: Our search yielded 40 matched pairs of reviews. The two sets were similar in terms of which was first to publication, how many studies were included, and average sample sizes. The paired reviews included a total of 344 individual clinical trials: 111 (32.3%) studies were included only in a Cochrane review, 104 (30.2%) only in a non-Cochrane review, and 129 (37.5%) in both. Stated another way, 62.5% of studies were only included in one or the other meta-analytic literature. Overall, 37.5% of pairs had discrepant results. The most common involved shifts in the width of 95% confidence intervals that would yield a different statistical interpretation of the significance of results (7 pairs). Additionally, 20% differed in the direction of the summary effect size (5 pairs) or reported greater than a 2-fold difference in its magnitude (3 pairs). Non-Cochrane reviews reported significantly higher effect sizes (P < 0.001) and lower precision (P < 0.001) than matched Cochrane reviews. Reviews reporting an effect size at least 2-fold greater than their matched pair were cited more frequently. CONCLUSIONS: Though results between topic-matched Cochrane and non-Cochrane reviews were quite similar, discrepant results were frequent, and the overlap of included studies was surprisingly low. Non-Cochrane reviews report larger effect sizes with lower precision than Cochrane reviews, indicating systematic differences, likely reflective of methodology, between the two types of reviews that could generate different interpretations of the interventions under question

Association between different classifications of discrepant results in matched pairs of meta-analyses and the number of times those analyses were cited elsewhere in the published literature.

<p>This figure reports the number of times that Cochrane and non-Cochrane reviews were cited by other articles in the medical literature using the bibliometric feature in Google Scholar. Each pair of box and whisker plots corresponds to a given category of reviews. The first pair lists pairs of reviews that were concordant according to our definitions, meaning that the results of the contrasted analyses agreed. The next three sets of pairs reflect the three different patterns of discrepant results. These were discrepancies based on: shifts in width of confidence intervals that yield a different interpretation of the significant of the effect size (pair 2); instances where one review reported an effect size at least twice that of its match (pair 3); and instances where the effect size reverses (pair 4). To note, the numbers of subsequent citations is quite similar between the Cochrane and non-Cochrane pairs except in the case of discrepancies around the magnitude of the effect size. In those cases, the reviews reporting the larger effect sizes were cited far more often than those reviews reporting the smaller effect size.</p

Summary effect sizes for matched Cochrane and non-Cochrane paired meta-analyses.

<p>The figure presents Forest plots of effect sizes with 95% confidence intervals for each pair of Cochrane (left) and non-Cochrane (right) reviews. Each of the 40 matched pairs has been sorted based on effect size from the Cochrane review in ascending order of effect size.</p

Systematic differences between the Cochrane and non-Cochrane matched meta-analyses, in terms of a) natural log of effect size, and b) standard error of effect size.

<p>This figure regresses on a natural log scale pairs of Cochrane and non-Cochrane reviews in terms of effect size (Fig 3a) and standard error (Fig 3b). Each point on the scatter plot represents the intersection point of a Cochrane review with its matched pair in the non-Cochrane literature. In both cases, using T and F tests, the relationships are strongly correlated. However, in both, the slope of the line reveal that, on average, non-Cochrane reviews report slightly larger effect sizes but with larger standard errors (i.e., lower precision) than their matched Cochrane review.</p