Evolution of Heterogeneity (I²) Estimates and Their 95% Confidence Intervals in Large Meta-Analyses

<div><h3>Background</h3><p>Assessment of heterogeneity is essential in systematic reviews and meta-analyses of clinical trials. The most commonly used heterogeneity measure, <em>I²</em>, provides an estimate of the proportion of variability in a meta-analysis that is explained by differences between the included trials rather than by sampling error. Recent studies have raised concerns about the reliability of <em>I²</em> estimates, due to their dependence on the precision of included trials and time-dependent biases. Authors have also advocated use of 95% confidence intervals (CIs) to express the uncertainty associated with <em>I²</em> estimates. However, no previous studies have explored how many trials and events are required to ensure stable and reliable <em>I²</em> estimates, or how 95% CIs perform as evidence accumulates.</p> <h3>Methodology/Principal Findings</h3><p>To assess the stability and reliability of <em>I²</em> estimates and their 95% CIs, in relation to the cumulative number of trials and events in meta-analysis, we looked at 16 large Cochrane meta-analyses - each including a sufficient number of trials and events to reliably estimate <em>I²</em> - and monitored the <em>I²</em> estimates and their 95% CIs for each year of publication. In 10 of the 16 meta-analyses, the <em>I²</em> estimates fluctuated more than 40% over time. The median number of events and trials required before the cumulative <em>I²</em> estimates stayed within +/−20% of the final <em>I²</em> estimate was 467 and 11. No major fluctuations were observed after 500 events and 14 trials. The 95% confidence intervals provided good coverage over time.</p> <h3>Conclusions/Significance</h3><p><em>I²</em> estimates need to be interpreted with caution when the meta-analysis only includes a limited number of events or trials. Confidence intervals for <em>I²</em> estimates provide good coverage as evidence accumulates, and are thus valuable for reflecting the uncertainty associated with estimating <em>I²</em>.</p> </div

Fluctuation spans of <i>I²</i> values and the number of trials and events required to become stabile.

<p>Fluctuation spans of <i>I²</i> values and the number of trials and events required to become stabile.</p

Presents the evolution of the cumulative <i>I²</i> estimates and their associated 95% confidence intervals (CIs) over the accumulation of trials in meta-analyses (1) to (8).

<p>The cumulative <i>I²</i> are represented by the dot-dashed line ( ), and their associated cumulative 95% CIs are represented by the dotted lines ( ).</p

Presents the evolution of the cumulative <i>I²</i> estimates and their associated 95% confidence intervals (CIs) over the accumulation of trials in meta-analyses (9) to (16).

<p>The cumulative <i>I²</i> are represented by the dot-dashed line ( ), and their associated cumulative 95% CIs are represented by the dotted lines ( ).</p

Matrix of assumptions about missing participant data respectively in intervention and control arms of a trial.

<p>Assumptions of incidence among participants with missing data in the intervention arm typically decrease going from right (100%) to left. Assumptions of incidence among participants with missing data in the control arm typically decrease going from bottom (100%) to top. Assumptions shaded in green take into account the incidence rates in all trials included in the systematic review, and not only the trial under consideration. Assumptions shaded in yellow take into account the incidence rates within the trial under consideration. RILTFU/FU can have different values in the control and intervention groups respectively. Assumptions shaded in orange are extreme and typically implausible.</p

Presents the evolution of the cumulative <i>I²</i> estimates and their associated 95% confidence intervals (CIs) over the accumulation of events in meta-analyses (1) to (8).

<p>The cumulative <i>I²</i> are represented by the dot-dashed line ( ), and their associated cumulative 95% CIs are represented by the dotted lines ( ).</p

Analytic methods to account for participants with missing data.

¶<p><i>y</i> and <i>z</i> refer to RI_LTFU/FU in the intervention and control arm respectively: <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0057132#pone.0057132-Akl2" target="_blank">[6]</a>.</p>*<p>This is a special case of RI_LTFU/FU method where <i>y</i> = <i>z</i>  = 1.</p

Status of participants and their outcome data at both the trial and systematic review levels.

<p> † It is possible that the investigators collected but did not report the data. * Data analysis might have included assumptions about missing participant data. ITT =  intention to treat; CCA =  complete case analysis. § Refers to ineligible participants mistakenly randomized and meeting the conditions for appropriate exclusion (see text), and to participants with other reasons for appropriate exclusion (e.g., subsequently found not to have condition of interest, or never underwent a procedure for which the intervention is intended).</p

Flow of participants in a trial that excluded participants from analysis for missing participant data.

<p>Flow of participants in a trial that excluded participants from analysis for missing participant data.</p

Characteristics of the 16 included meta-analyses.

<p>Characteristics of the 16 included meta-analyses.</p