Estimating interactions and subgroup-specific treatment effects in meta-analysis without aggregation bias: A within-trial framework

Estimation of within-trial interactions in meta-analysis is crucial for reliable assessment of how treatment effects vary across participant subgroups. However, current methods have various limitations. Patients, clinicians and policy-makers need reliable estimates of treatment effects within specific covariate subgroups, on relative and absolute scales, in order to target treatments appropriately - which estimation of an interaction effect does not in itself provide. Also, the focus has been on covariates with only two subgroups, and may exclude relevant data if only a single subgroup is reported. Therefore, in this article we further develop the "within-trial" framework by providing practical methods to (1) estimate within-trial interactions across two or more subgroups; (2) estimate subgroup-specific ("floating") treatment effects that are compatible with the within-trial interactions and make maximum use of available data; and (3) clearly present this data using novel implementation of forest plots. We described the steps involved and apply the methods to two examples taken from previously published meta-analyses, and demonstrate a straightforward implementation in Stata based upon existing code for multivariate meta-analysis. We discuss how the within-trial framework and plots can be utilised with aggregate (or "published") source data, as well as with individual participant data, to effectively demonstrate how treatment effects differ across participant subgroups

Implications of analysing time-to-event outcomes as binary in meta-analysis: empirical evidence from the Cochrane Database of Systematic Reviews

BACKGROUND: Systematic reviews and meta-analysis of time-to-event outcomes are frequently published within the Cochrane Database of Systematic Reviews (CDSR). However, these outcomes are handled differently across meta-analyses. They can be analysed on the hazard ratio (HR) scale or can be dichotomized and analysed as binary outcomes using effect measures such as odds ratios (OR) or risk ratios (RR). We investigated the impact of reanalysing meta-analyses from the CDSR that used these different effect measures. METHODS: We extracted two types of meta-analysis data from the CDSR: either recorded in a binary form only ("binary"), or in binary form together with observed minus expected and variance statistics ("OEV"). We explored how results for time-to-event outcomes originally analysed as "binary" change when analysed using the complementary log-log (clog-log) link on a HR scale. For the data originally analysed as HRs ("OEV"), we compared these results to analysing them as binary on a HR scale using the clog-log link or using a logit link on an OR scale. RESULTS: The pooled HR estimates were closer to 1 than the OR estimates in the majority of meta-analyses. Important differences in between-study heterogeneity between the HR and OR analyses were also observed. These changes led to discrepant conclusions between the OR and HR scales in some meta-analyses. Situations under which the clog-log link performed better than logit link and vice versa were apparent, indicating that the correct choice of the method does matter. Differences between scales arise mainly when event probability is high and may occur via differences in between-study heterogeneity or via increased within-study standard error in the OR relative to the HR analyses. CONCLUSIONS: We identified that dichotomising time-to-event outcomes may be adequate for low event probabilities but not for high event probabilities. In meta-analyses where only binary data are available, the complementary log-log link may be a useful alternative when analysing time-to-event outcomes as binary, however the exact conditions need further exploration. These findings provide guidance on the appropriate methodology that should be used when conducting such meta-analyses

Measuring the impact of methodological research

Providing evidence of impact highlights the benefits of medical research to society. Such evidence is increasingly requested by research funders and commonly relies on citation analysis. However, other indicators may be more informative. Although frameworks to demonstrate the impact of clinical research have been reported, no complementary framework exists for methodological research. Therefore, we assessed the impact of methodological research projects conducted or completed between 2009 and 2012 at the UK Medical Research Council Clinical Trials Unit Hub for Trials Methodology Research Hub, with a view to developing an appropriate framework

Meta-analytical methods to identify who benefits most from treatments: daft, deluded, or deft approach?

Identifying which individuals beneft most from particular treatments or other interventions underpins so-called personalised or stratifed medicine. However, single trials are typically underpowered for exploring whether participant characteristics, such as age or disease severity, determine an individual's response to treatment. A meta-analysis of multiple trials, particularly one where individual participant data (IPD) are available, provides greater power to investigate interactions between participant characteristics (covariates) and treatment e?ects. We use a published IPD meta-analysis to illustrate three broad approaches used for testing such interactions. Based on another systematic review of recently published IPD meta-analyses, we also show that all three approaches can be applied to aggregate data as well as IPD. We also summarise which methods of analysing and presenting interactions are in current use, and describe their advantages and disadvantages. We recommend that testing for interactions using within-trials information alone (the def approach) becomes standard practice, alongside graphical presentation that directly visualises this

Comparison of aggregate and individual participant data approaches to meta-analysis of randomised trials : An observational study

BACKGROUND: It remains unclear when standard systematic reviews and meta-analyses that rely on published aggregate data (AD) can provide robust clinical conclusions. We aimed to compare the results from a large cohort of systematic reviews and meta-analyses based on individual participant data (IPD) with meta-analyses of published AD, to establish when the latter are most likely to be reliable and when the IPD approach might be required. METHODS AND FINDINGS: We used 18 cancer systematic reviews that included IPD meta-analyses: all of those completed and published by the Meta-analysis Group of the MRC Clinical Trials Unit from 1991 to 2010. We extracted or estimated hazard ratios (HRs) and standard errors (SEs) for survival from trial reports and compared these with IPD equivalents at both the trial and meta-analysis level. We also extracted or estimated the number of events. We used paired t tests to assess whether HRs and SEs from published AD differed on average from those from IPD. We assessed agreement, and whether this was associated with trial or meta-analysis characteristics, using the approach of Bland and Altman. The 18 systematic reviews comprised 238 unique trials or trial comparisons, including 37,082 participants. A HR and SE could be generated for 127 trials, representing 53% of the trials and approximately 79% of eligible participants. On average, trial HRs derived from published AD were slightly more in favour of the research interventions than those from IPD (HRAD to HRIPD ratio = 0.95, p = 0.007), but the limits of agreement show that for individual trials, the HRs could deviate substantially. These limits narrowed with an increasing number of participants (p < 0.001) or a greater number (p < 0.001) or proportion (p < 0.001) of events in the AD. On average, meta-analysis HRs from published AD slightly tended to favour the research interventions whether based on fixed-effect (HRAD to HRIPD ratio = 0.97, p = 0.088) or random-effects (HRAD to HRIPD ratio = 0.96, p = 0.044) models, but the limits of agreement show that for individual meta-analyses, agreement was much more variable. These limits tended to narrow with an increasing number (p = 0.077) or proportion of events (p = 0.11) in the AD. However, even when the information size of the AD was large, individual meta-analysis HRs could still differ from their IPD equivalents by a relative 10% in favour of the research intervention to 5% in favour of control. We utilised the results to construct a decision tree for assessing whether an AD meta-analysis includes sufficient information, and when estimates of effects are most likely to be reliable. A lack of power at the meta-analysis level may have prevented us identifying additional factors associated with the reliability of AD meta-analyses, and we cannot be sure that our results are generalisable to all outcomes and effect measures. CONCLUSIONS: In this study we found that HRs from published AD were most likely to agree with those from IPD when the information size was large. Based on these findings, we provide guidance for determining systematically when standard AD meta-analysis will likely generate robust clinical conclusions, and when the IPD approach will add considerable value

Use of multiple covariates in assessing treatment-effect modifiers: A methodological review of individual participant data meta-analyses

Individual participant data (IPD) meta-analyses of randomised trials are considered a reliable way to assess participant-level treatment effect modifiers but may not make the best use of the available data. Traditionally, effect modifiers are explored one covariate at a time, which gives rise to the possibility that evidence of treatment-covariate interaction may be due to confounding from a different, related covariate. We aimed to evaluate current practice when estimating treatment-covariate interactions in IPD meta-analysis, specifically focusing on involvement of additional covariates in the models. We reviewed 100 IPD meta-analyses of randomised trials, published between 2015 and 2020, that assessed at least one treatment-covariate interaction. We identified four approaches to handling additional covariates: (1) Single interaction model (unadjusted): No additional covariates included (57/100 IPD meta-analyses); (2) Single interaction model (adjusted): Adjustment for the main effect of at least one additional covariate (35/100); (3) Multiple interactions model: Adjustment for at least one two-way interaction between treatment and an additional covariate (3/100); and (4) Three-way interaction model: Three-way interaction formed between treatment, the additional covariate and the potential effect modifier (5/100). IPD is not being utilised to its fullest extent. In an exemplar dataset, we demonstrate how these approaches lead to different conclusions. Researchers should adjust for additional covariates when estimating interactions in IPD meta-analysis providing they adjust their main effects, which is already widely recommended. Further, they should consider whether more complex approaches could provide better information on who might benefit most from treatments, improving patient choice and treatment policy and practice

Using individual participant data to improve network meta-analysis projects.

A network meta-analysis combines the evidence from existing randomised trials about the comparative efficacy of multiple treatments. It allows direct and indirect evidence about each comparison to be included in the same analysis, and provides a coherent framework to compare and rank treatments. A traditional network meta-analysis uses aggregate data (eg, treatment effect estimates and standard errors) obtained from publications or trial investigators. An alternative approach is to obtain, check, harmonise and meta-analyse the individual participant data (IPD) from each trial. In this article, we describe potential advantages of IPD for network meta-analysis projects, emphasising five key benefits: (1) improving the quality and scope of information available for inclusion in the meta-analysis, (2) examining and plotting distributions of covariates across trials (eg, for potential effect modifiers), (3) standardising and improving the analysis of each trial, (4) adjusting for prognostic factors to allow a network meta-analysis of conditional treatment effects and (5) including treatment-covariate interactions (effect modifiers) to allow relative treatment effects to vary by participant-level covariate values (eg, age, baseline depression score). A running theme of all these benefits is that they help examine and reduce heterogeneity (differences in the true treatment effect between trials) and inconsistency (differences in the true treatment effect between direct and indirect evidence) in the network. As a consequence, an IPD network meta-analysis has the potential for more precise, reliable and informative results for clinical practice and even allows treatment comparisons to be made for individual patients and targeted populations conditional on their particular characteristics

Association between tocilizumab, sarilumab and all-cause mortality at 28 days in hospitalised patients with COVID-19:A network meta-analysis

BACKGROUND: A recent prospective meta-analysis demonstrated that interleukin-6 antagonists are associated with lower all-cause mortality in hospitalised patients with COVID-19, compared with usual care or placebo. However, emerging evidence suggests that clinicians are favouring the use of tocilizumab over sarilumab. A new randomised comparison of these agents from the REMAP-CAP trial shows similar effects on in-hospital mortality. Therefore, we initiated a network meta-analysis, to estimate pairwise associations between tocilizumab, sarilumab and usual care or placebo with 28-day mortality, in COVID-19 patients receiving concomitant corticosteroids and ventilation, based on all available direct and indirect evidence. METHODS: Eligible trials randomised hospitalised patients with COVID-19 that compared tocilizumab or sarilumab with usual care or placebo in the prospective meta-analysis or that directly compared tocilizumab with sarilumab. Data were restricted to patients receiving corticosteroids and either non-invasive or invasive ventilation at randomisation. Pairwise associations between tocilizumab, sarilumab and usual care or placebo for all-cause mortality 28 days after randomisation were estimated using a frequentist contrast-based network meta-analysis of odds ratios (ORs), implementing multivariate fixed-effects models that assume consistency between the direct and indirect evidence. FINDINGS: One trial (REMAP-CAP) was identified that directly compared tocilizumab with sarilumab and supplied results on all-cause mortality at 28-days. This network meta-analysis was based on 898 eligible patients (278 deaths) from REMAP-CAP and 3710 eligible patients from 18 trials (1278 deaths) from the prospective meta-analysis. Summary ORs were similar for tocilizumab [0·82 [0·71–0·95, p = 0·008]] and sarilumab [0·80 [0·61–1·04, p = 0·09]] compared with usual care or placebo. The summary OR for 28-day mortality comparing tocilizumab with sarilumab was 1·03 [95%CI 0·81–1·32, p = 0·80]. The p-value for the global test of inconsistency was 0·28. CONCLUSIONS: Administration of either tocilizumab or sarilumab was associated with lower 28-day all-cause mortality compared with usual care or placebo. The association is not dependent on the choice of interleukin-6 receptor antagonist

The INVEST project: investigating the use of evidence synthesis in the design and analysis of clinical trials.

BACKGROUND: When designing and analysing clinical trials, using previous relevant information, perhaps in the form of evidence syntheses, can reduce research waste. We conducted the INVEST (INVestigating the use of Evidence Synthesis in the design and analysis of clinical Trials) survey to summarise the current use of evidence synthesis in trial design and analysis, to capture opinions of trialists and methodologists on such use, and to understand any barriers. METHODS: Our sampling frame was all delegates attending the International Clinical Trials Methodology Conference in November 2015. Respondents were asked to indicate (1) their views on the use of evidence synthesis in trial design and analysis, (2) their own use during the past 10 years and (3) the three greatest barriers to use in practice. RESULTS: Of approximately 638 attendees of the conference, 106 (17%) completed the survey, half of whom were statisticians. Support was generally high for using a description of previous evidence, a systematic review or a meta-analysis in trial design. Generally, respondents did not seem to be using evidence syntheses as often as they felt they should. For example, only 50% (42/84 relevant respondents) had used a meta-analysis to inform whether a trial is needed compared with 74% (62/84) indicating that this is desirable. Only 6% (5/81 relevant respondents) had used a value of information analysis to inform sample size calculations versus 22% (18/81) indicating support for this. Surprisingly large numbers of participants indicated support for, and previous use of, evidence syntheses in trial analysis. For example, 79% (79/100) of respondents indicated that external information about the treatment effect should be used to inform aspects of the analysis. The greatest perceived barrier to using evidence synthesis methods in trial design or analysis was time constraints, followed by a belief that the new trial was the first in the area. CONCLUSIONS: Evidence syntheses can be resource-intensive, but their use in informing the design, conduct and analysis of clinical trials is widely considered desirable. We advocate additional research, training and investment in resources dedicated to ways in which evidence syntheses can be undertaken more efficiently, offering the potential for cost savings in the long term

Which patients with metastatic hormone-sensitive prostate cancer benefit from docetaxel: a systematic review and meta-analysis of individual participant data from randomised trials

BACKGROUND: Adding docetaxel to androgen deprivation therapy (ADT) improves survival in patients with metastatic, hormone-sensitive prostate cancer, but uncertainty remains about who benefits most. We therefore aimed to obtain up-to-date estimates of the overall effects of docetaxel and to assess whether these effects varied according to prespecified characteristics of the patients or their tumours. METHODS: The STOPCAP M1 collaboration conducted a systematic review and meta-analysis of individual participant data. We searched MEDLINE (from database inception to March 31, 2022), Embase (from database inception to March 31, 2022), the Cochrane Central Register of Controlled Trials (from database inception to March 31, 2022), proceedings of relevant conferences (from Jan 1, 1990, to Dec 31, 2022), and ClinicalTrials.gov (from database inception to March 28, 2023) to identify eligible randomised trials that assessed docetaxel plus ADT compared with ADT alone in patients with metastatic, hormone-sensitive prostate cancer. Detailed and updated individual participant data were requested directly from study investigators or through relevant repositories. The primary outcome was overall survival. Secondary outcomes were progression-free survival and failure-free survival. Overall pooled effects were estimated using an adjusted, intention-to-treat, two-stage, fixed-effect meta-analysis, with one-stage and random-effects sensitivity analyses. Missing covariate values were imputed. Differences in effect by participant characteristics were estimated using adjusted two-stage, fixed-effect meta-analysis of within-trial interactions on the basis of progression-free survival to maximise power. Identified effect modifiers were also assessed on the basis of overall survival. To explore multiple subgroup interactions and derive subgroup-specific absolute treatment effects we used one-stage flexible parametric modelling and regression standardisation. We assessed the risk of bias using the Cochrane Risk of Bias 2 tool. This study is registered with PROSPERO, CRD42019140591. FINDINGS: We obtained individual participant data from 2261 patients (98% of those randomised) from three eligible trials (GETUG-AFU15, CHAARTED, and STAMPEDE trials), with a median follow-up of 72 months (IQR 55-85). Individual participant data were not obtained from two additional small trials. Based on all included trials and patients, there were clear benefits of docetaxel on overall survival (hazard ratio [HR] 0·79, 95% CI 0·70 to 0·88; p<0·0001), progression-free survival (0·70, 0·63 to 0·77; p<0·0001), and failure-free survival (0·64, 0·58 to 0·71; p<0·0001), representing 5-year absolute improvements of around 9-11%. The overall risk of bias was assessed to be low, and there was no strong evidence of differences in effect between trials for all three main outcomes. The relative effect of docetaxel on progression-free survival appeared to be greater with increasing clinical T stage (pinteraction=0·0019), higher volume of metastases (pinteraction=0·020), and, to a lesser extent, synchronous diagnosis of metastatic disease (pinteraction=0·077). Taking into account the other interactions, the effect of docetaxel was independently modified by volume and clinical T stage, but not timing. There was no strong evidence that docetaxel improved absolute effects at 5 years for patients with low-volume, metachronous disease (-1%, 95% CI -15 to 12, for progression-free survival; 0%, -10 to 12, for overall survival). The largest absolute improvement at 5 years was observed for those with high-volume, clinical T stage 4 disease (27%, 95% CI 17 to 37, for progression-free survival; 35%, 24 to 47, for overall survival). INTERPRETATION: The addition of docetaxel to hormone therapy is best suited to patients with poorer prognosis for metastatic, hormone-sensitive prostate cancer based on a high volume of disease and potentially the bulkiness of the primary tumour. There is no evidence of meaningful benefit for patients with metachronous, low-volume disease who should therefore be managed differently. These results will better characterise patients most and, importantly, least likely to gain benefit from docetaxel, potentially changing international practice, guiding clinical decision making, better informing treatment policy, and improving patient outcomes. FUNDING: UK Medical Research Council and Prostate Cancer UK