Meta-analysis of risk prediction studies

This thesis identifies and demonstrates the methodological challenges of meta-analysing risk prediction models using either aggregate data or individual patient data (IPD). Firstly, a systematic review of published breast cancer models is performed, to summarise their content and performance using aggregate data. It is found that models were not available for comparison. To address this issue, a systematic review is performed to examine articles that develop and/or validate a risk prediction model using IPD from multiple studies. This identifies that most articles only use the IPD for model development, and thus ignore external validation, and also ignore clustering of patients within studies. In response to these issues, IPD is obtained from an article which uses parathyroid hormone (PTH) assay (a continuous variable) to predict postoperative hypocalcaemia after thyroidectomy. It is shown that ignoring clustering is inappropriate, as it ignores potential between-study heterogeneity in discrimination and calibration performance. This dataset was also used to evaluate an imputation method for dealing with missing thresholds when IPD are unavailable, and the simulation results indicate the approach performs well, though further research is required. This thesis therefore makes a positive contribution towards meta-analysis of risk prediction models to improve clinical practice

Developing and validating risk prediction models in an individual participant data meta-analysis

BACKGROUND: Risk prediction models estimate the risk of developing future outcomes for individuals based on one or more underlying characteristics (predictors). We review how researchers develop and validate risk prediction models within an individual participant data (IPD) meta-analysis, in order to assess the feasibility and conduct of the approach. METHODS: A qualitative review of the aims, methodology, and reporting in 15 articles that developed a risk prediction model using IPD from multiple studies. RESULTS: The IPD approach offers many opportunities but methodological challenges exist, including: unavailability of requested IPD, missing patient data and predictors, and between-study heterogeneity in methods of measurement, outcome definitions and predictor effects. Most articles develop their model using IPD from all available studies and perform only an internal validation (on the same set of data). Ten of the 15 articles did not allow for any study differences in baseline risk (intercepts), potentially limiting their model’s applicability and performance in some populations. Only two articles used external validation (on different data), including a novel method which develops the model on all but one of the IPD studies, tests performance in the excluded study, and repeats by rotating the omitted study. CONCLUSIONS: An IPD meta-analysis offers unique opportunities for risk prediction research. Researchers can make more of this by allowing separate model intercept terms for each study (population) to improve generalisability, and by using ‘internal-external cross-validation’ to simultaneously develop and validate their model. Methodological challenges can be reduced by prospectively planned collaborations that share IPD for risk prediction

Breast Cancer Index is a predictive biomarker of treatment benefit and outcome from extended tamoxifen therapy: final analysis of the Trans-aTTom study

PURPOSE: The Breast Cancer Index (BCI) HOXB13/IL17BR (H/I) ratio predicts benefit from extended endocrine therapy in hormone receptor–positive (HR(+)) early-stage breast cancer. Here, we report the final analysis of the Trans-aTTom study examining BCI (H/I)'s predictive performance. EXPERIMENTAL DESIGN: BCI results were available for 2,445 aTTom trial patients. The primary endpoint of recurrence-free interval (RFI) and secondary endpoints of disease-free interval (DFI) and disease-free survival (DFS) were examined using Cox proportional hazards regression and log-rank test. RESULTS: Final analysis of the overall study population (N = 2,445) did not show a significant improvement in RFI with extended tamoxifen [HR, 0.90; 95% confidence interval (CI), 0.69–1.16; P = 0.401]. Both the overall study population and N0 group were underpowered due to the low event rate in the N0 group. In a pre-planned analysis of the N(+) subset (N = 789), BCI (H/I)-High patients derived significant benefit from extended tamoxifen (9.7% absolute benefit: HR, 0.33; 95% CI, 0.14–0.75; P = 0.016), whereas BCI (H/I)-Low patients did not (−1.2% absolute benefit; HR, 1.11; 95% CI, 0.76–1.64; P = 0.581). A significant treatment-to-biomarker interaction was demonstrated on the basis of RFI, DFI, and DFS (P = 0.037, 0.040, and 0.025, respectively). BCI (H/I)-High patients remained predictive of benefit from extended tamoxifen in the N(+)/HER2(−) subgroup (9.4% absolute benefit: HR, 0.35; 95% CI, 0.15–0.81; P = 0.047). A three-way interaction evaluating BCI (H/I), treatment, and HER2 status was not statistically significant (P = 0.849). CONCLUSIONS: Novel findings demonstrate that BCI (H/I) significantly predicts benefit from extended tamoxifen in HR(+) N(+) patients with HER2(−) disease. Moreover, BCI (H/I) demonstrates significant treatment to biomarker interaction across survival outcomes

Summarising and validating test accuracy results across multiple studies for use in clinical practice

Following a meta-analysis of test accuracy studies, the translation of summary results into clinical practice is potentially problematic. The sensitivity, specificity and positive (PPV) and negative (NPV) predictive values of a test may differ substantially from the average meta-analysis findings, because of heterogeneity. Clinicians thus need more guidance: given the meta-analysis, is a test likely to be useful in new populations, and if so, how should test results inform the probability of existing disease (for a diagnostic test) or future adverse outcome (for a prognostic test)? We propose ways to address this. Firstly, following a meta-analysis, we suggest deriving prediction intervals and probability statements about the potential accuracy of a test in a new population. Secondly, we suggest strategies on how clinicians should derive post-test probabilities (PPV and NPV) in a new population based on existing meta-analysis results and propose a cross-validation approach for examining and comparing their calibration performance. Application is made to two clinical examples. In the first example, the joint probability that both sensitivity and specificity will be >80% in a new population is just 0.19, because of a low sensitivity. However, the summary PPV of 0.97 is high and calibrates well in new populations, with a probability of 0.78 that the true PPV will be at least 0.95. In the second example, post-test probabilities calibrate better when tailored to the prevalence in the new population, with cross-validation revealing a probability of 0.97 that the observed NPV will be within 10% of the predicted NPV

Mutational analysis of disease relapse in patients allografted for acute myeloid leukemia

Disease relapse is the major cause of treatment failure after allogeneic stem cell transplantation (allo-SCT) in acute myeloid leukemia (AML). To identify AML-associated genes prognostic of AML relapse post–allo-SCT, we resequenced 35 genes in 113 adults at diagnosis, 49 of whom relapsed. Two hundred sixty-two mutations were detected in 102/113 (90%) patients. An increased risk of relapse was observed in patients with mutations in WT1 (P = .018), DNMT3A (P = .045), FLT3 ITD (P = .071), and TP53 (P = .06), whereas mutations in IDH1 were associated with a reduced risk of disease relapse (P = .018). In 29 patients, we additionally compared mutational profiles in bone marrow at diagnosis and relapse to study changes in clonal structure at relapse. In 13/29 patients, mutational profiles altered at relapse. In 9 patients, mutations present at relapse were not detected at diagnosis. In 15 patients, additional available pre–allo-SCT samples demonstrated that mutations identified posttransplant but not at diagnosis were detectable immediately prior to transplant in 2 of 15 patients. Taken together, these observations, if confirmed in larger studies, have the potential to inform the design of novel strategies to reduce posttransplant relapse highlighting the potential importance of post–allo-SCT interventions with a broad antitumor specificity in contrast to targeted therapies based on mutational profile at diagnosis

Summarising and validating test accuracy results across multiple studies for use in clinical practice

Following a meta-analysis of test accuracy studies, the translation of summary results into clinical practice is potentially problematic. The sensitivity, specificity and positive (PPV) and negative (NPV) predictive values of a test may differ substantially from the average meta-analysis findings, because of heterogeneity. Clinicians thus need more guidance: given the meta-analysis, is a test likely to be useful in new populations, and if so, how should test results inform the probability of existing disease (for a diagnostic test) or future adverse outcome (for a prognostic test)? We propose ways to address this. Firstly, following a meta-analysis, we suggest deriving prediction intervals and probability statements about the potential accuracy of a test in a new population. Secondly, we suggest strategies on how clinicians should derive post-test probabilities (PPV and NPV) in a new population based on existing meta-analysis results and propose a cross-validation approach for examining and comparing their calibration performance. Application is made to two clinical examples. In the first example, the joint probability that both sensitivity and specificity will be >80% in a new population is just 0.19, because of a low sensitivity. However, the summary PPV of 0.97 is high and calibrates well in new populations, with a probability of 0.78 that the true PPV will be at least 0.95. In the second example, post-test probabilities calibrate better when tailored to the prevalence in the new population, with cross-validation revealing a probability of 0.97 that the observed NPV will be within 10% of the predicted NPV