External validation of 87 clinical prediction models supporting clinical decisions for breast cancer patients

Introduction: Numerous prediction models have been developed to support treatment-related decisions for breast cancer patients. External validation, a prerequisite for implementation in clinical practice, has been performed for only a few models. This study aims to externally validate published clinical prediction models using population-based Dutch data. Methods: Patient-, tumor- and treatment-related data were derived from the Netherlands Cancer Registry (NCR). Model performance was assessed using the area under the receiver operating characteristic curve (AUC), scaled Brier score, and model calibration. Net benefit across applicable risk thresholds was evaluated with decision curve analysis. Results: After assessing 922 models, 87 (9%) were included for validation. Models were excluded due to an incomplete model description (n = 262 (28%)), lack of required data (n = 521 (57%)), previously validated or developed with NCR data (n = 45 (5%)), or the associated NCR sample size was insufficient (n = 7 (1%)). The included models predicted survival (33 (38%) overall, 27 (31%) breast cancer-specific, and 3 (3%) other cause-specific), locoregional recurrence (n = 7 (8%)), disease free survival (n = 7 (8%)), metastases (n = 5 (6%)), lymph node involvement (n = 3 (3%)), pathologic complete response (n = 1 (1%)), and surgical margins (n = 1 (1%)). Seven models (8%) showed poor (AUC&lt;0.6), 39 (45%) moderate (AUC:0.6–0.7), 38 (46%) good (AUC:0.7–0.9), and 3 (3%) excellent (AUC≥0.9) discrimination. Using the scaled Brier score, worse performance than an uninformative model was found in 34 (39%) models. Conclusion: Comprehensive registry data supports broad validation of published prediction models. Model performance varies considerably in new patient populations, affirming the importance of external validation studies before applying models in clinical practice. Well performing models could be clinically useful in a Dutch setting after careful impact evaluation.</p

Developing, validating, and evaluating clinical prediction models in breast and prostate cancer

Clinical prediction models are statistical tools that can be used to estimate the probability of a patient to either have a specific outcome or to develop an outcome in time. This probability is estimated based on patient or disease-specific input variables. It provides insights into the diagnosis (e.g. disease status) or prognosis (e.g. 5-year survival probability) of a patient, and can subsequently be used to support (shared) decision-making regarding the optimal management of the disease. Prediction models are developed and evaluated using data from patients that can be classified in similar patient groups (e.g. diagnosed with estrogen receptor positive breast cancer), but with varying disease characteristics (e.g. tumor stage, treatment received, nodal involvement etc.). Before the available models are used to support in routine healthcare decision-making some challenges on the identification of currently existing models (accessibility), review of the quality of the models (transparency), assessment how well they perform on external validation (generalizability), and investigation of the potential benefit of recalibrating the validated models (updating). Subsequently, models showing adequate performance will be ready for implementation in clinical practice after clearly defined intended model use is described (interpretation), and the intended model use is substantiated by evidence regarding added value (impact assessment). In this thesis, multiple studies aiming to overcome the challenges are described using examples on breast and prostate cancer. Since breast and prostate cancer are among the top three most commonly diagnosed cancers in women and men, respectively, there is a large amount of data available to establish clinical prediction models for patients diagnosed with breast or prostate cancer. Currently available models for breast and prostate cancer are required to be critically assessed to demonstrate which models are valuable and which information is still lacking when used in Dutch care

The majority of 922 prediction models supporting breast cancer decision-making are at high risk of bias

Objectives To systematically review the currently available prediction models that may support treatment decision-making in breast cancer. Study Design and Setting Literature was systematically searched to identify studies reporting on development of prediction models aiming to support breast cancer treatment decision-making, published between January 2010 and December 2020. Quality and risk of bias were assessed using the Prediction model Risk Of Bias (ROB) Assessment Tool (PROBAST). Results After screening 20,460 studies, 534 studies were included, reporting on 922 models. The 922 models predicted: mortality (n = 417 45%), recurrence (n = 217, 24%), lymph node involvement (n = 141, 15%), adverse events (n = 58, 6%), treatment response (n = 56, 6%), or other outcomes (n = 33, 4%). In total, 285 models (31%) lacked a complete description of the final model and could not be applied to new patients. Most models (n = 878, 95%) were considered to contain high ROB. Conclusion A substantial overlap in predictor variables and outcomes between the models was observed. Most models were not reported according to established reporting guidelines or showed methodological flaws during the development and/or validation of the model. Further development of prediction models with thorough quality and validity assessment is an essential first step for future clinical application

Validation of the online prediction model CancerMath in the Dutch breast cancer population

Purpose CancerMath predicts the expected benefit of adjuvant systemic therapy on overall (OS) and breast cancer-specific survival (BCSS). Here, CancerMath was validated in Dutch breast cancer patients. Methods All operated women diagnosed with stage I–III primary invasive breast cancer in 2005 were identified from the Netherlands Cancer Registry. Calibration was assessed by comparing 5- and 10-year predicted and observed OS/BCSS using χ2 tests. A difference > 3% was considered as clinically relevant. Discrimination was assessed by area under the receiver operating characteristic (AUC) curves. Results Altogether, 8032 women were included. CancerMath underestimated 5- and 10-year OS by 2.2% and 1.9%, respectively. AUCs of 5- and 10-year OS were both 0.77. Divergence between predicted and observed OS was most pronounced in grade II, patients without positive nodes, tumours 1.01–2.00 cm, hormonal receptor positive disease and patients 60–69 years. CancerMath underestimated 5- and 10-year BCSS by 0.5% and 0.6%, respectively. AUCs were 0.78 and 0.73, respectively. No significant difference was found in any subgroup. Conclusion CancerMath predicts OS accurately for most patients with early breast cancer although outcomes should be interpreted with care in some subgroups. BCSS is predicted accurately in all subgroups. Therefore, CancerMath can reliably be used in (Dutch) clinical practice

External Validation of Models Predicting the Probability of Lymph Node Involvement in Prostate Cancer Patients

Background: Multiple statistical models predicting lymph node involvement (LNI) in prostate cancer (PCa) exist to support clinical decision-making regarding extended pelvic lymph node dissection (ePLND). Objective: To validate models predicting LNI in Dutch PCa patients. Design, setting, and participants: Sixteen prediction models were validated using a patient cohort of 1001 men who underwent ePLND. Patient characteristics included serum prostate specific antigen (PSA), cT stage, primary and secondary Gleason scores, number of biopsy cores taken, and number of positive biopsy cores. Outcome measurements and statistical analysis: Model performance was assessed using the area under the receiver operating characteristic curve (AUC). Calibration plots were used to visualize over- or underestimation by the models. Results and limitations: LNI was identified in 276 patients (28%). Patients with LNI had higher PSA, higher primary Gleason pattern, higher Gleason score, higher number of nodes harvested, higher number of positive biopsy cores, and higher cT stage compared to patients without LNI. Predictions generated by the 2012 Briganti nomogram (AUC 0.76) and the Memorial Sloan Kettering Cancer Center (MSKCC) web calculator (AUC 0.75) were the most accurate. Calibration had a decisive role in selecting the most accurate models because of overlapping confidence intervals for the AUCs. Underestimation of LNI probability in patients had a predicted probability of <20%. The omission of model updating was a limitation of the study. Conclusions: Models predicting LNI in PCa patients were externally validated in a Dutch patient cohort. The 2012 Briganti and MSKCC nomograms were identified as the most accurate prediction models available. Patient summary: In this report we looked at how well models were able to predict the risk of prostate cancer spreading to the pelvic lymph nodes. We found that two models performed similarly in predicting the most accurate probabilities. Nomograms developed by Briganti et al and the Memorial Sloan Kettering Cancer Center were best at predicting lymph node involvement in prostate cancer patients. These models support clinical decision-making on whether to perform pelvic lymph node dissection

External validation of the Memorial Sloan Kettering Cancer Centre and Briganti nomograms for the prediction of lymph node involvement of prostate cancer using clinical stage assessed by magnetic resonance imaging

Objectives: To evaluate the impact of using clinical stage assessed by multiparametric magnetic resonance imaging (mpMRI) on the performance of two established nomograms for the prediction of pelvic lymph node involvement (LNI) in patients with prostate cancer. Patients and Methods: Patients undergoing robot-assisted extended pelvic lymph node dissection (ePLND) from 2015 to 2019 at three teaching hospitals were retrospectively evaluated. Risk of LNI was calculated four times for each patient, using clinical tumour stage (T-stage) assessed by digital rectal examination (DRE) and by mpMRI, in the Memorial Sloan Kettering Cancer Centre (MSKCC; 2018) and Briganti (2012) nomograms. Discrimination (area under the curve [AUC]), calibration, and the net benefit of these four strategies were assessed and compared. Results: A total of 1062 patients were included, of whom 301 (28%) had histologically proven LNI. Using DRE T-stage resulted in AUCs of 0.71 (95% confidence interval [CI] 0.70–0.72) for the MSKCC and 0.73 (95% CI 0.72–0.74) for the Briganti nomogram. Using mpMRI T-stage, the AUCs were 0.72 (95% CI 0.71–0.73) for the MSKCC and 0.75 (95% CI 0.74–0.76) for the Briganti nomogram. mpMRI T-stage resulted in equivalent calibration compared with DRE T-stage. Combined use of mpMRI T-stage and the Briganti 2012 nomogram was shown to be superior in terms of AUC, calibration, and net benefit. Use of mpMRI T-stage led to increased sensitivity for the detection of LNI for all risk thresholds in both models, countered by a decreased specificity, compared with DRE T-stage. Conclusion: T-stage as assessed by mpMRI is an appropriate alternative for T-stage assessed by DRE to determine nomogram-based risk of LNI in patients with prostate cancer, and was associated with improved model performance of both the MSKCC 2018 and Briganti 2012 nomograms