Pitfalls of single-study external validation illustrated with a model predicting functional outcome after aneurysmal subarachnoid hemorrhage

Background: Prediction models are often externally validated with data from a single study or cohort. However, the interpretation of performance estimates obtained with single-study external validation is not as straightforward as assumed. We aimed to illustrate this by conducting a large number of external validations of a prediction model for functional outcome in subarachnoid hemorrhage (SAH) patients.Methods: We used data from the Subarachnoid Hemorrhage International Trialists (SAHIT) data repository (n = 11,931, 14 studies) to refit the SAHIT model for predicting a dichotomous functional outcome (favorable versus unfavorable), with the (extended) Glasgow Outcome Scale or modified Rankin Scale score, at a minimum of three months after discharge. We performed leave-one-cluster-out cross-validation to mimic the process of multiple single-study external validations. Each study represented one cluster. In each of these validations, we assessed discrimination with Harrell’s c-statistic and calibration with calibration plots, the intercepts, and the slopes. We used random effects meta-analysis to obtain the (reference) mean performance estimates and between-study heterogeneity (I2-statistic). The influence of case-mix variation on discriminative performance was assessed with the model-based c-statistic and we fitted a “membership model” to obtain a gross estimate of transportability. Results: Across 14 single-study external validations, model performance was highly variable. The mean c-statistic was 0.74 (95%CI 0.70–0.78, range 0.52–0.84, I2 = 0.92), the mean intercept was -0.06 (95%CI -0.37–0.24, range -1.40–0.75, I2 = 0.97), and the mean slope was 0.96 (95%CI 0.78–1.13, range 0.53–1.31, I2 = 0.90). The decrease in discriminative performance was attributable to case-mix variation, between-study heterogeneity, or a combination of both. Incidentally, we observed poor generalizability or transportability of the model. Conclusions: We demonstrate two potential pitfalls in the interpretation of model performance with single-study external validation. With single-study external validation. (1) model performance is highly variable and depends on the choice of validation data and (2) no insight is provided into generalizability or transportability of the model that is needed to guide local implementation. As such, a single single-study external validation can easily be misinterpreted and lead to a false appreciation of the clinical prediction model. Cross-validation is better equipped to address these pitfalls.</p

Dynamics and prognostic value of serum neurofilament light chain in Guillain-Barré syndrome

Background: Neurofilament light chain (NfL) is a biomarker for axonal damage in several neurological disorders. We studied the longitudinal changes in serum NfL in patients with Guillain-Barré syndrome (GBS) in relation to disease severity, electrophysiological subtype, treatment response, and prognosis. Methods: We included patients with GBS who participated in a double-blind, randomised, placebo-controlled trial that evaluated the effects of a second course of intravenous immunoglobulin (IVIg) on clinical outcomes. Serum NfL levels were measured before initiation of treatment and at one, two, four, and twelve weeks using a Simoa HD-X Analyzer. Serum NfL dynamics were analysed using linear mixed-effects models. Logistic regression was employed to determine the associations of serum NfL with clinical outcome and the prognostic value of serum NfL after correcting for known prognostic markers.Findings: NfL levels were tested in serum from 281 patients. Serum NfL dynamics were associated with disease severity and electrophysiological subtype. Strong associations were found between high levels of serum NfL at two weeks and inability to walk unaided at four weeks (OR = 1.74, 95% CI = 1.27–2.45), and high serum NfL levels at four weeks and inability to walk unaided at 26 weeks (OR = 2.79, 95% CI = 1.72–4.90). Baseline serum NfL had the most significant prognostic value for ability to walk, independent of known predictors of outcome. The time to regain ability to walk unaided was significantly longer for patients with highest serum NfL levels at baseline (p = 0.0048) and week 2 (p &lt; 0.0001). No differences in serum NfL were observed between patients that received a second IVIg course vs. IVIg and placebo.Interpretation: Serum NfL levels are associated with disease severity, axonal involvement, and poor outcome in GBS. Serum NfL potentially represents a biomarker to monitor neuronal damage in GBS and an intermediate endpoint to evaluate the effects of treatment. </p

Comparative effectiveness of 6x R-CHOP21 versus 6x R-CHOP21 + 2 R for patients with advanced-stage diffuse large B-cell lymphoma

First-line treatment for advanced-stage diffuse large B-cell lymphoma (DLBCL) typically involves 6x R-CHOP21 or 6x R-CHOP21 with two additional rituximab administrations (6x R-CHOP21 + 2 R). In contemporary practice, this treatment choice might be guided by interim PET scan results. This nationwide, population-based study investigates the comparative effectiveness of these treatment regimens in an era where interim PET-guided treatment decisions were not standard practice. Utilizing the Netherlands Cancer Registry, we identified 1577 adult patients diagnosed with advanced-stage DLBCL between 2014–2018 who completed either 6x R-CHOP21 (43%) or 6x R-CHOP21 + 2 R (57%). We used propensity scores to assess differences in event-free survival (EFS) and overall survival (OS). At five years, EFS (hazard ratio of 6x R-CHOP21 + 2 R versus 6x R-CHOP21 [HR] = 0.89; 95% confidence interval [CI], 0.72–1.09) and OS (HR = 0.93; 95% CI, 0.73–1.18) were not significantly different between both regimens. In exploratory risk-stratified analysis according to the International Prognostic Index (IPI), high-IPI patients (i.e., scores of 4-5) benefit most from 6x R-CHOP21 + 2 R (5-year absolute risk difference of EFS = 16.8%; 95% CI, −0.4%−34.1% and OS = 12.1%; 95% CI, −5.4–29.6%). Collectively, this analysis reveals no significant differences on average in EFS and OS between the two treatments. However, the potential benefits for high-risk patients treated with 6x R-CHOP21 + 2 R underscore the need for future research.</p

Number of life-years lost at the time of diagnosis and several years post-diagnosis in patients with solid malignancies: a population-based study in the Netherlands, 1989–2019Research in context

Summary: Background: Loss of life expectancy (LOLE) may provide more intuitive information on the impact of cancer than relative survival over a fixed time horizon (e.g., 5-year relative survival). We aimed to assess the evolution of the LOLE using a nationwide, population-based cohort including patients diagnosed with one of 17 most frequent solid malignancies. Methods: From the Netherlands Cancer Registry, we selected adult patients diagnosed with one of the 17 most frequent solid malignancies in the Netherlands during 1989–2019, with survival follow-up until 2022. We used flexible parametric survival models to estimate the LOLE at diagnosis and the LOLE after surviving several years post-diagnosis (conditional LOLE; CLOLE) by cancer type, calendar year, age, sex, and disease stage. Findings: For all cancers combined, the LOLE consistently decreased from 1989 to 2019. This decrease was most pronounced for males with prostate cancer (e.g., from 6.9 [95% confidence interval [CI], 6.7–7.1] to 2.7 [95% CI, 2.5–3.0] for 65-year-olds) and females with breast cancer (e.g., from 6.6 [95% CI, 6.4–6.7] to 1.9 [95% CI, 1.8–2.0] for 65-year-olds). The LOLE among patients with cancers of the head and neck or the central nervous system remained constant over time. Overall, the CLOLE showed that the life years lost among patients with cancer decreased with each additional year survived post-diagnosis. For example, the LOLE at diagnosis for 65-year-old females diagnosed with breast cancer in 2019 was 1.9 [95% CI, 1.8–2.0] compared with 1.7 [95% CI, 1.6–1.8], 1.0 [95% CI, 0.9–1.1], and 0.5 [95% CI, 0.5–0.6] when surviving one, five, and ten years post-diagnosis, respectively. Estimates for other combinations of patient and tumour characteristics are available in a publicly available web-based application. Interpretation: Our findings suggested that the evolution of LOLE substantially varies across cancer type, age, and disease stage. LOLE estimates help patients better understand the impact of their specific cancer diagnosis on their life expectancy. Funding: None

The evolution of the loss of life expectancy in patients with chronic myeloid leukaemia: a population-based study in the Netherlands, 1989–2018

Studies on the conditional life expectancy of patients with chronic myeloid leukaemia (CML) are lacking. Using data from the Netherlands Cancer Registry, we examined the life expectancy of patients with CML in the Netherlands diagnosed during 1989–2018. As of the early 2010s, the life expectancy of patients with CML who survived several years after diagnosis came narrowly close to the general population’s life expectancy, regardless of age. This finding can essentially be ascribed to the introduction and broader application of tyrosine kinase inhibitors (TKIs) and provide optimism to patients with CML who can look forward to a near-normal life expectancy in a modern TKI era

Development and validation of a novel model to predict recurrence-free survival and melanoma-specific survival after sentinel lymph node biopsy in patients with melanoma:an international, retrospective, multicentre analysis

Background: The introduction of adjuvant systemic treatment for patients with high-risk melanomas necessitates accurate staging of disease. However, inconsistencies in outcomes exist between disease stages as defined by the American Joint Committee on Cancer (8th edition). We aimed to develop a tool to predict patient-specific outcomes in people with melanoma rather than grouping patients according to disease stage. Methods: Patients older than 13 years with confirmed primary melanoma who underwent sentinel lymph node biopsy (SLNB) between Oct 29, 1997, and Nov 11, 2013, at four European melanoma centres (based in Berlin, Germany; Amsterdam and Rotterdam, the Netherlands; and Warsaw, Poland) were included in the development cohort. Potential predictors of recurrence-free and melanoma-specific survival assessed were sex, age, presence of ulceration, primary tumour location, histological subtype, Breslow thickness, sentinel node status, number of sentinel nodes removed, maximum diameter of the largest sentinel node metastasis, and Dewar classification. A prognostic model and nomogram were developed to predict 5-year recurrence-free survival on a continuous scale in patients with stage pT1b or higher melanomas. This model was also calibrated to predict melanoma-specific survival. Model performance was assessed by discrimination (area under the time-dependent receiver operating characteristics curve [AUC]) and calibration. External validation was done in a cohort of patients with primary melanomas who underwent SLNB between Jan 30, 1997, and Dec 12, 2013, at the Melanoma Institute Australia (Sydney, NSW, Australia).Findings: The development cohort consisted of 4071 patients, of whom 2075 (51%) were female and 1996 (49%) were male. 889 (22%) had sentinel node-positive disease and 3182 (78%) had sentinel node-negative disease. The validation cohort comprised 4822 patients, of whom 1965 (41%) were female and 2857 (59%) were male. 891 (18%) had sentinel node-positive disease and 3931 (82%) had sentinel node-negative disease. Median follow-up was 4·8 years (IQR 2·3–7·8) in the development cohort and 5·0 years (2·2–8·9) in the validation cohort. In the development cohort, 5-year recurrence-free survival was 73·5% (95% CI 72·0–75·1) and 5-year melanoma-specific survival was 86·5% (85·3–87·8). In the validation cohort, the corresponding estimates were 66·1% (64·6–67·7) and 83·3% (82·0–84·6), respectively. The final model contained six prognostic factors: sentinel node status, Breslow thickness, presence of ulceration, age at SLNB, primary tumour location, and maximum diameter of the largest sentinel node metastasis. In the development cohort, for the model's prediction of recurrence-free survival, the AUC was 0·80 (95% CI 0·78–0·81); for prediction of melanoma-specific survival, the AUC was 0·81 (0·79–0·84). External validation showed good calibration for both outcomes, with AUCs of 0·73 (0·71–0·75) and 0·76 (0·74–0·78), respectively.Interpretation: Our prediction model and nomogram accurately predicted patient-specific risk probabilities for 5-year recurrence-free and melanoma-specific survival. These tools could have important implications for clinical decision making when considering adjuvant treatments in patients with high-risk melanomas. </p

Development and validation of a novel model to predict recurrence-free survival and melanoma-specific survival after sentinel lymph node biopsy in patients with melanoma:an international, retrospective, multicentre analysis

Background: The introduction of adjuvant systemic treatment for patients with high-risk melanomas necessitates accurate staging of disease. However, inconsistencies in outcomes exist between disease stages as defined by the American Joint Committee on Cancer (8th edition). We aimed to develop a tool to predict patient-specific outcomes in people with melanoma rather than grouping patients according to disease stage. Methods: Patients older than 13 years with confirmed primary melanoma who underwent sentinel lymph node biopsy (SLNB) between Oct 29, 1997, and Nov 11, 2013, at four European melanoma centres (based in Berlin, Germany; Amsterdam and Rotterdam, the Netherlands; and Warsaw, Poland) were included in the development cohort. Potential predictors of recurrence-free and melanoma-specific survival assessed were sex, age, presence of ulceration, primary tumour location, histological subtype, Breslow thickness, sentinel node status, number of sentinel nodes removed, maximum diameter of the largest sentinel node metastasis, and Dewar classification. A prognostic model and nomogram were developed to predict 5-year recurrence-free survival on a continuous scale in patients with stage pT1b or higher melanomas. This model was also calibrated to predict melanoma-specific survival. Model performance was assessed by discrimination (area under the time-dependent receiver operating characteristics curve [AUC]) and calibration. External validation was done in a cohort of patients with primary melanomas who underwent SLNB between Jan 30, 1997, and Dec 12, 2013, at the Melanoma Institute Australia (Sydney, NSW, Australia).Findings: The development cohort consisted of 4071 patients, of whom 2075 (51%) were female and 1996 (49%) were male. 889 (22%) had sentinel node-positive disease and 3182 (78%) had sentinel node-negative disease. The validation cohort comprised 4822 patients, of whom 1965 (41%) were female and 2857 (59%) were male. 891 (18%) had sentinel node-positive disease and 3931 (82%) had sentinel node-negative disease. Median follow-up was 4·8 years (IQR 2·3–7·8) in the development cohort and 5·0 years (2·2–8·9) in the validation cohort. In the development cohort, 5-year recurrence-free survival was 73·5% (95% CI 72·0–75·1) and 5-year melanoma-specific survival was 86·5% (85·3–87·8). In the validation cohort, the corresponding estimates were 66·1% (64·6–67·7) and 83·3% (82·0–84·6), respectively. The final model contained six prognostic factors: sentinel node status, Breslow thickness, presence of ulceration, age at SLNB, primary tumour location, and maximum diameter of the largest sentinel node metastasis. In the development cohort, for the model's prediction of recurrence-free survival, the AUC was 0·80 (95% CI 0·78–0·81); for prediction of melanoma-specific survival, the AUC was 0·81 (0·79–0·84). External validation showed good calibration for both outcomes, with AUCs of 0·73 (0·71–0·75) and 0·76 (0·74–0·78), respectively.Interpretation: Our prediction model and nomogram accurately predicted patient-specific risk probabilities for 5-year recurrence-free and melanoma-specific survival. These tools could have important implications for clinical decision making when considering adjuvant treatments in patients with high-risk melanomas. </p