External validations of cardiovascular clinical prediction models: a large-scale review of the literature

Background: There are many clinical prediction models (CPMs) available to inform treatment decisions for patients with cardiovascular disease. However, the extent to which they have been externally tested, and how well they generally perform has not been broadly evaluated. Methods: A SCOPUS citation search was run on March 22, 2017 to identify external validations of cardiovascular CPMs in the Tufts Predictive Analytics and Comparative Effectiveness CPM Registry. We assessed the extent of external validation, performance heterogeneity across databases, and explored factors associated with model performance, including a global assessment of the clinical relatedness between the derivation and validation data. Results: We identified 2030 external validations of 1382 CPMs. Eight hundred seven (58%) of the CPMs in the Registry have never been externally validated. On average, there were 1.5 validations per CPM (range, 0-94). The median external validation area under the receiver operating characteristic curve was 0.73 (25th-75th percentile [interquartile range (IQR)], 0.66-0.79), representing a median percent decrease in discrimination of -11.1% (IQR, -32.4% to +2.7%) compared with performance on derivation data. 81% (n=1333) of validations reporting area under the receiver operating characteristic curve showed discrimination below that reported in the derivation dataset. 53% (n=983) of the validations report some measure of CPM calibration. For CPMs evaluated more than once, there was typically a large range of performance. Of 1702 validations classified by relatedness, the percent change in discrimination was -3.7% (IQR, -13.2 to 3.1) for closely related validations (n=123), -9.0 (IQR, -27.6 to 3.9) for related validations (n=862), and -17.2% (IQR, -42.3 to 0) for distantly related validations (n=717; P<0.001). Conclusions: Many published cardiovascular CPMs have never been externally validated, and for those that have, apparent performance during development is often overly optimistic. A single external validation appears insufficient to broadly understand the performance heterogeneity across different settings.Analysis and support of clinical decision makingDevelopment and application of statistical models for medical scientific researc

Supplementary Material for: The RoPE Score and Right-to-Left Shunt Severity by Transcranial Doppler in the CODICIA Study

<b><i>Background:</i></b> For patients with cryptogenic stroke (CS) and patent foramen ovale (PFO), it is unknown whether the magnitude of right-to-left shunt (RLSh) measured by contrast transcranial Doppler (c-TCD) is correlated with the likelihood an identified PFO is related to CS as determined by the Risk of Paradoxical Embolism (RoPE) score. Additionally, for patients with CS, it is unknown whether PFO assessment by c-TCD is more sensitive for identifying RLSh compared with transesophageal echocardiography (TEE). Our aim was to determine the significance of RLSh grade by c-TCD in patients with PFO and CS. <b><i>Methods:</i></b> We evaluated patients with CS who had RLSh quantified by c-TCD in the Multicenter Study into RLSh in Cryptogenic Stroke (CODICIA) to determine whether there is an association between c-TCD shunt grade and the RoPE Score. For patients who underwent c-TCD and TEE, we determined whether there is agreement in identifying and grading RLSh between these two modalities. <b><i>Results:</i></b> The RoPE score predicted the presence versus the absence of RLSh documented by c-TCD (c-statistic = 0.66). For patients with documented RLSh by c-TCD, shunt severity was correlated with increasing RoPE score (rank correlation (r) = 0.15, p = 0.01). Among 293 patients who had both c-TCD and TEE performed, c-TCD was more sensitive (98.7%) for detecting RLSh. Of the 97 patients with no PFO identified on TEE, 28 (29%) had a large amount of RLSh seen on c-TCD. <b><i>Conclusions:</i></b> For patients with CS, severity of RLSh by c-TCD is positively correlated with the RoPE score, indicating that this technique for shunt grading identifies patients more likely to have pathogenic rather than incidental PFOs. c-TCD is also more sensitive in detecting RLSh than TEE. These findings suggest an important role for c-TCD in the evaluation of PFO in the setting of CS

Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models

Background: While clinical prediction models (CPMs) are used increasingly commonly to guide patient care, the performance and clinical utility of these CPMs in new patient cohorts is poorly understood. Methods: We performed 158 external validations of 104 unique CPMs across 3 domains of cardiovascular disease (primary prevention, acute coronary syndrome, and heart failure). Validations were performed in publicly available clinical trial cohorts and model performance was assessed using measures of discrimination, calibration, and net benefit. To explore potential reasons for poor model performance, CPM-clinical trial cohort pairs were stratified based on relatedness, a domain-specific set of characteristics to qualitatively grade the similarity of derivation and validation patient populations. We also examined the model-based C-statistic to assess whether changes in discrimination were because of differences in case-mix between the derivation and validation samples. The impact of model updating on model performance was also assessed. Results: Discrimination decreased significantly between model derivation (0.76 [interquartile range 0.73-0.78]) and validation (0.64 [interquartile range 0.60-0.67], P<0.001), but approximately half of this decrease was because of narrower case-mix in the validation samples. CPMs had better discrimination when tested in related compared with distantly related trial cohorts. Calibration slope was also significantly higher in related trial cohorts (0.77 [interquartile range, 0.59-0.90]) than distantly related cohorts (0.59 [interquartile range 0.43-0.73], P=0.001). When considering the full range of possible decision thresholds between half and twice the outcome incidence, 91% of models had a risk of harm (net benefit below default strategy) at some threshold; this risk could be reduced substantially via updating model intercept, calibration slope, or complete re-estimation. Conclusions: There are significant decreases in model performance when applying cardiovascular disease CPMs to new patient populations, resulting in substantial risk of harm. Model updating can mitigate these risks. Care should be taken when using CPMs to guide clinical decision-making.Development and application of statistical models for medical scientific researc