Minimum sample size for external validation of a clinical prediction model with a binary outcome

In prediction model research, external validation is needed to examine an existing model's performance using data independent to that for model development. Current external validation studies often suffer from small sample sizes and consequently imprecise predictive performance estimates. To address this, we propose how to determine the minimum sample size needed for a new external validation study of a prediction model for a binary outcome. Our calculations aim to precisely estimate calibration (Observed/Expected and calibration slope), discrimination (C-statistic), and clinical utility (net benefit). For each measure, we propose closed-form and iterative solutions for calculating the minimum sample size required. These require specifying: (i) target SEs (confidence interval widths) for each estimate of interest, (ii) the anticipated outcome event proportion in the validation population, (iii) the prediction model's anticipated (mis)calibration and variance of linear predictor values in the validation population, and (iv) potential risk thresholds for clinical decision-making. The calculations can also be used to inform whether the sample size of an existing (already collected) dataset is adequate for external validation. We illustrate our proposal for external validation of a prediction model for mechanical heart valve failure with an expected outcome event proportion of 0.018. Calculations suggest at least 9835 participants (177 events) are required to precisely estimate the calibration and discrimination measures, with this number driven by the calibration slope criterion, which we anticipate will often be the case. Also, 6443 participants (116 events) are required to precisely estimate net benefit at a risk threshold of 8%. Software code is provided.</p

Reporting and Methods in Clinical Prediction Research: A Systematic Review

Walter Bouwmeester and colleagues investigated the reporting and methods of prediction studies in 2008, in six high-impact general medical journals, and found that the majority of prediction studies do not follow current methodological recommendations

Bifurcation treatment with novel, highly flexible drug-eluting coronary stents in all-comers: 2-year outcome in patients of the DUTCH PEERS trial

BACKGROUND: Percutaneous coronary intervention (PCI) in bifurcated lesions with second-generation drug-eluting stents (DES) was associated with increased myocardial infarction (MI) rates. Flexible stent designs that accommodate well to vessel tapering may be of benefit in challenging anatomies such as bifurcated target lesions, but so far data are scarce. METHODS: We analyzed the 2-year follow-up data of the DUTCH PEERS (TWENTE II) trial, which randomized 1811 all-comer patients to PCI with newer generation resolute integrity zotarolimus-eluting (Medtronic) or promus element everolimus-eluting stents (Boston Scientific). In bifurcated lesions, provisional stenting was generally performed. Target vessel failure is a composite endpoint, consisting of cardiac death, target vessel MI, or target vessel revascularization. RESULTS: Patients with at least one bifurcated lesion (n = 465, 25.7 %) versus patients with non-bifurcated target lesions only (n = 1346, 74.3 %) showed similar rates of clinical endpoints including target vessel failure (9.2 versus 7.9 %, p = 0.36) and definite stent thrombosis (0.4 versus 1.0 %, p = 0.38). Target vessel MI was more common in patients with bifurcated lesions (3.4 versus 1.6 %, p = 0.02); but after multivariate analysis with propensity score adjustment, bifurcation treatment was found not to be an independent predictor of target vessel MI (HR 1.40, 95 % CI 0.71-2.76; p = 0.34). Among patients with bifurcated lesions, DES type and side-branch size did not affect outcome, but periprocedural MI occurred more often after two-stent approaches (9.0 versus 2.1 %; p = 0.002). CONCLUSION: All-comer patients treated for bifurcated and non-bifurcated target lesions showed similar and low rates of clinical endpoints, suggesting that the DES used are efficacious and safe for treating bifurcated target lesions