External validation and calibration of IVFpredict:A national prospective cohort study of 130,960 in vitro fertilisation Cycles

© 2015 Smith et al. Background Accurately predicting the probability of a live birth after in vitro fertilisation (IVF) is important for patients, healthcare providers and policy makers. Two prediction models (Templeton and IVFpredict) have been previously developed from UK data and are widely used internationally. The more recent of these, IVFpredict, was shown to have greater predictive power in the development dataset. The aim of this study was external validation of the two models and comparison of their predictive ability. Methods and Findings 130,960 IVF cycles undertaken in the UK in 2008-2010 were used to validate and compare the Templeton and IVFpredict models. Discriminatory power was calculated using the area under the receiver-operator curve and calibration assessed using a calibration plot and Hosmer-Lemeshow statistic. The scaled modified Brier score, with measures of reliability and resolution, were calculated to assess overall accuracy. Both models were compared after updating for current live birth rates to ensure that the average observed and predicted live birth rates were equal. The discriminative power of both methods was comparable: the area under the receiver-operator curve was 0.628 (95% confidence interval (CI): 0.625-0.631) for IVFpredict and 0.616 (95% CI: 0.613-0.620) for the Templeton model. IVFpredict had markedly better calibration and higher diagnostic accuracy, with calibration plot intercept of 0.040 (95% CI: 0.017-0.063) and slope of 0.932 (95% CI: 0.839 - 1.025) compared with 0.080 (95% CI: 0.044-0.117) and 1.419 (95% CI: 1.149-1.690) for the Templeton model. Both models underestimated the live birth rate, but this was particularly marked in the Templeton model. Updating the models to reflect improvements in live birth rates since the models were developed enhanced their performance, but IVFpredict remained superior. Conclusion External validation in a large population cohort confirms IVFpredict has superior discrimination and calibration for informing patients, clinicians and healthcare policy makers of the probability of live birth following IVF

Can differences in IVF success rates between centres be explained by patient characteristics and sample size?

Pregnancy rates cannot be used reliably for comparison of IVF clinic performance because of differences in patients between clinics. We investigate if differences in pregnancy chance between IVF centres remain after adjustment for patient mix. We prospectively collected IVF and ICSI treatment data from 11 out of 13 IVF centres in the Netherlands, between 2002 and 2004. Adjustment for sampling variation was made using a random effects model. A prognostic index for subfertility-related factors was used to adjust for differences in patient mix. The remaining variability between centres was split into random variation and true differences. The crude 1-year ongoing pregnancy chance per centre differed by nearly a factor 3 between centres, with hazard ratios (HRs) of 0.48 (95% CI: 0.34-0.69) to 1.34 (95% CI: 1.18-1.51) compared with the mean 1-year ongoing pregnancy chance of all centres. After accounting for sampling variation, the difference shrank since HRs became 0.66 (95% CI: 0.51-0.85) to 1.28 (95% CI: 1.13-1.44). After adjustment for patient mix, the difference narrowed somewhat further to HRs of 0.74 (95% CI: 0.57-0.94) to 1.33 (95% CI: 1.20-1.48) and 17% of the variation between centres could be explained by patient mix. The 1-year cumulative ongoing pregnancy rate in the two most extreme centres was 36% and 55%. Only a minor part of the differences in pregnancy chance between IVF centres is explained by patient mix. Further research is needed to elucidate the causes of the remaining differences