Evaluation of clinical prediction models (part 3): calculating the sample size required for an external validation study

An external validation study evaluates the performance of a prediction model in new data, but many of these studies are too small to provide reliable answers. In the third article of their series on model evaluation, Riley and colleagues describe how to calculate the sample size required for external validation studies, and propose to avoid rules of thumb by tailoring calculations to the model and setting at hand

A blended preconception lifestyle programme for couples undergoing IVF:lessons learned from a multicentre randomized controlled trial

Study question: What is the effect of a blended preconception lifestyle programme on reproductive and lifestyle outcomes of couples going through their first 12 months of IVF as compared to an attention control condition?Summery answer:This randomized controlled trial (RCT) was stopped prematurely because of the coronavirus disease 2019 (Covid-19) pandemic but the available data did not suggest that a blended preconception lifestyle programme could meaningfully affect time to ongoing pregnancy or other reproductive and lifestyle outcomes.What is know already:Increasing evidence shows associations between a healthy lifestyle and IVF success rates. Lifestyle programmes provided through a mobile phone application have yet to be evaluated by RCTs in couples undergoing IVF.Study design, size, duration:A multicentre RCT (1:1) was carried out. The RCT started in January 2019 and was prematurely stopped because of the Covid-19 pandemic, leading to a reduced sample size (211 couples initiating IVF) and change in primary outcome (cumulative ongoing pregnancy to time to ongoing pregnancy).Participants/materials, setting, methods:Heterosexual couples initiating IVF in five fertility clinics were randomized between an attention control arm and an intervention arm for 12 months. The attention control arm received treatment information by mobile phone in addition to standard care. The intervention arm received the blended preconception lifestyle (PreLiFe)-programme in addition to standard care. The PreLiFe-programme included a mobile application, offering tailored advice and skills training on diet, physical activity and mindfulness, in combination with motivational interviewing over the telephone. The primary outcome was 'time to ongoing pregnancy'. Secondary reproductive outcomes included the Core Outcome Measures for Infertility Trials and IVF discontinuation. Changes in the following secondary lifestyle outcomes over 3 and 6 months were studied in both partners: diet quality, fruit intake, vegetable intake, total moderate to vigorous physical activity, sedentary behaviour, emotional distress, quality of life, BMI, and waist circumference. Finally, in the intervention arm, acceptability of the programme was evaluated and actual use of the mobile application part of the programme was tracked. Analysis was according to intention to treat.Main results and the role of chance:A total of 211 couples were randomized (105 control arm, 106 intervention arm). The hazard ratio of the intervention for time to ongoing pregnancy was 0.94 (95% CI 0.63 to 1.4). Little to no effect on other reproductive or lifestyle outcomes was identified. Although acceptability of the programme was good (6/10), considerable proportions of men (38%) and 9% of women did not actively use all the modules of the mobile application (diet, physical activity, or mindfulness).Limitations, reasons for caution:The findings of this RCT should be considered exploratory, as the Covid-19 pandemic limited its power and the actual use of the mobile application was low.Wider implications of the findings:This is the first multicentre RCT evaluating the effect of a blended preconception lifestyle programme for women and their partners undergoing IVF on both reproductive and lifestyle outcomes. This exploratory RCT highlights the need for further studies into optimal intervention characteristics and actual use of preconception lifestyle programmes, as well as RCTs evaluating effectiveness.Study fonding/competing intrest(s):Supported by the Research foundation Flanders (Belgium) (FWO-TBM; reference: T005417N). No competing interests to declare.Trial registration number:ClinicalTrials.gov Identifier: NCT03790449TRIAL REGISTRATION DATE 31 December 2018DATE OF FIRST PATIENT'S ENROLMENT 2 January 201

Managing pregnancy of unknown location based on initial serum progesterone and serial serum hCG: development and validation of a two-step triage protocol.

A uniform rationalized management protocol for pregnancies of unknown location (PUL) is lacking. We developed a two-step triage protocol based on presenting serum progesterone (step 1) and hCG ratio two days later (step 2) to select PUL at high-risk of ectopic pregnancy (EP).Cohort study of 2753 PUL (301 EP), involving a secondary analysis of prospectively and consecutively collected PUL at two London-based university teaching hospitals. Using a chronological split we used 1449 PUL for development and 1304 for validation. We aimed to select PUL as low-risk with high confidence (high negative predictive value, NPV) while classifying most EP as high-risk (high sensitivity). The first triage step selects low-risk PUL at presentation using a serum progesterone threshold. The remaining PUL are triaged using a novel logistic regression risk model based on hCG ratio and initial serum progesterone (second step), defining low-risk as an estimated EP risk <5%.On validation, initial serum progesterone ≤2nmol/l (step 1) selected 16.1% PUL as low-risk. Second step classification with the risk model M6P selected an additional 46.0% of all PUL as low-risk. Overall, the two-step protocol classified 62.1% of PUL as low-risk, with an NPV of 98.6% and a sensitivity of 92.0%. When the risk model was used in isolation (i.e. without the first step), 60.5% of PUL were classified as low-risk with 99.1% NPV and 94.9% sensitivity.The two-step protocol can efficiently classify PUL into being at high or low risk of complications

Critical appraisal of artificial intelligence-based prediction models for cardiovascular disease

The medical field has seen a rapid increase in the development of artificial intelligence (AI)-based prediction models. With the introduction of such AI-based prediction model tools and software in cardiovascular patient care, the cardiovascular researcher and healthcare professional are challenged to understand the opportunities as well as the limitations of the AI-based predictions. In this article, we present 12 critical questions for cardiovascular health professionals to ask when confronted with an AI-based prediction model. We aim to support medical professionals to distinguish the AI-based prediction models that can add value to patient care from the AI that does not

Evaluating the risk of ovarian cancer before surgery using the ADNEX model to differentiate between benign, borderline, early and advanced stage invasive, and secondary metastatic tumours: prospective multicentre diagnostic study

Objectives To develop a risk prediction model to preoperatively discriminate between benign, borderline, stage I invasive, stage II-IV invasive, and secondary metastatic ovarian tumours. Design Observational diagnostic study using prospectively collected clinical and ultrasound data. Setting 24 ultrasound centres in 10 countries. Participants Women with an ovarian (including para-ovarian and tubal) mass and who underwent a standardised ultrasound examination before surgery. The model was developed on 3506 patients recruited between 1999 and 2007, temporally validated on 2403 patients recruited between 2009 and 2012, and then updated on all 5909 patients. Main outcome measures Histological classification and surgical staging of the mass. Results The Assessment of Different NEoplasias in the adneXa (ADNEX) model contains three clinical and six ultrasound predictors: age, serum CA-125 level, type of centre (oncology centres v other hospitals), maximum diameter of lesion, proportion of solid tissue, more than 10 cyst locules, number of papillary projections, acoustic shadows, and ascites. The area under the receiver operating characteristic curve (AUC) for the classic discrimination between benign and malignant tumours was 0.94 (0.93 to 0.95) on temporal validation. The AUC was 0.85 for benign versus borderline, 0.92 for benign versus stage I cancer, 0.99 for benign versus stage II-IV cancer, and 0.95 for benign versus secondary metastatic. AUCs between malignant subtypes varied between 0.71 and 0.95, with an AUC of 0.75 for borderline versus stage I cancer and 0.82 for stage II-IV versus secondary metastatic. Calibration curves showed that the estimated risks were accurate. Conclusions The ADNEX model discriminates well between benign and malignant tumours and offers fair to excellent discrimination between four types of ovarian malignancy. The use of ADNEX has the potential to improve triage and management decisions and so reduce morbidity and mortality associated with adnexal pathology

Treatment of steroid-induced elevated intraocular pressure with anecortave acetate: a randomized clinical trial.

PURPOSE: The present study is the first randomized clinical trial designed to evaluate the intraocular pressure (IOP)-lowering effect of anecortave acetate (AA) administered at 3 doses (3, 15, or 30 mg) as an anterior juxtascleral depot (AJD) in patients experiencing elevated IOP due to corticosteroid therapy. METHODS: This was a double-masked, randomized, placebo-controlled, multicenter, parallel group trial. Eligible patients had an IOP of at least 24 mmHg and an IOP increase of at least 10 mmHg relative to their IOP before treatment with steroids. A target IOP was established for each patient at baseline. Patients were randomized to 1 of the 4 treatment groups: vehicle, 3 mg AA, 15 mg AA, or 30 mg AA. All patients then received a 0.5 mL AJD of the assigned treatment. Patients returned for scheduled examination visits at weeks 1, 2, 4, 6, months 3, 4, 5, and 6. IOP was measured at each visit as well as best corrected visual acuity (logMAR), ocular motility, eyelid responsiveness, slit lamp examination, and assessment of any adverse events. In addition, at baseline and at exit, a dilated fundus examination was carried out and the lens was examined using LOCS II criteria. RESULTS: Seventy patients were randomized to treatment. At week 4, eyes in the vehicle group showed a 3.4 mmHg (9.1%) decrease from baseline. Reductions for the 3 mg AA (3.1 mmHg, 10.7%) and the 30 mg AA groups (5.4 mmHg, 16.6%) were not significantly different than for vehicle control. However, IOP for the 15 mg AA group at week 4 was reduced 11.5 mmHg (31.3%) from baseline, which was statistically significant (P=0.0487). The mean time to treatment failure was 32.2, 38.9, 56.3, and 32.6 days for the vehicle, 3 mg AA, 15 mg AA, and 30 mg AA groups, respectively. Adverse events were assessed at each post-treatment visit. There were no serious adverse events that were determined to be related to the test article or its administration. CONCLUSIONS: AA can be of benefit to some patients requiring treatment with corticosteroids, but suffering from the side effect of elevated IOP

Evaluation of clinical prediction models (part 1): from development to external validation

Evaluating the performance of a clinical prediction model is crucial to establish its predictive accuracy in the populations and settings intended for use. In this article, the first in a three part series, Collins and colleagues describe the importance of a meaningful evaluation using internal, internal-external, and external validation, as well as exploring heterogeneity, fairness, and generalisability in model performance

Evaluation of clinical prediction models (part 1):from development to external validation

Evaluating the performance of a clinical prediction model is crucial to establish its predictive accuracy in the populations and settings intended for use. In this article, the first in a three part series, Collins and colleagues describe the importance of a meaningful evaluation using internal, internal-external, and external validation, as well as exploring heterogeneity, fairness, and generalisability in model performance