Simulation-based power calculations for planning a two-stage individual participant data meta-analysis

BACKGROUND Researchers and funders should consider the statistical power of planned Individual Participant Data (IPD) meta-analysis projects, as they are often time-consuming and costly. We propose simulation-based power calculations utilising a two-stage framework, and illustrate the approach for a planned IPD meta-analysis of randomised trials with continuous outcomes where the aim is to identify treatment-covariate interactions. METHODS The simulation approach has four steps: (i) specify an underlying (data generating) statistical model for trials in the IPD meta-analysis; (ii) use readily available information (e.g. from publications) and prior knowledge (e.g. number of studies promising IPD) to specify model parameter values (e.g. control group mean, intervention effect, treatment-covariate interaction); (iii) simulate an IPD meta-analysis dataset of a particular size from the model, and apply a two-stage IPD meta-analysis to obtain the summary estimate of interest (e.g. interaction effect) and its associated p-value; (iv) repeat the previous step (e.g. thousands of times), then estimate the power to detect a genuine effect by the proportion of summary estimates with a significant p-value. RESULTS In a planned IPD meta-analysis of lifestyle interventions to reduce weight gain in pregnancy, 14 trials (1183 patients) promised their IPD to examine a treatment-BMI interaction (i.e. whether baseline BMI modifies intervention effect on weight gain). Using our simulation-based approach, a two-stage IPD meta-analysis has < 60% power to detect a reduction of 1 kg weight gain for a 10-unit increase in BMI. Additional IPD from ten other published trials (containing 1761 patients) would improve power to over 80%, but only if a fixed-effect meta-analysis was appropriate. Pre-specified adjustment for prognostic factors would increase power further. Incorrect dichotomisation of BMI would reduce power by over 20%, similar to immediately throwing away IPD from ten trials. CONCLUSIONS Simulation-based power calculations could inform the planning and funding of IPD projects, and should be used routinely

Minimum sample size for external validation of a clinical prediction model with a continuous outcome.

Clinical prediction models provide individualized outcome predictions to inform patient counseling and clinical decision making. External validation is the process of examining a prediction model's performance in data independent to that used for model development. Current external validation studies often suffer from small sample sizes, and subsequently imprecise estimates of a model's predictive performance. To address this, we propose how to determine the minimum sample size needed for external validation of a clinical prediction model with a continuous outcome. Four criteria are proposed, that target precise estimates of (i) R2 (the proportion of variance explained), (ii) calibration-in-the-large (agreement between predicted and observed outcome values on average), (iii) calibration slope (agreement between predicted and observed values across the range of predicted values), and (iv) the variance of observed outcome values. Closed-form sample size solutions are derived for each criterion, which require the user to specify anticipated values of the model's performance (in particular R2 ) and the outcome variance in the external validation dataset. A sensible starting point is to base values on those for the model development study, as obtained from the publication or study authors. The largest sample size required to meet all four criteria is the recommended minimum sample size needed in the external validation dataset. The calculations can also be applied to estimate expected precision when an existing dataset with a fixed sample size is available, to help gauge if it is adequate. We illustrate the proposed methods on a case-study predicting fat-free mass in children

Internal-external cross-validation helped to evaluate the generalizability of prediction models in large clustered datasets

OBJECTIVE: To illustrate how to evaluate the need of complex strategies for developing generalizable prediction models in large clustered datasets. STUDY DESIGN AND SETTING: We developed eight Cox regression models to estimate the risk of heart failure using a large population-level dataset. These models differed in the number of predictors, the functional form of the predictor effects (non-linear effects and interaction) and the estimation method (maximum likelihood and penalization). Internal-external cross-validation was used to evaluate the models' generalizability across the included general practices. RESULTS: Among 871,687 individuals from 225 general practices, 43,987 (5.5%) developed heart failure during a median follow-up time of 5.8 years. For discrimination, the simplest prediction model yielded a good concordance statistic, which was not much improved by adopting complex strategies. Between-practice heterogeneity in discrimination was similar in all models. For calibration, the simplest model performed satisfactorily. Although accounting for non-linear effects and interaction slightly improved the calibration slope, it also led to more heterogeneity in the observed/expected ratio. Similar results were found in a second case study involving patients with stroke. CONCLUSION: In large clustered datasets, prediction model studies may adopt internal-external cross-validation to evaluate the generalizability of competing models, and to identify promising modelling strategies

Prognostic factors associated with outcome following an epidural steroid injection for disc-related sciatica: a systematic review and narrative synthesis.

PURPOSE: Clinical guidelines recommend epidural steroid injection (ESI) as a treatment option for severe disc-related sciatica, but there is considerable uncertainty about its effectiveness. Currently, we know very little about factors that might be associated with good or poor outcomes from ESI. The aim of this systematic review was to synthesise and appraise the evidence investigating prognostic factors associated with outcomes following ESI for patients with imaging confirmed disc-related sciatica. METHODS: The search strategy involved the electronic databases Medline, Embase, CINAHL Plus, PsycINFO and reference lists of eligible studies. Selected papers were quality appraised independently by two reviewers using the Quality in Prognosis Studies tool. Between-study heterogeneity precluded statistical pooling of results. RESULTS: 3094 citations were identified; 15 studies were eligible. Overall study quality was low with all judged to have moderate or high risk of bias. Forty-two prognostic factors were identified but were measured inconsistently. The most commonly assessed prognostic factors were related to pain and function (n = 10 studies), imaging features (n = 8 studies), patient socio-demographics (n = 7 studies), health and lifestyle (n = 6 studies), clinical assessment findings (n = 4 studies) and injection level (n = 4 studies). No prognostic factor was found to be consistently associated with outcomes following ESI. Most studies found no association or results that conflicted with other studies. CONCLUSIONS: There is little, and low quality, evidence to guide practice in terms of factors that predict outcomes in patients following ESI for disc-related sciatica. The results can help inform some of the decisions about potential prognostic factors that should be assessed in future well-designed prospective cohort studies

Calculating the sample size required for developing a clinical prediction model.

Clinical prediction models aim to predict outcomes in individuals, to inform diagnosis or prognosis in healthcare. Hundreds of prediction models are published in the medical literature each year, yet many are developed using a dataset that is too small for the total number of participants or outcome events. This leads to inaccurate predictions and consequently incorrect healthcare decisions for some individuals. In this article, the authors provide guidance on how to calculate the sample size required to develop a clinical prediction model

The test accuracy of antenatal ultrasound definitions of fetal macrosomia to predict birth injury: A systematic review.

OBJECTIVES: To determine which ultrasound measurement for predicted fetal macrosomia most accurately predicts adverse delivery and neonatal outcomes. STUDY DESIGN: Four biomedical databases searched for studies published after 1966. Randomised trials or observational studies of women with singleton pregnancies, resulting in a term birth who have undergone an index test of interest measured and recorded as predicted fetal macrosomia ≥28 weeks. Adverse outcomes of interest included shoulder dystocia, brachial plexus injury (BPI) and Caesarean section. RESULTS: Twenty-five observational studies (13,285 participants) were included. For BPI, the only significant positive association was found for Abdominal Circumference (AC) to Head Circumference (HC) difference > 50 mm (OR 7.2, 95 % CI 1.8-29). Shoulder dystocia was significantly associated with abdominal diameter (AD) minus biparietal diameter (BPD) ≥ 2.6 cm (OR 4.2, 95 % CI 2.3-7.5, PPV 11 %) and AC > 90th centile (OR 2.3, 95 % CI 1.3-4.0, PPV 8.6 %) and an estimated fetal weight (EFW) > 4000 g (OR 2.1 95 %CI 1.0-4.1, PPV 7.2 %). CONCLUSIONS: Estimated fetal weight is the most widely used ultrasound marker to predict fetal macrosomia in the UK. This study suggests other markers have a higher positive predictive value for adverse outcomes associated with fetal macrosomia

Transparent reporting of multivariable prediction models developed or validated using clustered data (TRIPOD-Cluster): explanation and elaboration.

The TRIPOD-Cluster (transparent reporting of multivariable prediction models developed or validated using clustered data) statement comprises a 19 item checklist, which aims to improve the reporting of studies developing or validating a prediction model in clustered data, such as individual participant data meta-analyses (clustering by study) and electronic health records (clustering by practice or hospital). This explanation and elaboration document describes the rationale; clarifies the meaning of each item; and discusses why transparent reporting is important, with a view to assessing risk of bias and clinical usefulness of the prediction model. Each checklist item of the TRIPOD-Cluster statement is explained in detail and accompanied by published examples of good reporting. The document also serves as a reference of factors to consider when designing, conducting, and analysing prediction model development or validation studies in clustered data. To aid the editorial process and help peer reviewers and, ultimately, readers and systematic reviewers of prediction model studies, authors are recommended to include a completed checklist in their submission

Internal-external cross-validation helped to evaluate the generalizability of prediction models in large clustered datasets.

OBJECTIVE: To illustrate how to evaluate the need of complex strategies for developing generalizable prediction models in large clustered datasets. STUDY DESIGN AND SETTING: We developed eight Cox regression models to estimate the risk of heart failure using a large population-level dataset. These models differed in the number of predictors, the functional form of the predictor effects (non-linear effects and interaction) and the estimation method (maximum likelihood and penalization). Internal-external cross-validation was used to evaluate the models' generalizability across the included general practices. RESULTS: Among 871,687 individuals from 225 general practices, 43,987 (5.5%) developed heart failure during a median follow-up time of 5.8 years. For discrimination, the simplest prediction model yielded a good concordance statistic, which was not much improved by adopting complex strategies. Between-practice heterogeneity in discrimination was similar in all models. For calibration, the simplest model performed satisfactorily. Although accounting for non-linear effects and interaction slightly improved the calibration slope, it also led to more heterogeneity in the observed/expected ratio. Similar results were found in a second case study involving patients with stroke. CONCLUSION: In large clustered datasets, prediction model studies may adopt internal-external cross-validation to evaluate the generalizability of competing models, and to identify promising modelling strategies