Meta-analysis using individual participant data: one-stage and two-stage approaches, and why they may differ.

Meta-analysis using individual participant data (IPD) obtains and synthesises the raw, participant-level data from a set of relevant studies. The IPD approach is becoming an increasingly popular tool as an alternative to traditional aggregate data meta-analysis, especially as it avoids reliance on published results and provides an opportunity to investigate individual-level interactions, such as treatment-effect modifiers. There are two statistical approaches for conducting an IPD meta-analysis: one-stage and two-stage. The one-stage approach analyses the IPD from all studies simultaneously, for example, in a hierarchical regression model with random effects. The two-stage approach derives aggregate data (such as effect estimates) in each study separately and then combines these in a traditional meta-analysis model. There have been numerous comparisons of the one-stage and two-stage approaches via theoretical consideration, simulation and empirical examples, yet there remains confusion regarding when each approach should be adopted, and indeed why they may differ. In this tutorial paper, we outline the key statistical methods for one-stage and two-stage IPD meta-analyses, and provide 10 key reasons why they may produce different summary results. We explain that most differences arise because of different modelling assumptions, rather than the choice of one-stage or two-stage itself. We illustrate the concepts with recently published IPD meta-analyses, summarise key statistical software and provide recommendations for future IPD meta-analyses. © 2016 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd

Deriving percentage study weights in multi-parameter meta-analysis models: with application to meta-regression, network meta-analysis and one-stage individual participant data models.

Many meta-analysis models contain multiple parameters, for example due to multiple outcomes, multiple treatments or multiple regression coefficients. In particular, meta-regression models may contain multiple study-level covariates, and one-stage individual participant data meta-analysis models may contain multiple patient-level covariates and interactions. Here, we propose how to derive percentage study weights for such situations, in order to reveal the (otherwise hidden) contribution of each study toward the parameter estimates of interest. We assume that studies are independent, and utilise a decomposition of Fisher's information matrix to decompose the total variance matrix of parameter estimates into study-specific contributions, from which percentage weights are derived. This approach generalises how percentage weights are calculated in a traditional, single parameter meta-analysis model. Application is made to one- and two-stage individual participant data meta-analyses, meta-regression and network (multivariate) meta-analysis of multiple treatments. These reveal percentage study weights toward clinically important estimates, such as summary treatment effects and treatment-covariate interactions, and are especially useful when some studies are potential outliers or at high risk of bias. We also derive percentage study weights toward methodologically interesting measures, such as the magnitude of ecological bias (difference between within-study and across-study associations) and the amount of inconsistency (difference between direct and indirect evidence in a network meta-analysis)

Simulation-based power calculations for planning a two-stage individual participant data meta-analysis

BACKGROUND Researchers and funders should consider the statistical power of planned Individual Participant Data (IPD) meta-analysis projects, as they are often time-consuming and costly. We propose simulation-based power calculations utilising a two-stage framework, and illustrate the approach for a planned IPD meta-analysis of randomised trials with continuous outcomes where the aim is to identify treatment-covariate interactions. METHODS The simulation approach has four steps: (i) specify an underlying (data generating) statistical model for trials in the IPD meta-analysis; (ii) use readily available information (e.g. from publications) and prior knowledge (e.g. number of studies promising IPD) to specify model parameter values (e.g. control group mean, intervention effect, treatment-covariate interaction); (iii) simulate an IPD meta-analysis dataset of a particular size from the model, and apply a two-stage IPD meta-analysis to obtain the summary estimate of interest (e.g. interaction effect) and its associated p-value; (iv) repeat the previous step (e.g. thousands of times), then estimate the power to detect a genuine effect by the proportion of summary estimates with a significant p-value. RESULTS In a planned IPD meta-analysis of lifestyle interventions to reduce weight gain in pregnancy, 14 trials (1183 patients) promised their IPD to examine a treatment-BMI interaction (i.e. whether baseline BMI modifies intervention effect on weight gain). Using our simulation-based approach, a two-stage IPD meta-analysis has < 60% power to detect a reduction of 1 kg weight gain for a 10-unit increase in BMI. Additional IPD from ten other published trials (containing 1761 patients) would improve power to over 80%, but only if a fixed-effect meta-analysis was appropriate. Pre-specified adjustment for prognostic factors would increase power further. Incorrect dichotomisation of BMI would reduce power by over 20%, similar to immediately throwing away IPD from ten trials. CONCLUSIONS Simulation-based power calculations could inform the planning and funding of IPD projects, and should be used routinely

Minimum sample size for external validation of a clinical prediction model with a continuous outcome.

Clinical prediction models provide individualized outcome predictions to inform patient counseling and clinical decision making. External validation is the process of examining a prediction model's performance in data independent to that used for model development. Current external validation studies often suffer from small sample sizes, and subsequently imprecise estimates of a model's predictive performance. To address this, we propose how to determine the minimum sample size needed for external validation of a clinical prediction model with a continuous outcome. Four criteria are proposed, that target precise estimates of (i) R2 (the proportion of variance explained), (ii) calibration-in-the-large (agreement between predicted and observed outcome values on average), (iii) calibration slope (agreement between predicted and observed values across the range of predicted values), and (iv) the variance of observed outcome values. Closed-form sample size solutions are derived for each criterion, which require the user to specify anticipated values of the model's performance (in particular R2 ) and the outcome variance in the external validation dataset. A sensible starting point is to base values on those for the model development study, as obtained from the publication or study authors. The largest sample size required to meet all four criteria is the recommended minimum sample size needed in the external validation dataset. The calculations can also be applied to estimate expected precision when an existing dataset with a fixed sample size is available, to help gauge if it is adequate. We illustrate the proposed methods on a case-study predicting fat-free mass in children

Calculating the sample size required for developing a clinical prediction model.

Clinical prediction models aim to predict outcomes in individuals, to inform diagnosis or prognosis in healthcare. Hundreds of prediction models are published in the medical literature each year, yet many are developed using a dataset that is too small for the total number of participants or outcome events. This leads to inaccurate predictions and consequently incorrect healthcare decisions for some individuals. In this article, the authors provide guidance on how to calculate the sample size required to develop a clinical prediction model

Individual participant data meta-analysis to examine interactions between treatment effect and participant-level covariates: statistical recommendations for conduct and planning

Precision medicine research often searches for treatment‐covariate interactions, which refers to when a treatment effect (eg, measured as a mean difference, odds ratio, hazard ratio) changes across values of a participant‐level covariate (eg, age, gender, biomarker). Single trials do not usually have sufficient power to detect genuine treatment‐covariate interactions, which motivate the sharing of individual participant data (IPD) from multiple trials for meta‐analysis. Here, we provide statistical recommendations for conducting and planning an IPD meta‐analysis of randomized trials to examine treatment‐covariate interactions. For conduct, two‐stage and one‐stage statistical models are described, and we recommend: (i) interactions should be estimated directly, and not by calculating differences in meta‐analysis results for subgroups; (ii) interaction estimates should be based solely on within‐study information; (iii) continuous covariates and outcomes should be analyzed on their continuous scale; (iv) nonlinear relationships should be examined for continuous covariates, using a multivariate meta‐analysis of the trend (eg, using restricted cubic spline functions); and (v) translation of interactions into clinical practice is nontrivial, requiring individualized treatment effect prediction. For planning, we describe first why the decision to initiate an IPD meta‐analysis project should not be based on between‐study heterogeneity in the overall treatment effect; and second, how to calculate the power of a potential IPD meta‐analysis project in advance of IPD collection, conditional on characteristics (eg, number of participants, standard deviation of covariates) of the trials (potentially) promising their IPD. Real IPD meta‐analysis projects are used for illustration throughout

The prognostic utility of tests of platelet function for the detection of 'aspirin resistance' in patients with established cardiovascular or cerebrovascular disease: a systematic review and economic evaluation.

BACKGROUND: The use of aspirin is well established for secondary prevention of cardiovascular disease. However, a proportion of patients suffer repeat cardiovascular events despite being prescribed aspirin treatment. It is uncertain whether or not this is due to an inherent inability of aspirin to sufficiently modify platelet activity. This report aims to investigate whether or not insufficient platelet function inhibition by aspirin ('aspirin resistance'), as defined using platelet function tests (PFTs), is linked to the occurrence of adverse clinical outcomes, and further, whether or not patients at risk of future adverse clinical events can be identified through PFTs. OBJECTIVES: To review systematically the clinical effectiveness and cost-effectiveness evidence regarding the association between PFT designation of 'aspirin resistance' and the risk of adverse clinical outcome(s) in patients prescribed aspirin therapy. To undertake exploratory model-based cost-effectiveness analysis on the use of PFTs. DATA SOURCES: Bibliographic databases (e.g. MEDLINE from inception and EMBASE from 1980), conference proceedings and ongoing trial registries up to April 2012. METHODS: Standard systematic review methods were used for identifying clinical and cost studies. A risk-of-bias assessment tool was adapted from checklists for prognostic and diagnostic studies. (Un)adjusted odds and hazard ratios for the association between 'aspirin resistance', for different PFTs, and clinical outcomes are presented; however, heterogeneity between studies precluded pooling of results. A speculative economic model of a PFT and change of therapy strategy was developed. RESULTS: One hundred and eight relevant studies using a variety of PFTs, 58 in patients on aspirin monotherapy, were analysed in detail. Results indicated that some PFTs may have some prognostic utility, i.e. a trend for more clinical events to be associated with groups classified as 'aspirin resistant'. Methodological and clinical heterogeneity prevented a quantitative summary of prognostic effect. Study-level effect sizes were generally small and absolute outcome risk was not substantially different between 'aspirin resistant' and 'aspirin sensitive' designations. No studies on the cost-effectiveness of PFTs for 'aspirin resistance' were identified. Based on assumptions of PFTs being able to accurately identify patients at high risk of clinical events and such patients benefiting from treatment modification, the economic model found that a test-treat strategy was likely to be cost-effective. However, neither assumption is currently evidence based. LIMITATIONS: Poor or incomplete reporting of studies suggests a potentially large volume of inaccessible data. Analyses were confined to studies on patients prescribed aspirin as sole antiplatelet therapy at the time of PFT. Clinical and methodological heterogeneity across studies precluded meta-analysis. Given the lack of robust data the economic modelling was speculative. CONCLUSIONS: Although evidence indicates that some PFTs may have some prognostic value, methodological and clinical heterogeneity between studies and different approaches to analyses create confusion and inconsistency in prognostic results, and prevented a quantitative summary of their prognostic effect. Protocol-driven and adequately powered primary studies are needed, using standardised methods of measurements to evaluate the prognostic ability of each test in the same population(s), and ideally presenting individual patient data. For any PFT to inform individual risk prediction, it will likely need to be considered in combination with other prognostic factors, within a prognostic model. STUDY REGISTRATION: This study is registered as PROSPERO 2012:CRD42012002151. FUNDING: The National Institute for Health Research Health Technology Assessment programme

Minimum sample size for developing a multivariable prediction model using multinomial logistic regression.

AIMS: Multinomial logistic regression models allow one to predict the risk of a categorical outcome with > 2 categories. When developing such a model, researchers should ensure the number of participants (n) is appropriate relative to the number of events (Ek) and the number of predictor parameters (pk) for each category k. We propose three criteria to determine the minimum n required in light of existing criteria developed for binary outcomes. PROPOSED CRITERIA: The first criterion aims to minimise the model overfitting. The second aims to minimise the difference between the observed and adjusted R2 Nagelkerke. The third criterion aims to ensure the overall risk is estimated precisely. For criterion (i), we show the sample size must be based on the anticipated Cox-snell R2 of distinct 'one-to-one' logistic regression models corresponding to the sub-models of the multinomial logistic regression, rather than on the overall Cox-snell R2 of the multinomial logistic regression. EVALUATION OF CRITERIA: We tested the performance of the proposed criteria (i) through a simulation study and found that it resulted in the desired level of overfitting. Criterion (ii) and (iii) were natural extensions from previously proposed criteria for binary outcomes and did not require evaluation through simulation. SUMMARY: We illustrated how to implement the sample size criteria through a worked example considering the development of a multinomial risk prediction model for tumour type when presented with an ovarian mass. Code is provided for the simulation and worked example. We will embed our proposed criteria within the pmsampsize R library and Stata modules