Development and application of statistical methods for prognosis research

A pivotal component of prognosis research is the prediction of future outcome risk. This thesis applies, develops and evaluates novel statistical methods for development and validation of risk prediction (prognostic) models. In the first part, a literature review of published prediction models shows that the Cox model remains the most common approach for developing a model using survival data; however, this avoids modelling the baseline hazard and therefore restricts individualised predictions. Flexible parametric survival models are shown to address this by flexibly modelling the baseline hazard, thereby enabling individualised risk predictions over time. Clinical application reveals discrepant mortality rates for different hip replacement procedures, and identifies common issues when developing models using clinical trial data. In the second part, univariate and multivariate random-effects meta-analyses are proposed to summarise a model’s performance across multiple validation studies. The multivariate approach accounts for correlation in multiple statistics (e.g. C-statistic and calibration slope), and allows joint predictions about expected model performance in applied settings. This allows competing implementation strategies (e.g. regarding baseline hazard choice) to be compared and ranked. A simulation study also provides recommendations for the scales on which to combine performance statistics to best satisfy the between-study normality assumption in random-effects meta-analysis

Comparison and reproducibility of standard and high temporal resolution myocardial tissue tagging in patients with severe aortic stenosis

Objectives The aim of this study was to compare and assess the reproducibility of left ventricular (LV) circumferential peak systolic strain (PeakEcc) and strain rate (SR) measurements using standard and high temporal resolution myocardial tissue tagging in patients with severe aortic stenosis (AS). Background Myocardial tissue tagging with cardiac magnetic resonance (CMR) can be used to quantify strain and SR, however, there are little data on the reproducibility. Diastolic SR may be of particular interest as it may be the most sensitive marker of diastolic dysfunction often occurring early in the course of disease. Methods Eight patients with isolated severe AS without obstructive coronary artery disease were prospectively enrolled. They underwent CMR in a 1.5T scanner (Siemens Avanto) on two separate occasions, median interval 12 days. Complementary tagged (CSPAMM) images were acquired with both a single breath-hold (SBH: temporal resolution 42ms), and a multiple brief expiration breath-hold (MBH: high temporal resolution 17ms) sequence. Mid-wall PeakEcc was measured in the LV at mid-ventricular level with HARP Version 2.7 (Diagnosoft, USA). SR was calculated from the strain data; SR=Ecc2-Ecc1/Time2-Time1. PeakEcc , peak systolic and diastolic SR were read from curves of strain and SR against time. The MBH SR curves were filtered with a moving average (MA) to reduce noise sensitivity, results from a sample width of three and five were examined. Differences between SBH and MBH were assessed using Wilcoxon signed-rank test as not all measures were normally distributed. Reproducibility assessments were carried out on all techniques. Results PeakEcc was significantly higher with MBH vs. SBH, but reproducibility was slightly worse. Results are summarised in Table 1. Systolic SR was approximately equal with all techniques although MBH using MA of five led to a borderline significant reduction. Diastolic SR was higher when measured with MBH although only significant using MA of three. Systolic and diastolic SR measures were more reproducible with MBH compared with SBH, except for the diastolic SR using MA of three, which was substantially worse. Strain and SR curves for the same patient are shown in Figure 1

Guide to presenting clinical prediction models for use in clinical settings

For permission to use (where not already granted under a licence) please go to. Clinical prediction models estimate the risk of existing disease or future outcome for an individual, which is conditional on the values of multiple predictors such as age, sex, and biomarkers. In this article, Bonnett and colleagues provide a guide to presenting clinical prediction models so that they can be implemented in practice, if appropriate. They describe how to create four presentation formats and discuss the advantages and disadvantages of each format. A key message is the need for stakeholder engagement to determine the best presentation option in relation to the clinical context of use and the intended users

Predictors of outcome in sciatica patients following an epidural steroid injection:the POiSE prospective observational cohort study protocol

INTRODUCTION: Sciatica can be very painful and, in most cases, is due to pressure on a spinal nerve root from a disc herniation with associated inflammation. For some patients, the pain persists, and one management option is a spinal epidural steroid injection (ESI). The aim of an ESI is to relieve leg pain, improve function and reduce the need for surgery. ESIs work well in some patients but not in others, but we cannot identify these patient subgroups currently. This study aims to identify factors, including patient characteristics, clinical examination and imaging findings, that help in predicting who does well and who does not after an ESI. The overall objective is to develop a prognostic model to support individualised patient and clinical decision-making regarding ESI. METHODS: POiSE is a prospective cohort study of 439 patients with sciatica referred by their clinician for an ESI. Participants will receive weekly text messages until 12 weeks following their ESIand then again at 24 weeks following their ESI to collect data on leg pain severity. Questionnaires will be sent to participants at baseline, 6, 12 and 24 weeks after their ESI to collect data on pain, disability, recovery and additional interventions. The prognosis for the cohort will be described. The primary outcome measure for the prognostic model is leg pain at 6 weeks. Prognostic models will also be developed for secondary outcomes of disability and recovery at 6 weeks and additional interventions at 24 weeks following ESI. Statistical analyses will include multivariable linear and logistic regression with mixed effects model. ETHICS AND DISSEMINATION: The POiSE study has received ethical approval (South Central Berkshire B Research Ethics Committee 21/SC/0257). Dissemination will be guided by our patient and public engagement group and will include scientific publications, conference presentations and social media.</p

Evaluation of clinical prediction models (part 2): how to undertake an external validation study

External validation studies are an important but often neglected part of prediction model research. In this article, the second in a series on model evaluation, Riley and colleagues explain what an external validation study entails and describe the key steps involved, from establishing a high quality dataset to evaluating a model’s predictive performance and clinical usefulness

Transparent reporting of multivariable prediction models developed or validated using clustered data: TRIPOD-Cluster checklist

The increasing availability of large combined datasets (or big data), such as those from electronic health records and from individual participant data meta-analyses, provides new opportunities and challenges for researchers developing and validating (including updating) prediction models. These datasets typically include individuals from multiple clusters (such as multiple centres, geographical locations, or different studies). Accounting for clustering is important to avoid misleading conclusions and enables researchers to explore heterogeneity in prediction model performance across multiple centres, regions, or countries, to better tailor or match them to these different clusters, and thus to develop prediction models that are more generalisable. However, this requires prediction model researchers to adopt more specific design, analysis, and reporting methods than standard prediction model studies that do not have any inherent substantial clustering. Therefore, prediction model studies based on clustered data need to be reported differently so that readers can appraise the study methods and findings, further increasing the use and implementation of such prediction models developed or validated from clustered datasets

Transparent reporting of multivariable prediction models developed or validated using clustered data (TRIPOD-Cluster): explanation and elaboration

The TRIPOD-Cluster (transparent reporting of multivariable prediction models developed or validated using clustered data) statement comprises a 19 item checklist, which aims to improve the reporting of studies developing or validating a prediction model in clustered data, such as individual participant data meta-analyses (clustering by study) and electronic health records (clustering by practice or hospital). This explanation and elaboration document describes the rationale; clarifies the meaning of each item; and discusses why transparent reporting is important, with a view to assessing risk of bias and clinical usefulness of the prediction model. Each checklist item of the TRIPOD-Cluster statement is explained in detail and accompanied by published examples of good reporting. The document also serves as a reference of factors to consider when designing, conducting, and analysing prediction model development or validation studies in clustered data. To aid the editorial process and help peer reviewers and, ultimately, readers and systematic reviewers of prediction model studies, authors are recommended to include a completed checklist in their submission

Evaluation of clinical prediction models (part 2):how to undertake an external validation study

External validation studies are an important but often neglected part of prediction model research. In this article, the second in a series on model evaluation, Riley and colleagues explain what an external validation study entails and describe the key steps involved, from establishing a high quality dataset to evaluating a model’s predictive performance and clinical usefulness.</p

External validation of clinical prediction models:simulation-based sample size calculations were more reliable than rules-of-thumb

INTRODUCTION: Sample size "rules-of-thumb" for external validation of clinical prediction models suggest at least 100 events and 100 non-events. Such blanket guidance is imprecise, and not specific to the model or validation setting. We investigate factors affecting precision of model performance estimates upon external validation, and propose a more tailored sample size approach.METHODS: Simulation of logistic regression prediction models to investigate factors associated with precision of performance estimates. Then, explanation and illustration of a simulation-based approach to calculate the minimum sample size required to precisely estimate a model's calibration, discrimination and clinical utility.RESULTS: Precision is affected by the model's linear predictor (LP) distribution, in addition to number of events and total sample size. Sample sizes of 100 (or even 200) events and non-events can give imprecise estimates, especially for calibration. The simulation-based calculation accounts for the LP distribution and (mis)calibration in the validation sample. Application identifies 2430 required participants (531 events) for external validation of a deep vein thrombosis diagnostic model.CONCLUSION: Where researchers can anticipate the distribution of the model's LP (eg, based on development sample, or a pilot study), a simulation-based approach for calculating sample size for external validation offers more flexibility and reliability than rules-of-thumb.</p

Multivariate meta-analysis of individual participant data helped externally validate the performance and implementation of a prediction model

Objectives Our aim was to improve meta-analysis methods for summarizing a prediction model's performance when individual participant data are available from multiple studies for external validation. Study Design and Setting We suggest multivariate meta-analysis for jointly synthesizing calibration and discrimination performance, while accounting for their correlation. The approach estimates a prediction model's average performance, the heterogeneity in performance across populations, and the probability of "good" performance in new populations. This allows different implementation strategies (e.g., recalibration) to be compared. Application is made to a diagnostic model for deep vein thrombosis (DVT) and a prognostic model for breast cancer mortality. Results In both examples, multivariate meta-analysis reveals that calibration performance is excellent on average but highly heterogeneous across populations unless the model's intercept (baseline hazard) is recalibrated. For the cancer model, the probability of "good" performance (defined by C statistic ≥ 0.7 and calibration slope between 0.9 and 1.1) in a new population was 0.67 with recalibration but 0.22 without recalibration. For the DVT model, even with recalibration, there was only a 0.03 probability of "good" performance. Conclusion Multivariate meta-analysis can be used to externally validate a prediction model's calibration and discrimination performance across multiple populations and to evaluate different implementation strategies