Incorporating Novel Risk Markers into Established Risk Prediction Models

Introduction: Risk prediction models are used as part of formal risk assessment for disease and health events in UK primary care. To improve the accuracy of risk prediction, new risk factors are being added to established risk prediction models. However, current methods used to evaluate the added value of these new risk factors have shown to be limited. These limitations can be addressed using health economic methodology, but is yet to be used to evaluate and compare risk prediction models by means of their effectiveness and cost. Methods: A cost effectiveness analysis was performed using a decision tree framework. The decision tree was populated risk model effects and cost measures. The cost-effectiveness analysis derived the incremental cost effectiveness ratio (ICER) using the Youden Index and Harrell’s C-Index performance measures, and the net monetary benefit (INB). A probabilistic sensitivity analysis was performed, based on 10,000 iterations. A range of £0-£100,000 was used for the willingness to pay (WTP), which when combined with the INB, provided the probability the new risk factor was cost effective. This method was applied in two exemplar prospective cohort studies; adding family history (FH) to cardiovascular disease (CVD) risk prediction; and bone mineral density (BMD) to fracture risk prediction. Results: A cost-effectiveness analysis using a decision tree framework was shown to be an effective way of evaluating the added value of the new risk factor. Adding FH to standard CVD risk factors produced an ICER of £799.91 (-£5,962.15 to £5,968.22) and £7,788.76 (-£42,760.16 to £48,962.39) per percentage unit increase in the Youden Index and Harrell’s C-Index, respectively. The maximum probability of FH being cost effective is 0.7, with a minimum WTP of £15,000 (Youden Index). Further, treating low risk patients with statin therapy incorrectly was less costly (£788.40) than not treating them (£916.16). Adding continuous BMD measurement to standard fracture risk factors produced an ICER of £367.25 (-£4,241.88 to £4,828.50) and £4,480.54 (-£22,816.84 to £22,970.55) per percentage unit increase in the Youden Index and Harrell’s C-Index, respectively. The maximum probability BMD being cost-effective is 0.8, with a minimum WTP of £32,500 (Youden Index). Further, using BMD in a binary format to indicate osteoporotic patients, did not improve Harrell’s C-Index of standard fracture risk prediction (∆C-Index=-0.62%). Conclusion: A cost-effectiveness analysis was a novel method to compare two risk prediction models; and to evaluate the added value of a new risk factor. It identifies the added value of a new risk factor; encompassing the statistical and clinical improvement, and cost consequences when using the new risk factor in an established risk prediction model. Based on the added value of FH and BMD, there is a good evidence base to add these risk factors into routine risk assessment of the respective conditions. Increased use of this method could help standardise risk prediction and increase comparability of risk prediction models within diseases; producing a league table approach to evaluate, appraise and identify beneficial new risk factors and better risk prediction models

Does bone mineral density improve the predictive accuracy of fracture risk assessment?: a prospective cohort study in Northern Denmark

Objective: To evaluate the added predictive accuracy of bone mineral density (BMD) to fracture risk assessment.Design Prospective cohort study using data between 01 January 2010 and 31 December 2012. Setting: North Denmark Osteoporosis Clinic of referred patients presenting with at least one fracture risk factor to the referring doctor.Participants Patients aged 40–90 years; had BMD T-score recorded at the hip and not taking osteoporotic preventing drugs for more than 1 year prior to baseline. Main outcome measures: Incident diagnoses of osteoporotic fractures (hip, spine, forearm, humerus and pelvis) were identified using the National Patient Registry of Denmark during 01 January 2012–01 January 2014. Cox regression was used to develop a fracture model based on predictors in the Fracture Risk Assessment Tool (FRAX®), with and without, binary and continuous BMD. Change in Harrell’s C-Index and Reclassification tables were used to describe the added statistical value of BMD. Results: Adjusting for predictors included in FRAX®, patients with osteoporosis (T-score ≤−2.5) had 75% higher hazard of a fracture compared with patients with higher BMD (HR: 1.75 (95% CI 1.28 to 2.38)). Forty per cent lower hazard was found per unit increase in continuous BMD T-score (HR: 0.60 (95% CI 0.52 to 0.69)).Accuracy improved marginally, and Harrell’s C-Index increased by 1.2% when adding continuous BMD (0.76 to 0.77). Reclassification tables showed continuous BMD shifted 529 patients into different risk categories; 292 of these were reclassified correctly (57%; 95% CI 55% to 64%). Adding binary BMD however no improvement: Harrell’s C-Index decreased by 0.6%. Conclusions: Continuous BMD marginally improves fracture risk assessment. Importantly, this was only found when using continuous BMD measurement for osteoporosis. It is suggested that future focus should be on evaluation of this risk factor using routinely collected data and on the development of more clinically relevant methodology to assess the added value of a new risk factor

Antidepressant use and risk of adverse outcomes in older people: population based cohort study

Objectives To investigate the association between antidepressant treatment and risk of several potential adverse outcomes in older people with depression and to examine risks by class of antidepressant, duration of use, and dose

Evaluation of clinical prediction models (part 2):how to undertake an external validation study

External validation studies are an important but often neglected part of prediction model research. In this article, the second in a series on model evaluation, Riley and colleagues explain what an external validation study entails and describe the key steps involved, from establishing a high quality dataset to evaluating a model’s predictive performance and clinical usefulness.</p

Systematic review identifies the design and methodological conduct of studies on machine learning-based prediction models

Background and ObjectivesWe sought to summarize the study design, modelling strategies, and performance measures reported in studies on clinical prediction models developed using machine learning techniques.MethodsWe search PubMed for articles published between 01/01/2018 and 31/12/2019, describing the development or the development with external validation of a multivariable prediction model using any supervised machine learning technique. No restrictions were made based on study design, data source, or predicted patient-related health outcomes.ResultsWe included 152 studies, 58 (38.2% [95% CI 30.8–46.1]) were diagnostic and 94 (61.8% [95% CI 53.9–69.2]) prognostic studies. Most studies reported only the development of prediction models (n = 133, 87.5% [95% CI 81.3–91.8]), focused on binary outcomes (n = 131, 86.2% [95% CI 79.8–90.8), and did not report a sample size calculation (n = 125, 82.2% [95% CI 75.4–87.5]). The most common algorithms used were support vector machine (n = 86/522, 16.5% [95% CI 13.5–19.9]) and random forest (n = 73/522, 14% [95% CI 11.3–17.2]). Values for area under the Receiver Operating Characteristic curve ranged from 0.45 to 1.00. Calibration metrics were often missed (n = 494/522, 94.6% [95% CI 92.4–96.3]).ConclusionOur review revealed that focus is required on handling of missing values, methods for internal validation, and reporting of calibration to improve the methodological conduct of studies on machine learning–based prediction models

Evaluation of clinical prediction models (part 1):from development to external validation

Evaluating the performance of a clinical prediction model is crucial to establish its predictive accuracy in the populations and settings intended for use. In this article, the first in a three part series, Collins and colleagues describe the importance of a meaningful evaluation using internal, internal-external, and external validation, as well as exploring heterogeneity, fairness, and generalisability in model performance

Psychological morbidity and return to work after injury: multicentre cohort study

Background: The benefits of work for physical, psychological and financial wellbeing are well documented. Return to work (RTW) after unintentional injury is often delayed, and psychological morbidity may contribute to this delay. The impact of psychological morbidity on RTW after a wide range of unintentional injuries in the UK has not been adequately quantified. Aims: To quantify the role of psychological factors including anxiety, depression and post-traumatic distress on RTW following unintentional injuries. Design and Setting: Longitudinal multi-centre prospective study in Nottingham, Bristol, Leicester and Guildford, UK Method: Participants (n=273) were 16-69 year olds admitted to hospital following unintentional injury and, in paid employment prior to injury. They were surveyed at baseline, 1, 2, 4 and 12 months following injury on demographic and injury characteristics, psychological morbidity and RTW status. Associations between demographic, injury and psychological factors and RTW status were quantified using random effects logistic regression. Results: The odds of RTW reduced as depression scores one month post-injury increased (OR 0.87, 95%CI 0.79, 0.95) and as length of hospital stay increased (OR 0.91, 95%CI 0.86, 0.96). Those experiencing threatening life events following injury (OR 0.27, 95%CI 0.10, 0.72) and with higher scores on the crisis social support scale (OR 0.93, 95%CI 0.88, 0.99) had a lower odds of RTW. Multiple imputation analysis found similar results except crisis social support did not remain significant. Conclusion: Primary care professionals can identify patients at risk of delayed RTW who may benefit from management of psychological morbidity and support to RTW

Comparison of coronary heart disease genetic assessment with conventional cardiovascular risk assessment in primary care: reflections on a feasibility study

This study assesses the feasibility of collecting genetic samples and self-reported outcome measures after cardiovascular risk assessment, and presenting the genetic test results to participants. Coronary heart disease (CHD) genetic tests are increasingly available through direct-to-consumer marketing, but their potential clinical impact on cardiovascular risk assessment is unclear. Observational study in 10 British general practices in Central England. A total of 320 individuals, who had completed conventional cardiovascular risk assessment, were offered CHD genetic test, with follow-up outcome questionnaire at eight months for lifestyle change and State-Trait Anxiety. A total of 119 (37%) participants returned genetic test specimens, with over a third reporting family history of CHD in a specified relative; 79 (66.4%) were categorized above-average risk on conventional cardiovascular risk assessment, 65 of whom (82.3%) were only average risk on genetic assessment. The dietary fat questionnaire was poorly completed while study participation was not associated with increased anxiety (mean increase in anxiety score=2.1; 95% CI −0.1–4.3; P=0.06). As a feasibility study, over a third of individuals offered genetic testing in primary care, as part of CVD risk assessment, took up the offer. Although intervention did not appear to increase anxiety, this needs further evaluation. To improve generalizability and effect size, future studies should actively engage individuals from wider socio-economic backgrounds who may not have already contemplated lifestyle change. The current research suggests general practitioners will face the clinical challenge of patients presenting with direct-to-consumer genetic results that are inconsistent with conventional cardiovascular risk assessment

Risk of bias assessments in individual participant data meta-analyses of test accuracy and prediction models:a review shows improvements are needed

OBJECTIVES: Risk of bias assessments are important in meta-analyses of both aggregate and individual participant data (IPD). There is limited evidence on whether and how risk of bias of included studies or datasets in IPD meta-analyses (IPDMAs) is assessed. We review how risk of bias is currently assessed, reported, and incorporated in IPDMAs of test accuracy and clinical prediction model studies and provide recommendations for improvement.STUDY DESIGN AND SETTING: We searched PubMed (January 2018-May 2020) to identify IPDMAs of test accuracy and prediction models, then elicited whether each IPDMA assessed risk of bias of included studies and, if so, how assessments were reported and subsequently incorporated into the IPDMAs.RESULTS: Forty-nine IPDMAs were included. Nineteen of 27 (70%) test accuracy IPDMAs assessed risk of bias, compared to 5 of 22 (23%) prediction model IPDMAs. Seventeen of 19 (89%) test accuracy IPDMAs used Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2), but no tool was used consistently among prediction model IPDMAs. Of IPDMAs assessing risk of bias, 7 (37%) test accuracy IPDMAs and 1 (20%) prediction model IPDMA provided details on the information sources (e.g., the original manuscript, IPD, primary investigators) used to inform judgments, and 4 (21%) test accuracy IPDMAs and 1 (20%) prediction model IPDMA provided information or whether assessments were done before or after obtaining the IPD of the included studies or datasets. Of all included IPDMAs, only seven test accuracy IPDMAs (26%) and one prediction model IPDMA (5%) incorporated risk of bias assessments into their meta-analyses. For future IPDMA projects, we provide guidance on how to adapt tools such as Prediction model Risk Of Bias ASsessment Tool (for prediction models) and QUADAS-2 (for test accuracy) to assess risk of bias of included primary studies and their IPD.CONCLUSION: Risk of bias assessments and their reporting need to be improved in IPDMAs of test accuracy and, especially, prediction model studies. Using recommended tools, both before and after IPD are obtained, will address this.</p