Appraising prediction research: a guide and meta-review on bias and applicability assessment using the Prediction model Risk Of Bias ASsessment Tool (PROBAST)

Over the past few years, a large number of prediction models have been published, often of poor methodological quality. Seemingly objective and straightforward, prediction models provide a risk estimate for the outcome of interest, usually based on readily available clinical information. Yet, using models of substandard methodological rigour, especially without external validation, may result in incorrect risk estimates and consequently misclassification. To assess and combat bias in prediction research the prediction model risk of bias assessment tool (PROBAST) was published in 2019. This risk of bias (ROB) tool includes four domains and 20 signalling questions highlighting methodological flaws, and provides guidance in assessing the applicability of the model. In this paper, the PROBAST will be discussed, along with an in-depth review of two commonly encountered pitfalls in prediction modelling that may induce bias: overfitting and composite endpoints. We illustrate the prevalence of potential bias in prediction models with a meta-review of 50 systematic reviews that used the PROBAST to appraise their included studies, thus including 1510 different studies on 2104 prediction models. All domains showed an unclear or high ROB; these results were markedly stable over time, highlighting the urgent need for attention on bias in prediction research. This article aims to do just that by providing (1) the clinician with tools to evaluate the (methodological) quality of a clinical prediction model, (2) the researcher working on a review with methods to appraise the included models, and (3) the researcher developing a model with suggestions to improve model quality.Clinical epidemiolog

Renal function decline in older men and women with advanced CKD:Results from the EQUAL study

INTRODUCTION: Understanding the mechanisms underlying the differences in renal decline between men and women may improve sex-specific clinical monitoring and management. To this end, we aimed to compare the slope of renal function decline in older men and women in chronic kidney disease (CKD) Stages 4 and 5, taking into account informative censoring related to the sex-specific risks of mortality and dialysis initiation. METHODS: The European QUALity Study on treatment in advanced CKD (EQUAL) study is an observational prospective cohort study in Stages 4 and 5 CKD patients ≥65 years not on dialysis. Data on clinical and demographic patient characteristics were collected between April 2012 and December 2018. Estimated glomerular filtration rate (eGFR) was calculated using the CKD Epidemiology Collaboration equation. eGFR trajectory by sex was modelled using linear mixed models, and joint models were applied to deal with informative censoring. RESULTS: We included 7801 eGFR measurements in 1682 patients over a total of 2911 years of follow-up. Renal function declined by 14.0% [95% confidence interval (CI) 12.9–15.1%] on average each year. Renal function declined faster in men (16.2%/year, 95% CI 15.9–17.1%) compared with women (9.6%/year, 95% CI 6.3–12.1%), which remained largely unchanged after accounting for various mediators and for informative censoring due to mortality and dialysis initiation. Diabetes was identified as an important determinant of renal decline specifically in women. CONCLUSION: In conclusion, renal function declines faster in men compared with women, which remained similar after adjustment for mediators and despite a higher risk of informative censoring in men. We demonstrate a disproportional negative impact of diabetes specifically in women

Renal function decline in older men and women with advanced chronic kidney disease-results from the EQUAL study

Introduction. Understanding the mechanisms underlying the differences in renal decline between men and women may improve sex-specific clinical monitoring and management. To this end, we aimed to compare the slope of renal function decline in older men and women in chronic kidney disease (CKD) Stages 4 and 5, taking into account informative censoring related to the sex-specific risks of mortality and dialysis initiation.Methods. The European QUALity Study on treatment in advanced CKD (EQUAL) study is an observational prospective cohort study in Stages 4 and 5 CKD patients >= 65years not on dialysis. Data on clinical and demographic patient characteristics were collected between April 2012 and December 2018. Estimated glomerular filtration rate (eGFR) was calculated using the CKD Epidemiology Collaboration equation. eGFR trajectory by sex was modelled using linear mixed models, and joint models were applied to deal with informative censoring.Results. We included 7801 eGFR measurements in 1682 patients over a total of 2911years of follow-up. Renal function declined by 14.0% [95% confidence interval (CI) 12.9-15.1%] on average each year. Renal function declined faster in men (16.2%/year, 95% CI 15.9-17.1%) compared with women (9.6%/year, 95% CI 6.3-12.1%), which remained largely unchanged after accounting for various mediators and for informative censoring due to mortality and dialysis initiation. Diabetes was identified as an important determinant of renal decline specifically in women.Conclusion. In conclusion, renal function declines faster in men compared with women, which remained similar after adjustment for mediators and despite a higher risk of informative censoring in men. We demonstrate a disproportional negative impact of diabetes specifically in women.Clinical epidemiolog

Kidney failure prediction models: a comprehensive external validation study in patients with advanced CKD

Background: Various prediction models have been developed to predict the risk of kidney failure in patients with CKD. However, guideline-recommended models have yet to be compared head to head, their validation in patients with advanced CKD is lacking, and most do not account for competing risks.Methods: To externally validate 11 existing models of kidney failure, taking the competing risk of death into account, we included patients with advanced CKD from two large cohorts: the European Quality Study (EQUAL), an ongoing European prospective, multicenter cohort study of older patients with advanced CKD, and the Swedish Renal Registry (SRR), an ongoing registry of nephrology-referred patients with CKD in Sweden. The outcome of the models was kidney failure (defined as RRT-treated ESKD). We assessed model performance with discrimination and calibration.Results: The study included 1580 patients from EQUAL and 13,489 patients from SRR. The average c statistic over the 11 validated models was 0.74 in EQUAL and 0.80 in SRR, compared with 0.89 in previous validations. Most models with longer prediction horizons overestimated the risk of kidney failure considerably. The 5-year Kidney Failure Risk Equation (KFRE) overpredicted risk by 10%-18%. The four- and eight-variable 2-year KFRE and the 4-year Grams model showed excellent calibration and good discrimination in both cohorts.Conclusions: Some existing models can accurately predict kidney failure in patients with advanced CKD. KFRE performed well for a shorter time frame (2 years), despite not accounting for competing events. Models predicting over a longer time frame (5 years) overestimated risk because of the competing risk of death. The Grams model, which accounts for the latter, is suitable for longer-term predictions (4 years).Clinical epidemiolog

Development and external validation study combining existing models and recent data into an up-to-date prediction model for evaluating kidneys from older deceased donors for transplantation

With a rising demand for kidney transplantation, reliable pre-transplant assessment of organ quality becomes top priority. In clinical practice, physicians are regularly in doubt whether suboptimal kidney offers from older donors should be accepted. Here, we externally validate existing prediction models in a European population of older deceased donors, and subsequently developed and externally validated an adverse outcome prediction tool. Recipients of kidney grafts from deceased donors 50 years of age and older were included from the Netherlands Organ Transplant Registry (NOTR) and United States organ transplant registry from 2006-2018. The predicted adverse outcome was a composite of graft failure, death or chronic kidney disease stage 4 plus within one year after transplantation, modelled using logistic regression. Discrimination and calibration were assessed in internal, temporal and external validation. Seven existing models were validated with the same cohorts. The NOTR development cohort contained 2510 patients and 823 events. The temporal validation within NOTR had 837 patients and the external validation used 31987 patients in the United States organ transplant registry. Discrimination of our full adverse outcome model was moderate in external validation (C-statistic 0.63), though somewhat better than discrimination of the seven existing prediction models (average C-statistic 0.57). The model's calibration was highly accurate. Thus, since existing adverse outcome kidney graft survival models performed poorly in a population of older deceased donors, novel models were developed and externally validated, with maximum achievable performance in a population of older deceased kidney donors. These models could assist transplant clinicians in deciding whether to accept a kidney from an older donor.Clinical epidemiolog

A systematic review and external validation of stroke prediction models demonstrates poor performance in dialysis patients

OBJECTIVE: To systematically review and externally assess the predictive performance of models for ischemic stroke in incident dialysis patients. STUDY DESIGN AND SETTING: Two reviewers systematically searched and selected ischemic stroke models. Risk of bias was assessed with the PROBAST. Predictive performance was evaluated within NECOSAD, a large prospective multicentre cohort of incident dialysis patients. For discrimination, c-statistics were calculated; calibration was assessed by plotting predicted and observed probabilities for stroke, and calibration-in-the-large. RESULTS: 77 prediction models for stroke were identified, of which 15 were validated. Risk of bias was high, with all of these models scoring high risk in one or more domains. In NECOSAD, of the 1955 patients 127 (6.5%) suffered an ischemic stroke during the follow-up of 2.5 years. Compared to the original studies, most models performed worse with all models showing poor calibration and discriminative abilities (c-statistics ranging from 0.49 to 0.66). The Framingham showed reasonable calibration, however with a c-statistic of 0.57 (95% CI 0.50-0.63), the discrimination was poor. CONCLUSION: This external validation demonstrates the weak predictive performance of ischemic stroke models in incident dialysis patients. Instead of using these models in this fragile population, either existing models should be updated, or novel models should be developed and validated

External validation of prognostic models: what, why, how, when and where?

Prognostic models that aim to improve the prediction of clinical events, individualized treatment and decision-making are increasingly being developed and published. However, relatively few models are externally validated and validation by independent researchers is rare. External validation is necessary to determine a prediction model's reproducibility and generalizability to new and different patients. Various methodological considerations are important when assessing or designing an external validation study. In this article, an overview is provided of these considerations, starting with what external validation is, what types of external validation can be distinguished and why such studies are a crucial step towards the clinical implementation of accurate prediction models. Statistical analyses and interpretation of external validation results are reviewed in an intuitive manner and considerations for selecting an appropriate existing prediction model and external validation population are discussed. This study enables clinicians and researchers to gain a deeper understanding of how to interpret model validation results and how to translate these results to their own patient population.Clinical epidemiolog

Performance of bleeding risk scores in dialysis patients

Background Bleeding risk scores have been created to identify patients with an increased bleeding risk, which could also be useful in dialysis patients. However, the predictive performances of these bleeding risk scores in dialysis patients are unknown. Therefore, the aim of this study was to validate existing bleeding risk scores in dialysis patients.Methods A cohort of 1745 incident dialysis patients was prospectively followed for 3 years during which bleeding events were registered. We evaluated the discriminative performance of the Hypertension, Abnormal kidney and liver function, Stroke, Bleeding, Labile INR, Elderly and Drugs or alcohol (HASBLED), the AnTicoagulation and Risk factors In Atrial fibrillation (ATRIA), the Hepatic or kidney disease, Ethanol abuse, Malignancy, Older age, Reduced platelet count or Reduced platelet function, Hypertension, Anaemia, Genetic factors, Excessive fall risk and Stroke (HEMORR2HAGES) and the Outcomes Registry for Better Informed Treatment (ORBIT) bleeding risk scores by calculating C-statistics with 95% confidence intervals (CI). In addition, calibration was evaluated by comparing predicted and observed risks.Results Of the 1745 dialysis patients, 183 patients had a bleeding event, corresponding to an incidence rate of 5.23/100 person-years. The HASBLED [C-statistic of 0.58 (95% CI 0.54-0.62)], ATRIA [C-statistic of 0.55 (95% CI 0.51-0.60)], HEMORR2HAGES [C-statistic of 0.56 (95% CI 0.52-0.61)] and ORBIT [C-statistic of 0.56 (95% CI 0.52-0.61)] risk scores had poor discriminative performances in dialysis patients. Furthermore, the calibration analyses showed that patients with a low risk of bleeding according to the HASBLED, ATRIA, HEMORR2HAGES and ORBIT bleeding risk scores had higher incidence rates for bleeding in our cohort than predicted.Conclusions The HASBLED, ATRIA, HEMORR2HAGES and ORBIT bleeding risk scores had poor predictive abilities in dialysis patients. Therefore, these bleeding risk scores may not be useful in this population.Clinical epidemiolog

Removing race from the CKD-EPI equation and its impact on prognosis in a predominantly White European population

Background While American nephrology societies recommend using the 2021 Chronic Kidney Disease Epidemiology Collaboration (CKD-EPI) estimated glomerular filtration rate (eGFR) equation without a Black race coefficient, it is unknown how this would impact disease distribution, prognosis and kidney failure risk prediction in predominantly White non-US populations. Methods We studied 1.6 million Stockholm adults with serum/plasma creatinine measurements between 2007 and 2019. We calculated changes in eGFR and reclassification across KDIGO GFR categories when changing from the 2009 to 2021 CKD-EPI equation; estimated associations between eGFR and the clinical outcomes kidney failure with replacement therapy (KFRT), (cardiovascular) mortality and major adverse cardiovascular events using Cox regression; and investigated prognostic accuracy (discrimination and calibration) of both equations within the Kidney Failure Risk Equation. Results Compared with the 2009 equation, the 2021 equation yielded a higher eGFR by a median [interquartile range (IQR)] of 3.9 (2.9-4.8) mL/min/1.73 m(2), which was larger at older age and for men. Consequently, 9.9% of the total population and 36.2% of the population with CKD G3a-G5 was reclassified to a higher eGFR category. Reclassified individuals exhibited a lower risk of KFRT, but higher risks of all-cause/cardiovascular death and major adverse cardiovascular events, compared with non-reclassified participants of similar eGFR. eGFR by both equations strongly predicted study outcomes, with equal discrimination and calibration for the Kidney Failure Risk Equation. Conclusions Implementing the 2021 CKD-EPI equation in predominantly White European populations would raise eGFR by a modest amount (larger at older age and in men) and shift a major proportion of CKD patients to a higher eGFR category. eGFR by both equations strongly predicted outcomes.Clinical epidemiolog

Kidney Failure Prediction Models: A Comprehensive External Validation Study in Patients with Advanced CKD

Background Various prediction models have been developed to predict the risk of kidney failure in patients with CKD. However, guideline-recommended models have yet to be compared head to head, their validation in patients with advanced CKD is lacking, and most do not account for competing risks. Methods To externally validate 11 existing models of kidney failure, taking the competing risk of death into account, we included patients with advanced CKD from two large cohorts: the European Quality Study (EQUAL), an ongoing European prospective, multicenter cohort study of older patients with advanced CKD, and the Swedish Renal Registry (SRR), an ongoing registry of nephrology-referred patients with CKD in Sweden. The outcome of the models was kidney failure (defined as RRT-treated ESKD). We assessed model performance with discrimination and calibration. Results The study included 1580 patients from EQUAL and 13,489 patients from SRR. The average c statistic over the 11 validated models was 0.74 in EQUAL and 0.80 in SRR, compared with 0.89 in previous validations. Most models with longer prediction horizons overestimated the risk of kidney failure considerably. The 5-year Kidney Failure Risk Equation (KFRE) overpredicted risk by 10%-8%. The four- and eight-variable 2-year KFRE and the 4-year Grams model showed excellent calibration and good discrimination in both cohorts. Conclusions Some existing models can accurately predict kidney failure in patients with advanced CKD. KFRE performed well for a shorter time frame (2 years), despite not accounting for competing events. Models predicting over a longer time frame (5 years) overestimated risk because of the competing risk of death. The Grams model, which accounts for the latter, is suitable for longer-term predictions (4 years)