Development and external validation study combining existing models and recent data into an up-to-date prediction model for evaluating kidneys from older deceased donors for transplantation

With a rising demand for kidney transplantation, reliable pre-transplant assessment of organ quality becomes top priority. In clinical practice, physicians are regularly in doubt whether suboptimal kidney offers from older donors should be accepted. Here, we externally validate existing prediction models in a European population of older deceased donors, and subsequently developed and externally validated an adverse outcome prediction tool. Recipients of kidney grafts from deceased donors 50 years of age and older were included from the Netherlands Organ Transplant Registry (NOTR) and United States organ transplant registry from 2006-2018. The predicted adverse outcome was a composite of graft failure, death or chronic kidney disease stage 4 plus within one year after transplantation, modelled using logistic regression. Discrimination and calibration were assessed in internal, temporal and external validation. Seven existing models were validated with the same cohorts. The NOTR development cohort contained 2510 patients and 823 events. The temporal validation within NOTR had 837 patients and the external validation used 31987 patients in the United States organ transplant registry. Discrimination of our full adverse outcome model was moderate in external validation (C-statistic 0.63), though somewhat better than discrimination of the seven existing prediction models (average C-statistic 0.57). The model's calibration was highly accurate. Thus, since existing adverse outcome kidney graft survival models performed poorly in a population of older deceased donors, novel models were developed and externally validated, with maximum achievable performance in a population of older deceased kidney donors. These models could assist transplant clinicians in deciding whether to accept a kidney from an older donor

Prediction meets causal inference: the role of treatment in clinical prediction models

In this paper we study approaches for dealing with treatment when developing a clinical prediction model. Analogous to the estimand framework recently proposed by the European Medicines Agency for clinical trials, we propose a ‘predictimand’ framework of different questions that may be of interest when predicting risk in relation to treatment started after baseline. We provide a formal definition of the estimands matching these questions, give examples of settings in which each is useful and discuss appropriate estimators including their assumptions. We illustrate the impact of the predictimand choice in a dataset of patients with end-stage kidney disease. We argue that clearly defining the estimand is equally important in prediction research as in causal inference

Considerable Variability Among Transplant Nephrologists in Judging Deceased Donor Kidney Offers

Introduction: Transplant clinicians may disagree on whether or not to accept a deceased donor kidney offer. We investigated the interobserver variability between transplant nephrologists regarding organ acceptance and whether the use of a prediction model impacted their decisions.Methods: We developed an observational online survey with 6 real-life cases of deceased donor kidneys offered to a waitlisted recipient. Per case, nephrologists were asked to estimate the risk of adverse outcome and whether they would accept the offer for this patient, or for a patient of their own choice, and how certain they felt. These questions were repeated after revealing the risk of adverse outcome, calculated by a validated prediction model. Results: Sixty Dutch nephrologists completed the survey. The intraclass correlation coefficient of their estimated risk of adverse outcome was poor (0.20, 95% confidence interval [CI] 0.08–0.62). Interobserver agreement of the decision on whether or not to accept the kidney offer was also poor (Fleiss kappa 0.13, 95% CI 0.129–0.130). The acceptance rate before and after providing the outcome of the prediction model was significantly influenced in 2 of 6 cases. Acceptance rates varied considerably among transplant centers. Conclusion: In this study, the estimated risk of adverse outcome and subsequent decision to accept a suboptimal donor kidney varied greatly among transplant nephrologists. The use of a prediction model could influence this decision and may enhance nephrologists’ certainty about their decision.</p

Prediction versus aetiology: common pitfalls and how to avoid them

Prediction research is a distinct field of epidemiologic research, which should be clearly separated from aetiological research. Both prediction and aetiology make use of multivariable modelling, but the underlying research aim and interpretation of results are very different. Aetiology aims at uncovering the causal effect of a specific risk factor on an outcome, adjusting for confounding factors that are selected based on pre-existing knowledge of causal relations. In contrast, prediction aims at accurately predicting the risk of an outcome using multiple predictors collectively, where the final prediction model is usually based on statistically significant, but not necessarily causal, associations in the data at hand. In both scientific and clinical practice, however, the two are often confused, resulting in poor-quality publications with limited interpretability and applicability. A major problem is the frequently encountered aetiological interpretation of prediction results, where individual variables in a prediction model are attributed causal meaning. This article stresses the differences in use and interpretation of aetiological and prediction studies, and gives examples of common pitfall

Appraising prediction research: a guide and meta-review on bias and applicability assessment using the Prediction model Risk Of Bias ASsessment Tool (PROBAST)

Over the past few years, a large number of prediction models have been published, often of poor methodological quality. Seemingly objective and straightforward, prediction models provide a risk estimate for the outcome of interest, usually based on readily available clinical information. Yet, using models of substandard methodological rigour, especially without external validation, may result in incorrect risk estimates and consequently misclassification. To assess and combat bias in prediction research the prediction model risk of bias assessment tool (PROBAST) was published in 2019. This risk of bias (ROB) tool includes four domains and 20 signalling questions highlighting methodological flaws, and provides guidance in assessing the applicability of the model. In this paper, the PROBAST will be discussed, along with an in-depth review of two commonly encountered pitfalls in prediction modelling that may induce bias: overfitting and composite endpoints. We illustrate the prevalence of potential bias in prediction models with a meta-review of 50 systematic reviews that used the PROBAST to appraise their included studies, thus including 1510 different studies on 2104 prediction models. All domains showed an unclear or high ROB; these results were markedly stable over time, highlighting the urgent need for attention on bias in prediction research. This article aims to do just that by providing (1) the clinician with tools to evaluate the (methodological) quality of a clinical prediction model, (2) the researcher working on a review with methods to appraise the included models, and (3) the researcher developing a model with suggestions to improve model quality

TRIPOD statement: a preliminary pre-post analysis of reporting and methods of prediction models

Objectives To assess the difference in completeness of reporting and methodological conduct of published prediction models before and after publication of the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement.Methods In the seven general medicine journals with the highest impact factor, we compared the completeness of the reporting and the quality of the methodology of prediction model studies published between 2012 and 2014 (pre-TRIPOD) with studies published between 2016 and 2017 (post-TRIPOD). For articles published in the post-TRIPOD period, we examined whether there was improved reporting for articles (1) citing the TRIPOD statement, and (2) published in journals that published the TRIPOD statement.Results A total of 70 articles was included (pre-TRIPOD: 32, post-TRIPOD: 38). No improvement was seen for the overall percentage of reported items after the publication of the TRIPOD statement (pre-TRIPOD 74%, post-TRIPOD 76%, 95% CI of absolute difference: −4% to 7%). For the individual TRIPOD items, an improvement was seen for 16 (44%) items, while 3 (8%) items showed no improvement and 17 (47%) items showed a deterioration. Post-TRIPOD, there was no improved reporting for articles citing the TRIPOD statement, nor for articles published in journals that published the TRIPOD statement. The methodological quality improved in the post-TRIPOD period. More models were externally validated in the same article (absolute difference 8%, post-TRIPOD: 39%), used measures of calibration (21%, post-TRIPOD: 87%) and discrimination (9%, post-TRIPOD: 100%), and used multiple imputation for handling missing data (12%, post-TRIPOD: 50%).Conclusions Since the publication of the TRIPOD statement, some reporting and methodological aspects have improved. Prediction models are still often poorly developed and validated and many aspects remain poorly reported, hindering optimal clinical application of these models. Long-term effects of the TRIPOD statement publication should be evaluated in future studies

Prediction models for the mortality risk in chronic dialysis patients: A systematic review and independent external validation study

Objective: In medicine, many more prediction models have been developed than are implemented or used in clinical practice. These models cannot be recommended for clinical use before external validity is established. Though various models to predict mortality in dialysis patients have been published, very few have been validated and none are used in routine clinical practice. The aim of the current study was to identify existing models for predicting mortality in dialysis patients through a review and subsequently to externally validate these models in the same large independent patient cohort, in order to assess and compare their predictive capacities. Methods: A systematic review was performed following the preferred reporting items for systematic reviews and meta-analyses (PRISMA) guidelines. To account for missing data, multiple imputation was performed. The original prediction formulae were extracted from selected studies. The probability of death per model was calculated for each individual within the Netherlands Cooperative Study on the Adequacy of Dialysis (NECOSAD). The predictive performance of the models was assessed based on their discrimination and calibration. Results: In total, 16 articles were included in the systematic review. External validation was performed in 1,943 dialysis patients from NECOSAD for a total of seven models. The models performed moderately to well in terms of discrimination, with C-statistics ranging from 0.710 (interquartile range 0.708-0.711) to 0.752 (interquartile range 0.750-0.753) for a time frame of 1 year. According to the calibration, most models overestimated the probability of death. Conclusion: Overall, the performance of the models was poorer in the external validation than in the original population, affirming the importance of external validation. Floege et al’s models showed the highest predictive performance. The present study is a step forward in the use of a prediction model as a useful tool for nephrologists, using evidence-based medicine that combines individual clinical expertise, patients’ choices, and the best available external evidence

Renal function decline in older men and women with advanced chronic kidney disease-results from the EQUAL study

Introduction: Understanding the mechanisms underlying the differences in renal decline between men and women may improve sex-specific clinical monitoring and management. To this end, we aimed to compare the slope of renal function decline in older men and women in chronic kidney disease (CKD) Stages 4 and 5, taking into account informative censoring related to the sex-specific risks of mortality and dialysis initiation. Methods: The European QUALity Study on treatment in advanced CKD (EQUAL) study is an observational prospective cohort study in Stages 4 and 5 CKD patients ≥65 years not on dialysis. Data on clinical and demographic patient characteristics were collected between April 2012 and December 2018. Estimated glomerular filtration rate (eGFR) was calculated using the CKD Epidemiology Collaboration equation. eGFR trajectory by sex was modelled using linear mixed models, and joint models were applied to deal with informative censoring. Results: We included 7801 eGFR measurements in 1682 patients over a total of 2911 years of follow-up. Renal function declined by 14.0% [95% confidence interval (CI) 12.9-15.1%] on average each year. Renal function declined faster in men (16.2%/year, 95% CI 15.9-17.1%) compared with women (9.6%/year, 95% CI 6.3-12.1%), which remained largely unchanged after accounting for various mediators and for informative censoring due to mortality and dialysis initiation. Diabetes was identified as an important determinant of renal decline specifically in women. Conclusion: In conclusion, renal function declines faster in men compared with women, which remained similar after adjustment for mediators and despite a higher risk of informative censoring in men. We demonstrate a disproportional negative impact of diabetes specifically in women