Funder: UK National Institute for Health Research (NIHR) through the Cambridge Biomedical Research Centre (BRC

Cowan, Sarah L.

Goudie, Robert J. B.

Preller, Jacobus

Apollo (Cambridge)

RESEARCH LETTER Open AccessEvaluation of the E-PRE-DELIRIC predictionmodel for ICU delirium: a retrospectivevalidation in a UK general ICUSarah L. Cowan1, Jacobus Preller1 and Robert J. B. Goudie2*Keywords: Clinical scoring systems, Critical care, Delirium, Intensive care, Prediction model, ValidationIntroductionE-PRE-DELIRIC is a point-of-admission ICU deliriumrisk prediction tool [1], with reported good or moderateperformance [2–4]. In this study, we assessed its per-formance in a large UK teaching hospital general ICUusing routinely collected data, as approved by the localResearch Data Governance Committee.MethodsWe retrospectively analysed data for 2445 consecutiveICU admissions (November 2014 to June 2017). Patientswere routinely assessed for delirium, using twice dailyConfusion Assessment Method for the ICU (CAM-ICU)assessment [5]. As in previous E-PRE-DELIRIC studies[1–4], delirium was defined as any positive CAM-ICUassessment or antipsychotic initiation while on ICU.We adopted the original E-PRE-DELIRIC exclusioncriteria [1], excluding 683 ICU admissions for ICU stay< 24 h (425 admissions), incomplete CAM-ICU data(152), delirium on admission (50), comatose throughoutentire ICU stay (47), and age under 18 (9). Sixteen ad-missions were excluded due to missing E-PRE-DELIRICcomponents; 1746 admissions (1569 unique patients)remained for analysis; this 71.4% inclusion rate is con-sistent with previous studies (Table 1).Results and discussionSeven hundred sixty-three delirium cases were identified(43.7% of ICU admissions), a higher incidence than re-ported previously (Table 1). This is likely due to differ-ences in the study population compared to previousstudies: more patients were classified as urgent, themean APACHE II score was higher, and median lengthof stay (LoS) was longer (Table 1).The mean E-PRE-DELIRIC score was 0.269 (Q1–Q3;0.154–0.371). The histogram of E-PRE-DELIRIC scoresshows extensive overlap between patients who did anddid not develop delirium (Fig. 1a). The receiver operatorcharacteristic (ROC) curve (Fig. 1b) and the precision-recall (PR) curve (Fig. 1c), showing precision (positivepredictive value (PPV)) against recall (sensitivity), bothindicate moderate-to-poor discriminative performance.The area under the ROC (AUROC) was 0.628 (95% CI0.602–0.653). The area under the PR curve (AUPRC)was 0.534. For sensitivity > 0.1, PPV was between 0.437and 0.585, indicating only around half of the patientspredicted to develop delirium actually did, in a popula-tion with 43.7% incidence. Refitting the E-PRE-DELIRIClogistic regression model to our data hardly improveddiscrimination: AUROC was 0.648 (95% CI 0.622–0.673)and AUPRC was 0.566.The calibration plot, of predicted risk against observeddelirium rate, shows the risk of delirium is considerablyunderestimated, especially in patients with predicted riskof delirium less than 0.5 (Fig. 1d). Poor calibration iscorroborated by the calibration slope model logit(prob-ability of delirium) = alpha + beta ×logit(p), where p is© The Author(s). 2020 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License,which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you giveappropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate ifchanges were made. The images or other third party material in this article are included in the article's Creative Commonslicence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commonslicence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtainpermission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to thedata made available in this article, unless otherwise stated in a credit line to the data.* Correspondence: robert.goudie@mrc-bsu.cam.ac.uk2MRC Biostatistics Unit, School of Clinical Medicine, University of Cambridge,Cambridge CB2 0SR, UKFull list of author information is available at the end of the articleCowan et al. Critical Care          (2020) 24:123 https://doi.org/10.1186/s13054-020-2838-2Table 1 Patient characteristics in this study, the E-PRE-DELIRIC development dataset [1] and other validation studies [2–4]Factor This study Development dataset [1] DECISION study [2, 3] Green et al. [4]Admissions during study period, n 2445 – 2802 803Included in analysis, n (%) 1746 (71.4) 1962 (–) 2178 (77.7) 455 (56.7)Delirium, n (%) 763 (43.7) 481 (24.5) 466 (21.4) 160 (35.2)Age (years), mean (Q1–Q3, min/max) 58.6 (47.0–71.8, 18/94) 61.7 (53–74, 18/95) 62.1 (–) 66.7 (49.0–77.3, –/–)Male, n (%) 1010 (57.8) 1166 (59.4) 1324 (60.8) 241 (53.0)Admission category, n (%)Surgery 813 (46.6) 1019 (51.9) 1079 (49.5) –Medicine 837 (47.9) 683 (34.8) 859 (39.3) –Trauma 33 (1.9) 90 (4.6) 86 (4.0) –Neurology/neurosurgery 63 (3.6) 170 (8.7) 157 (7.2) –Urgent admission, n (%) 1534 (87.9) 1163 (59.3) 1345 (61.8) –APACHE II 20.0 (mean) – 17.4 (mean) 16 (median)ICU LoS (days), median (Q1–Q3, min/max) 4.5 (2.4–10.0, 1.0/184.0) 2.0 (1–6, 1/133) 3.0 (2–6, 1/96) 2.6 (1.5–4.4, –/–)ICU mortality, n (%) 210 (12.0) – – 17 (3.7)– indicates the figure was not reportedFig. 1 a Histogram of predicted risk of delirium by delirium status. b Receiver operator characteristic plot, with labels indicating the correspondingthreshold and the dashed line indicating the line of no discrimination. c Precision-recall plot, with the 43.7% observed incidence indicated bythe dashed line. d Calibration plot (with 95% CI), by tenths of predicted risk, with the dashed line indicating perfect calibrationCowan et al. Critical Care          (2020) 24:123 Page 2 of 3the E-PRE-DELIRIC score [6]. The estimated slopebeta = 0.58 (95% CI 0.46–0.71) is significantly below 1,indicating the predicted probabilities are overly variable;and the estimated intercept alpha = 0.84 (95% CI 0.74–0.95) is significantly above 0 when fixing beta = 1, indi-cating the predicted probabilities are predominantly toolow. E-PRE-DELIRIC is particularly poorly calibrated forthe surgical patients in the study, many of whom havemajor intraabdominal pathology: those with predictedrisk < 10% had an observed incidence of 26%.Of 763 delirium cases, 563 were CAM-ICU positive and200 were included due to antipsychotic initiation. When in-cluding only CAM-ICU-positive delirium, calibration wasimproved (alpha = 0.29) but remained overly variable (beta =0.52), while discrimination was similar (AUROC 0.615;AUPRC 0.396, with 32.2% observed incidence).While E-PRE-DELIRIC is intended as a point-of-admission score, some of its exclusion criteria are retro-spective (LoS; CAM-ICU completeness; comatosethroughout). To assess real-world performance, we re-peated our analysis without these criteria. The AUROC(0.615) and AUPRC (0.423, with 35.0% observed inci-dence) remained similar.ConclusionIn this population, the E-PRE-DELIRIC score is not asdiscriminative or as well calibrated as previously re-ported. PPV was only slightly higher than delirium inci-dence, meaning the utility of E-PRE-DELIRIC forguiding clinical decision-making in this population islimited.AbbreviationsAUROC: Area under the ROC curve; AUPRC: Area under the PR curve; CAM-ICU: Confusion Assessment Method for the ICU; E-PRE-DELIRIC: Earlyprediction model for delirium in ICU patients; ICU: Intensive care unit;LoS: Length of stay; PPV: Positive predictive value; PR: Precision-recall;ROC: Receiver operator characteristicAcknowledgementsThis study was supported by the UK National Institute for Health Research(NIHR) through the Cambridge Biomedical Research Centre (BRC), with dataprovided by Cambridge Clinical Informatics (Led by Drs. Afzal Chaudhry andLydia Drumright). RJBG was supported by the UK Medical Research Council[programme code MC_UU_00002/2]. We are grateful to Vince Taylor for hiscareful work extracting data for this study.Authors’ contributionsSLC interpreted and guided the statistical analyses and drafted the manuscript.JP conceived the study, interpreted the study results, and substantively revisedthe manuscript. RJBG designed the study, conducted and interpreted thestatistical analyses, and substantively revised the manuscript. All authors readand approved the final manuscript.FundingThis study was supported by the UK National Institute for Health Research(NIHR) through the Cambridge Biomedical Research Centre (BRC), with dataprovided by Cambridge Clinical Informatics (led by Drs. Afzal Chaudhry andLydia Drumright). RJBG was supported by the UK Medical Research Council[programme code MC_UU_00002/2]. The funders had no role in the design,collection, analysis, interpretation, or writing of the manuscript.Availability of data and materialsThe data that support the findings of this study are available from CambridgeClinical Informatics, but restrictions apply to the availability of these data, whichwere used under license for the current study, and so are not publicly available.The data are anonymised, but to preserve patient confidentiality and privacy,the Data Use Agreement states that the data cannot be deposited into openaccess repositories of any kind. Anyone wishing to access data must submitand receive approval for access to these data from the Cambridge ClinicalInformatics Research Data Governance Committee.Ethics approval and consent to participateThe use of the anonymous data used in this study was approved by theCambridge Clinical Informatics Research Data Governance Committee.Consent for publicationNot applicable.Competing interestsThe authors declare that they have no competing interests.Author details1Addenbrooke’s Hospital, Cambridge CB2 0QQ, UK. 2MRC Biostatistics Unit,School of Clinical Medicine, University of Cambridge, Cambridge CB2 0SR,UK.Received: 28 January 2020 Accepted: 19 March 2020References1. Wassenaar A, van den Boogaard M, van Achterberg T, Slooter AJC, KuiperMA, Hoogendoorn ME, et al. Multinational development and validation ofan early prediction model for delirium in ICU patients. Intensive Care Med.2015;41:1048–56.2. Wassenaar A, Schoonhoven L, Devlin JW, van Haren FMP, Slooter AJC,Jorens PG, et al. Delirium prediction in the intensive care unit: comparisonof two delirium prediction models. Crit Care. 2018;22:114.3. Wassenaar A, Schoonhoven L, Devlin JW, van Haren FMP, Slooter AJC,Jorens PG, et al. External validation of two models to predict delirium incritically ill adults using either the Confusion Assessment Method-ICU or theIntensive Care Delirium Screening Checklist for delirium assessment. CritCare Med. 2019;47:e827–35.4. Green C, Bonavia W, Toh C, Tiruvoipati R. Prediction of ICU delirium:Validation of current delirium predictive models in routine clinical practice.Crit Care Med. 2019;47:428–35.5. Ely EW, Margolin R, Francis J, May L, Truman B, Dittus R, et al. Evaluation ofdelirium in critically ill patients: validation of the Confusion AssessmentMethod for the Intensive Care Unit (CAM-ICU). Crit Care Med. 2001;29:1370–9.6. Moons KGM, Altman DG, Reitsma JB, Ioannidis JPA, Macaskill P, SteyerbergEW, et al. Transparent Reporting of a multivariable prediction model forIndividual Prognosis Or Diagnosis (TRIPOD): explanation and elaboration.Ann Intern Med. 2015;162:W1–W73.Publisher’s NoteSpringer Nature remains neutral with regard to jurisdictional claims inpublished maps and institutional affiliations.Cowan et al. Critical Care          (2020) 24:123 Page 3 of 3

Evaluation of the E-PRE-DELIRIC prediction model for ICU delirium: a retrospective validation in a UK general ICU

https://www.repository.cam.ac.uk/bitstream/1810/319350/2/13054_2020_Article_2838.pdf

Evaluation of the E-PRE-DELIRIC prediction model for ICU delirium: a retrospective validation in a UK general ICU

Abstract

Similar works

Full text

Available Versions

Apollo (Cambridge)