Decruyenaere, Johan

Van Biesen, Wim

Vanmassenhove, Jill

English

Ghent University Academic Bibliography

Nephrol Dial Transplant (2020) 35: 204–205doi: 10.1093/ndt/gfz226Advance Access publication 28 November 2019Prediction of acute kidney injury using artificial intelligence:are we there yet?Wim Van Biesen 1,2, Jill Vanmassenhove1 and Johan Decruyenaere2,31Renal Division, Ghent University Hospital, Ghent, Belgium, 2Justiﬁable Digital Health Consortium, Ghent University Hospital, Ghent, Belgiumand 3Department of Intensive Care, Ghent University Hospital, Ghent, BelgiumCorrespondence to: Wim Van Biesen; E-mail: Wim.vanbiesen@ugent.beManagement of acute kidney injury (AKI) is suboptimal andoften opportunities for AKI prevention are missed [1]. AKI isfrequently associated with other underlying conditions and will,in most cases, be handled by non-nephrologists in the hospitalsetting who are less experienced in diagnosing AKI. Thishas made the development of detection or prediction modelsof AKI a hot topic. Most of these models are restricted to esti-mating the risk for AKI at baseline, e.g. on admission or preop-eratively, and/or in specific settings, e.g. cardiac surgery orsepsis, thus hampering their generalizability [2]. Recently a ma-chine learning-based algorithm was shown to perform as wellas physicians in predicting AKI stages 2–3 on the day ofadmission, albeit in the restricted setting of the intensive careunit (ICU), on the basis of which the algorithm was developed[3]. More advanced prediction models have shown increasedclinical usefulness by allowing continuous, and nearly real-time,risk prediction, taking into account longitudinal patient data,and thereby the dynamic status of the patient, thus making im-plementation of an (automated) electronic alert system highlyattractive [4]. In addition, many models predict only advancedstages of AKI or the need for renal replacement therapy, thusmissing the opportunity to prevent or mitigate AKI.In Nature, Tomasev et al. [5] reported the development of acontinuous prediction model for AKI based on artificial intelli-gence (AI). The authors used longitudinal data from electronichealth records of >700 000 inpatients, as well as outpatients,across all specialties to train a deep learning recurrent neuralnetwork model. The system was trained using not only currentmedical data, but also previous data for up to 2 years before ad-mission, resulting in a flabbergasting 6 billion independent dataentries. For every case, presence or absence of AKI was labelledto allow supervised learning. The resulting model was able topredict AKI in 55.8% of all inpatient cases of diagnosed AKI,with a lead time of up to 48 hours and a ratio of two false alertsfor every true alert.This project illustrates the potential of using AI trained inbig data in medicine. However, it also reveals the limitationsand pitfalls of such an approach and the issues that need furtherresearch and attention.First, the performance of the model is not that impressive.The model has a low sensitivity of 55.8%, meaning half of theAKI episodes are missed. This may be a deliberate choice to in-flate the specificity. In settings with a low prevalence of AKI,even highly performing models will have to strike a balance be-tween high sensitivity, and thus missing fewer cases, and specif-icity, and thus reducing false positives and tackling alert fatigue.Further research is needed to explore which approach will, inclinical practice, result in substantial improvement in outcomes[6]. Of particular note, in a general hospital setting, missingAKI cases might be considered more problematic than alert fa-tigue. However, it is important to consider that low sensitivitymight simply be due to the fact that a large proportion of AKIcases simply cannot be predicted, just like one cannot predictthe side on which a coin will land in a coin toss experiment.These drawbacks are inherent to the laws of probability andhence apply to all diagnostic and prediction models, whethertraditional or machine learning based [7].Tomasev et al. [5] used supervised learning, in which inputfeatures are associated with pre-specified output labels based ona mathematical algorithm. This implies that during training themodel is provided with the correct label for each case and alsoduring training the model learns from its ‘mistakes’ by adaptingthe weights that associate the data with the label in the algo-rithm. Such an approach presumes that the categorizationused is: (i) ‘transparent’ and ‘uniform’, implying everybodyunderstands clearly and unequivocally what the category labelrepresents exactly; (ii) ‘relevant’, with categories making ameaningful distinction between cases; (iii) ‘unique’, meaningthat every case belongs to one, and only one, category and (iv)‘exhaustive’, meaning that all cases that exhibit meaningful dif-ferences can be assigned to a different category. Last, it also pre-sumes that all cases in the training set are labelled correctly.When these assumptions are violated, the resulting algorithmwill achieve a poor performance in practice.Most AKI prediction algorithms claim to be based onKidney Disease: Improving Global Outcomes criteria, but in re-ality they are based only on the creatinine level, and not on theurinary output criterion. This implies AKI predicted by theseVC The Author(s) 2019. Published by Oxford University Press on behalf of ERA-EDTA. All rights reserved. 204EDITORIAL Downloaded from https://academic.oup.com/ndt/article-abstract/35/2/204/5647338 by Ghent University user on 26 March 2020models might differ from AKI as understood in clinical practiceand, as such, the definition of what is predicted by the algorithmis not transparent nor uniquely defined. Importantly, urinaryoutput could represent the cheapest continuously available pre-dictor of AKI we have, and instructing healthcare staff to moni-tor diuresis might be more effective in improving AKImanagement than searching for new AKI biomarkers or devel-oping prediction algorithms for AKI [8].An electronic alert for AKI would have the most added valuein those settings with the highest risk for missing an AKI diag-nosis. However, in a retrospective data set, cases where a diag-nosis of AKI was not considered by the physician are highlylikely to lack the data to ground the diagnosis. One can opt toconsistently label these cases as ‘no AKI’, as in Tomasev et al.’spaper [5]. In this case, the label of ‘no AKI’ would have no rele-vance since it does not distinguish between ‘confirmed no AKI’,as grounded in the available data, and ‘uncertain AKI’, i.e. dataare not available to ground the diagnosis. ‘Uncertain AKI’might represent missed AKI cases or incorrect labelling in thetraining set, thus resulting in false negatives in the test setthrough supervised learning—in this way, cases missed by thephysician will also be missed by the prediction model. Onecould also opt to exclude cases that miss the data to ground thediagnosis. However, this would mean the training set will notinclude cases of ‘easily missed AKI’ and, as a consequence, willnot be exhaustive. The typical pattern of these cases will not berecognized in the test set, resulting in their random classifica-tion as either AKI or no AKI.Furthermore, the model is not only trained in using testresults, but also in the use of metadata, such as whether tests areordered, as well as their timing, regardless of the test results.There is evidence these metadata are more informative than ac-tual test results for predicting outcomes, including survival [9].However, such metadata rely heavily on the expertise of thephysician who will order the test because of a presumed risk ofAKI. This will result in the creation of a vicious circle within themodel. During training the model will blindly (i.e. without anyknowledge about the problem) associate the request of the phy-sician for a test to detect AKI with the later occurrence of AKI.The model will thus generate an alert for a potential AKI prob-lem that the physician already recognized himself.Furthermore, implementing such a model could have otherunexpected ‘side effects’ in daily clinical practice. If a userstrongly trusts in the performance of the electronic alert system,they might choose to ‘wait’ for an alert trigger from the modelbefore taking action, while the model, in fact, ‘needs’ theseactions to be taken first so it can estimate the risk of AKI andtrigger the alert. This may, of course, have detrimental effectson timely AKI recognition. Alternatively, users can order, as adefault setting in practice, all the necessary tests to feed thealgorithm for all patients. However, in this default setting, therelationship between the results of these tests and AKI will shiftdue to a change in the pre-test probability distributions betweenthe training set and the clinical practice data set. The modelmight not be calibrated to this new setting. In addition, it canalso lead to the well-known ‘neural net tank urban legend’ [10],as illustrated in an AI-based chest X-ray diagnosis model [11]in which the diagnosis was largely driven by the type of X-raymachine used, with differences in the machine used in the out-patient setting (low incidence of pneumonia) compared withthe ICU setting (high incidence of pneumonia).Last, an alert trigger needs to be timely to avoid or mitigatethe risk for AKI. Although a 48-hour prediction window wouldseem adequate, not all AKI diagnoses are predicted within 48hours before the onset of AKI by the prediction system. In fact,only 20% of AKI cases are predicted >24 hours before AKI on-set [5]. For the majority of cases, this might prove to be too latefor an effective intervention [2]. In addition, in the case ofAKI, intervention would consist of preventive measures thatshould be provided to all patients, not only to those withimpending AKI.The points mentioned above could unveal the Achilles tendonof this type of prediction models based on retrospective observa-tional datasets, and could jeopardize their clinical usefulness.FUNDINGW.V.B. is supported by a research grant from FondsWetenschappelijk Onderzoek (FWO.OPR.2019.0045.01).CONFLICT OF INTEREST STATEMENTNone declared. The content of this article has not been pub-lished previously in whole or part, also not in abstract format.REFERENCES1. Stewart JA. Adding insult to injury: care of patients with acute kidney injury.Br J Hosp Med (Lond) 2009; 70: 372–3732. Vanmassenhove J, Kielstein J, Jorres A et al. Management of patients at riskof acute kidney injury. Lancet 2017; 389: 2139–21513. Flechet M, Falini S, Bonetti C et al. Machine learning versus physicians’ pre-diction of acute kidney injury in critically ill adults: a prospective evaluationof the AKIpredictor. Crit Care 2019; 23: 2824. Colpaert K, Hoste EA, Steurbaut K et al. Impact of real-time electronic alert-ing of acute kidney injury on therapeutic intervention and progression ofRIFLE class. Crit Care Med 2012; 40: 1164–11705. Tomasev N, Glorot, X, Rae JW et al. A clinically applicable approach to con-tinuous prediction of future acute kidney injury.Nature 2019; 572: 116–1196. Kane-Gill SL, O’Connor MF, Rothschild JM et al. Technologic distractions(part 1): summary of approaches to manage alert quantity with intent to re-duce alert fatigue and suggestions for alert fatigue metrics. Crit Care Med2017; 45: 1481–14887. Shah NH, Milstein A, Bagley, SC. Making machine learning models clini-cally useful. JAMA 2019; 322: 1351–13528. Kolhe NV, Reilly T, Leung J et al. A simple care bundle for use in acute kid-ney injury: a propensity score matched cohort study. Nephrol DialTransplant 2016; 31: 1846–18549. Agniel D, Kohane IS, Weber GM. Biases in electronic health record datadue to processes within the healthcare system: retrospective observationalstudy. BMJ 2018; 361: k147910. The Neural Net Tank Urban Legend. https://www.gwern.net/Tanks (25September 2019, date last accessed)11. Couzin-Frankel J. Medicine contends with how to use artiﬁcial intelligence.Science 2019; 364: 1119–1120Received: 20.8.2019; Editorial decision: 14.10.2019Editorial 205Downloaded from https://academic.oup.com/ndt/article-abstract/35/2/204/5647338 by Ghent University user on 26 March 2020

Prediction of acute kidney injury using artificial intelligence : are we there yet?

https://biblio.ugent.be/publication/8655168/file/8655169.pdf

Prediction of acute kidney injury using artificial intelligence : are we there yet?

Abstract

Similar works

Full text

Available Versions

Ghent University Academic Bibliography