This paper reviews the current state of patient safety and the application of artificial intelligence (AI) techniques to patient safety. This paper defines patient safety broadly, not just inpatient care but across the continuum of care, including diagnostic errors, misdiagnosis, adverse events, injuries, and measurement issues. It outlines the major current uses of AI in patient safety and the relative adoption of these techniques in hospitals and health systems. It also outlines some of the limitations of these AI systems and the challenges with evaluation of these systems. Finally, it outlines the importance of developing a proactive agenda for AI in healthcare that includes marked increased funding of research and evaluation in this area

Classen, David C

Longhurst, Christopher

Thomas, Eric J

English

DigitalCommons@The Texas Medical Center

COMMENT OPENBending the patient safety curve: how much can AI help?David C. Classen 1✉, Christopher Longhurst 2 and Eric J. Thomas3This paper reviews the current state of patient safety and the application of artificial intelligence (AI) techniques to patient safety.This paper defines patient safety broadly, not just inpatient care but across the continuum of care, including diagnostic errors,misdiagnosis, adverse events, injuries, and measurement issues. It outlines the major current uses of AI in patient safety and therelative adoption of these techniques in hospitals and health systems. It also outlines some of the limitations of these AI systemsand the challenges with evaluation of these systems. Finally, it outlines the importance of developing a proactive agenda for AI inhealthcare that includes marked increased funding of research and evaluation in this area.npj Digital Medicine             (2023) 6:2 ; https://doi.org/10.1038/s41746-022-00731-5ARTIFICIAL INTELLIGENCE AND PATIENT SAFETYHealthcare has been through a rapid digitalization process overthe last decade with widespread adoption of electronic healthrecords (EHR) and electronic imaging systems that provide thefoundation for a broad spectrum of artificial intelligence (AI)approaches1–3. Despite this, patient safety challenges, includingmeasurement and prevention of diagnostic errors, medical errors,adverse events, iatrogenic injury, or other preventable harm to apatient, remain unfortunately common3. Current patient safetymeasurement approaches, which are the minimal requirements toimprove safety, are still mostly rooted in the pre digital era withvoluntary reporting of safety incidents and extensive manual rootcause analysis still commonly used, none of which leverage thenew digital infrastructure3. Many studies have shown theseapproaches detect <10% of all safety problems and fail toproactively prevent safety problems2. Their use persists becauseinstitutional patient safety infrastructure is driven in part byregulations, including Medicare requirements, which focus onmanual coding of safety problems using administrative data3.However, Medicare has just recently announced the movement tonew automated EHR-based safety measures for hospital reportingbeginning in 2023, which will catalyze the increasing use of HealthIT based approaches in patient safety measurement.At the same time, AI is increasingly adopted in other industrieslike banking, aviation, finance, and marketing to improveorganizational performance with well documented results. Overthe last five years, peer-reviewed publications, on AI in healthcarehave increased exponentially, but most have been focused onconcepts, development, and initial validation, with few outliningspecific operational uses, and even fewer with outcomesassessment, with a few notable exceptions4. Given this back-ground, we believe it is time to rigorously assess the use of AI inpatient safety and to begin setting an agenda for future research,evaluation, and practice in this field.A recent study evaluated the use of AI predictive analytics in UShealth systems with important findings5. It found that AIpredictive analytics was widely used in these health systems, with64% have a dedicated team or individual responsible for these AIalgorithms and that the most common areas of focus were sepsisidentification and hospital readmission risk prediction5. As wellthis study found that almost half of these health systems builttheir own AI predictive algorithms. This study found similarfindings to a survey we did with a group of physician leaders ininformation technology of US hospitals about their use of AIapproaches in the management of patient safety. With 30responses, we found broad adoption of AI in current clinicaloperations with more than half of healthcare delivery organiza-tions reporting AI solutions currently deployed in operations, andanother 30% planning to do so in the next 2 years. The survey alsofound that almost half of these applications touched on areas ofpatient safety such as clinical deterioration scoring, sepsisprediction, surgery complication prediction, and readmissionprediction. We also found that almost half of these AI applicationswere home grown by the organizations themselves rather thancreated by vendors, supporting the findings in the studymentioned above5.The rapidly increasing use of artificial intelligence (AI) inoperational clinical settings presents an opportunity for evaluationas there is limited research or funding for such about its efficacy orsafety. There has been steady progress in methods and toolsneeded to manipulate and transform electronic clinical data, andincreasingly mature data resources have supported the rapiddevelopment of sophisticated AI in some health care domainsincluding patient safety6–8. The broad adoption of personaldevices such as wrist watches that measure heart rhythms orportable glucose monitors or other patient self-monitoringdevices offer far broader data types than traditional EHR data,and integration of this multi-modal data into EHRs seems likely toyield earlier and more actionable AI predictions7. A recent studydemonstrated the impact of clinical deterioration software onpatient outcomes, this was homegrown in one large health systemand has not been disseminated beyond this system. Indeed, thereare a large number of vendors currently providing clinicaldeterioration software products, most of which has not beenrigorously evaluated4.There are few rigorous assessments of actual AI deployments inhealth care delivery systems, and while there is some limitedevidence for improved safety processes or outcomes when theseAI tools are deployed4–8, there is also evidence that these systemscan increase risk if the algorithms are tuned to give overlyconfident results9. For example, within AI risk prediction models,the sizeable literature on model development and validation is in1University of Utah School of Medicine, Division of Clinical Epidemiology, Salt Lake City, UT, USA. 2UC San Diego Health, Departments of Medicine and Pediatrics, San Diego, CA,USA. 3McGovern Medical School at the University of Texas Health Science Center Houston, and the UT Houston-Memorial Hermann Center for Healthcare Quality andSafety, Houston, USA. ✉email: dcclassen@hotmail.comwww.nature.com/npjdigitalmedPublished in partnership with Seoul National University Bundang Hospital1234567890():,;stark contrast to the scant data describing successful clinicaldeployment and impact of those models in health care settings.One study revealed significant problems with one vendor’s EHRsepsis prediction algorithm, which has been very widely deployedamong many health systems without any rigorous evaluation10.The prediction of sepsis for inpatients, a common conditionwith a high mortality rate, is an area of intense AI focus in healthcare10–12. Many studies have shown early detection and treatmentof patients with sepsis can markedly reduce mortality. Indeed, arecent review found over 1800 published studies of AI programsdeveloped to predict sepsis in patients hospitalized or in theemergency room. However, none of these models have beenwidely adopted11. The resulting vacuum has been filled by a largecommercial EHR vendor that developed its own proprietary modelwhich it deployed to hundreds of US hospitals without anypublished critical evaluation10. One of the health systems that usesthis commercial EHR sepsis prediction program performed anevaluation of this program in its own health system. The resultswere unexpected: the EHR vendor predictive program only pickedup 7% of 2552 patients with sepsis who were not treated withantibiotics in a timely fashion and failed to identify 1709 patientswith sepsis that the hospital did identify10. Obviously, this AI sepsisprediction algorithm was not subjected to rigorous externalevaluation but nevertheless was broadly adopted because the EHRvendor implemented it in its EHR package and thus made itconveniently available for its large install base of hospitals10. Nopublished evaluation on the impact of this proprietary EHR AIprogram on patients beyond this hospital has emerged and theimpacts both positive and negative that it may have caused in itsbroad hospital use is unknown.Another area of opportunity for AI in patient safety isautomated interpretation of radiology imaging, which is one ofthe largest categories of healthcare AI publications over the last 5years7. One example occurred at a large academic health systemthat had substantial AI resources and a widely used commercialEHR system. Within 2 weeks of the coronavirus pandemicdeclaration, this health system had already developed, tested,and implemented an AI algorithm for identification of COVIDpneumonia on radiology imaging, and found this AI algorithm wasboth well received and frequently used with a definitive impact onclinical decision making13. However, there are other reports thatsuggest that automated AI imaging diagnostic systems may notbe as successful14.These two use cases illustrate important current challenges withthe current broad adoption of AI programs across health systems.The sepsis algorithm developed by the EHR vendor came from alarge database this vendor has aggregated from client EHR dataand used data from 405,000 hospital admissions to develop thealgorithm. The fact that the algorithm performed so poorlyemphasizes that bigger is not always better and that AI bias,framing, and other issues can exist with AI applications developedeven from very large data sets9,12. This experience validates manyof the current concerns about AI algorithms and outlines the greatimportance of validating these algorithms in local data sets beforethey are adopted. The diagnostic Xray example reveals just howquickly these AI algorithms can be developed and deployedwithin a local institution and perhaps helps to explain why therehas been such broad adoption of AI diagnostic imagingalgorithms13. Clearly the challenges with successful implementa-tion of these AI algorithms reflects in many ways the hurdles andexperiences previously identified in deploying EHR based clinicaldecision support systems15. While some impediments weretechnical, more relate to the complexity of tailoring applicationsfor integration with existing capabilities in electronic healthrecords (EHRs), such as poor understanding of users’ needs andexpectations for information, poorly defined clinical processes,workflows, and objectives, and even concerns about legal liability.These impediments may be balanced by the potential for gain, asseveral reviews of closed malpractice claims found that more thanone-half of malpractice claims could have been potentiallyprevented by well-designed clinical decision support withadvanced analytics and AI16.These challenges with the use of AI in healthcare were exploredby a conference in October 2021 on patient safety and AIsponsored by the Robert Wood Johnson Foundation (RWJF) where110 patient safety and Health IT leaders invited from hospital andhealth systems, health insurers, vendors, researchers, regulatorsand other stakeholders shared their extensive experiences andapproaches to AI and began to build an agenda for patient safetyand AI17. The attendees described broad adoption of AI in theirorganizations—despite few published scientific studies of itseffectiveness or inherent safety by their organizations. Thismultistakeholder conference using Delphi like techniques devel-oped a top list of areas that need focused evaluation in AI andPatient Safety, (Table 1) as well as use cases for specific clinicalproblems in patient safety that are already being used and needfurther evaluation (Table 2).AI has significant potential to improve patient safety. However,given the lack of rigorous evaluation of AI in actual currentpractice, coupled with its surprisingly broad use, we believe thetime has come to create a national agenda for a critical evaluationof AI and patient safety. This critical evaluation needs to determineamong other things whether the broad current adoption of AI inhealth systems has actually improved patient safety. This agendamust of course include significant new federal research fundingfor this rigorous AI evaluation, especially in the area of patientsafety, if we are to learn from the already wide deployment of AI inhealthcare. The Robert Wood Johnson Conference cited above isjust the beginning off that process, which should involve majorstakeholders such as hospitals and health systems, public andTable 1. Areas of focus for AI and patient safety per RWJF conference.1. Develop AI/advanced analytics implementation models, implementation approaches, and methods for integration into clinical workflows2. Create a patient safety framework to guide measurement of AI impact: How to use AI to improve each dimension of safety from retrospectiveanalysis to real-time monitoring to future use of prediction3. Build an AI patient safety financial business case4. Reduce cognitive and total work burden with AI which should be interpretable and usable for frontline users.5. AI patient and consumer focused issues: study how patients and consumers will view and use these tools and how their use will impact patient-doctor and patient-healthcare team relationships6. Create ways to engage all the relevant stakeholders in AI use and design7. Develop effective governance/oversight and accountability for AI in clinical care8. Develop methods to learn and loop back to adjust AI algorithms to ensure equity—refine or change for different or changing populations9. Create AI to enhance adverse event/near miss monitoring and real time safety surveillance10. Create Use Cases for the application of AI to specific problems in patient safetyD.C. Classen et al.2npj Digital Medicine (2023)     2 Published in partnership with Seoul National University Bundang Hospital1234567890():,;private payors, outcomes researchers, vendors, regulators, andpatient advocates and be convened by a trusted entity such as theNational Academy of Medicine.Received: 24 June 2022; Accepted: 29 November 2022;REFERENCES1. Classen, D. C. et al. National Trends in the Safety Performance of Electronic HealthRecord Systems From 2009 to 2018. JAMA Netw. Open 3, e205547 (2020).2. Sammer, C. et al. Developing and evaluating an automated all-cause harm triggersystem. Jt Comm. J. Qual. Patient Saf. 43, 155–165 (2017).3. Classen, D. C., Griffin, F. A. & Berwick, D. M. Measuring patient safety in real time:an essential method for effectively improving the safety of care. Ann. Intern. Med.167, 882–883 (2017).4. Escobar, G. J. et al. Automated identification of adults at risk for in-hospital clinicaldeterioration. N. Engl. J. Med. 383, 1951–1960 (2020).5. Rojas, J. C., Rohweder, G., Guptill, J., Arora, V. M. & Umscheid, C. A. PredictiveAnalytics Programs at Large Healthcare Systems in the USA: a National Survey. J.Gen. Intern. Med. https://doi.org/10.1007/s11606-022-07517-1 (2022).6. Carlile, M. et al. Deployment of artificial intelligence for radiographic diagnosis ofCOVID-19 pneumonia in the emergency department. J. Am. Coll. Emerg. Physi-cians Open 1, 1459–1464 (2020).7. Kim, C. K. et al. An automated COVID-19 triage pipeline using artificial intelligencebased on chest radiographs and clinical data. NPJ Digit. Med. 5, 5 (2022).8. Classen, D., Li, M., Miller, S. & Ladner, D. An electronic health record-based real-time analytics program for patient safety surveillance and improvement. HealthAff. 37, 1805–1812 (2018).9. Shashikumar, S. P., Wardi, G., Malhotra, A. & Nemati, S. Artificial intelligence sepsisprediction algorithm learns to say, “I don’t know”. NPJ Digit. Med. 4, 1–9 (2021).10. Wong, A. et al. External validation of a widely implemented proprietary sepsisprediction model in hospitalized patients. JAMA Intern. Med. 181, 1065–1070(2021).11. Yan, M. Y., Gustad, L. T. & Nytrø, Ø. Sepsis prediction, early detection, andidentification using clinical text for machine learning: a systematic review. J. Am.Med. Inf. Assoc. 29, 559–575 (2022).12. Lauritsen, S. M. et al. The Framing of machine learning risk prediction modelsillustrated by evaluation of sepsis in general wards. npj Digit. Med. 4, 158 (2021).13. Harmon, S. A. et al. Artificial intelligence for the detection of COVID-19 pneu-monia on chest CT using multinational datasets. Nat. Commun. 11, 4080 (2020).14. Finlayson Fg et al. The clinician and dataset shift in artificial intelligence. N. Engl. J.Med. 385, 283–286 (2021).15. Shortliffe, E. H. & Sepúlveda, M. J. Clinical decision support in the era of artificialintelligence. JAMA 320, 2199–2200 (2018).16. Zuccotti, G. et al. Reducing risk with clinical decision support: a study of closedmalpractice claims. Appl. Clin. Inform. 5, 746–756 (2014).17. Advanced Analytics Conference Convenes Patient Safety Leaders in Utah. (JewishHealth Care Foundation Pittsburg PA, 2021).ACKNOWLEDGEMENTSThis manuscript and the conference mentioned herein was funded by the RobertWood Johnson Foundation who had no involvement in the preparation of thismanuscript.AUTHOR CONTRIBUTIONSConcept and design: D.C.C., C.L., E.J.T. Drafting of the manuscript: D.C.C., C.L., E.J.T.Critical Revision of manuscript: D.C.C., C.L., E.J.T.COMPETING INTERESTSThe authors declare no competing interests.ADDITIONAL INFORMATIONCorrespondence and requests for materials should be addressed to David C. Classen.Reprints and permission information is available at http://www.nature.com/reprintsPublisher’s note Springer Nature remains neutral with regard to jurisdictional claimsin published maps and institutional affiliations.Open Access This article is licensed under a Creative CommonsAttribution 4.0 International License, which permits use, sharing,adaptation, distribution and reproduction in anymedium or format, as long as you giveappropriate credit to the original author(s) and the source, provide a link to the CreativeCommons license, and indicate if changes were made. The images or other third partymaterial in this article are included in the article’s Creative Commons license, unlessindicated otherwise in a credit line to the material. If material is not included in thearticle’s Creative Commons license and your intended use is not permitted by statutoryregulation or exceeds the permitted use, you will need to obtain permission directlyfrom the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.© The Author(s) 2023Table 2. Top use cases for the application of AI to specific clinicalproblems in patient safety.1. Actionable real time patient safety electronic clinical qualitymeasures2. Surgical complication prediction3. Pressure ulcer prediction4. Hypoglycemia prediction5. Sepsis prediction6. Suicide prediction7. Diabetic eye AI screening8. Breast imaging cancer screening9. Chest x-ray imaging AI diagnosis10. Skin melanoma AI diagnosis11. Chest x-ray imaging AI cancer screening12. Patient self-managed electronic safety dashboardsD.C. Classen et al.3Published in partnership with Seoul National University Bundang Hospital npj Digital Medicine (2023)     2 

Bending the Patient Safety Curve: How Much Can AI Help?

https://digitalcommons.library.tmc.edu/cgi/viewcontent.cgi?article=1762&context=uthmed_docs

Bending the Patient Safety Curve: How Much Can AI Help?

Abstract

Similar works

Full text

Available Versions

DigitalCommons@The Texas Medical Center