A Reinforcement Learning Approach to Weaning of Mechanical Ventilation in Intensive Care Units

The management of invasive mechanical ventilation, and the regulation of sedation and analgesia during ventilation, constitutes a major part of the care of patients admitted to intensive care units. Both prolonged dependence on mechanical ventilation and premature extubation are associated with increased risk of complications and higher hospital costs, but clinical opinion on the best protocol for weaning patients off of a ventilator varies. This work aims to develop a decision support tool that uses available patient information to predict time-to-extubation readiness and to recommend a personalized regime of sedation dosage and ventilator support. To this end, we use off-policy reinforcement learning algorithms to determine the best action at a given patient state from sub-optimal historical ICU data. We compare treatment policies from fitted Q-iteration with extremely randomized trees and with feedforward neural networks, and demonstrate that the policies learnt show promise in recommending weaning protocols with improved outcomes, in terms of minimizing rates of reintubation and regulating physiological stability

Artificial Intelligence for the prediction of weaning readiness outcome in a multi-centrical clinical cohort of mechanically ventilated patients

Quando un paziente soffre di insufficienza respiratoria acuta, viene praticata la ventilazione meccanica (VM) finché questa non riesce a respirare di nuovo in autonomia. Il medico di Terapia Intensiva verifica ogni giorno se la VM può essere interrotta. Questo screening consiste in una prima fase, il Readiness Test (RT), che è composta da vari parametri clinici. Se questo test ha esito positivo, si sottopone il paziente a 30 minuti di respirazione spontanea (SBT). Se anche l'SBT viene superato con successo, la VM viene interrotta. Al contrario, se l’RT o l’SBT falliscono, il paziente rimane in VM e verrà rivalutato il giorno successivo. Quindi ogni giorno possono verificarsi tre scenari mutuamente esclusivi: l’SBT non verrà tentato, l’SBT fallirà o l’SBT avrà successo (portando quindi all’estubazione del paziente). Il modello di intelligenza artificiale sviluppato, è progettato per dedurre fin dalle prime ore del mattino quale dei tre scenari si verificherà probabilmente nel corso della giornata, partendo dai dati clinici del paziente, dalle informazioni raccolte nel diario clinico dei giorni precedenti e dall'intera storia di registrazione minuto-per-minuto dei vari parametri del ventilatore meccanico, provenienti da uno studio osservazionale retrospettivo multicentrico, condotto in Italia nel corso di 27 mesi. Questi dati vengono elaborati con un approccio di Deep Learning, attraverso una topologia di rete neurale multi-sorgente, alimentata da architetture ricorrenti multiple. Gli iper-parametri sono ottimizzati per selezionare il modello desiderato attraverso la convalida incrociata, riservando 36 pazienti su 182 per testare le prestazioni finali del modello su una serie di metriche, tra cui uno score personalizzato progettato per evidenziare l'impatto clinico. Il modello di intelligenza artificiale finale mostra un'accuratezza del 79% [74, 83%], uno score personalizzato di 0,01 [-0,04, 0,05], un MCC di 0,28 [0,17, 0,39], ottenendo un punteggio migliore rispetto agli altri modelli di confronto, tra cui XG Boost, addestrato solo sui dati clinici giornalieri del giorno precedente, che ha avuto un'accuratezza del 61% [56%, 66%], un MCC di 0,14 [0,06, 0,2] e uno score personalizzato di -0,05 [-0,08, -0,01]. Complessivamente, il modello di intelligenza artificiale è in grado di approssimare bene l'attuale gestione clinica giorno per giorno, fornendo suggerimenti al mattino presto. Inoltre, c'è ancora spazio per migliorare l'utilità clinica del modello considerando ulteriori dati di addestramento personalizzati.When someone suffers from acute respiratory failure, mechanical ventilation (MV) is performed until they can breathe on their own again. The doctor checks every day whether the MV can be stopped. This screening consists of a first phase, the Readiness Testing (RT), which includes various clinical parameters. If this test is successful, 30 minutes of spontaneous breathing (SBT) is attempted. If also the SBT is passed successfully, the VM is stopped. On the contrary, if RT or SBT fails, the patient will be re-evaluated the next day. So, every day three mutually exclusive scenarios may happen: SBT will not be attempted, SBT will fail, or SBT will succeed. Our artificial intelligence model is designed to infer early in the morning which of the three scenarios will probably occur during the day, starting from the patient's clinical data, from the information collected in the previous day’s clinical diary, and from whole minute-by-minute recording history of the various parameters of the mechanical ventilator, coming from a retrospective observational multi-centrical study, conducted in Italy over a course of 27 months. Those data are processed with a deep learning approach, through a multi-source neural network topology, powered by multiple recurrent architectures. Hyper-parameters are optimized to select the purposed model through cross-validation, setting aside 36 out of 182 patients for testing final model performance over a variety of metrics, including a custom score designed to highlight clinical impact. The final AI model had an accuracy of 79% [74, 83%], a custom score of 0.01 [-0.04, 0.05], a MCC of 0.28 [0.17, 0.39], scoring better than the other comparison models, including XG Boost that was trained on daily and baseline clinical data of the previous day only, which had an accuracy of 61% [56%, 66%], a MCC of 0.14 [0.06, 0.2] and a custom score of -0.05 [-0.08, -0.01]. Overall, AI model could approximate well what is the current clinical management throughout day-by-day providing suggestions early in the morning. Moreover, there are still space to improve the model clinical utility considering additional tailored training data

Artificial intelligence for clinical decision support for monitoring patients in cardiovascular ICUs: a systematic review

Background: Artificial intelligence (AI) and machine learning (ML) models continue to evolve the clinical decision support systems (CDSS). However, challenges arise when it comes to the integration of AI/ML into clinical scenarios. In this systematic review, we followed the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA), the population, intervention, comparator, outcome, and study design (PICOS), and the medical AI life cycle guidelines to investigate studies and tools which address AI/ML-based approaches towards clinical decision support (CDS) for monitoring cardiovascular patients in intensive care units (ICUs). We further discuss recent advances, pitfalls, and future perspectives towards effective integration of AI into routine practices as were identified and elaborated over an extensive selection process for state-of-the-art manuscripts. Methods: Studies with available English full text from PubMed and Google Scholar in the period from January 2018 to August 2022 were considered. The manuscripts were fetched through a combination of the search keywords including AI, ML, reinforcement learning (RL), deep learning, clinical decision support, and cardiovascular critical care and patients monitoring. The manuscripts were analyzed and filtered based on qualitative and quantitative criteria such as target population, proper study design, cross-validation, and risk of bias. Results: More than 100 queries over two medical search engines and subjective literature research were developed which identified 89 studies. After extensive assessments of the studies both technically and medically, 21 studies were selected for the final qualitative assessment. Discussion: Clinical time series and electronic health records (EHR) data were the most common input modalities, while methods such as gradient boosting, recurrent neural networks (RNNs) and RL were mostly used for the analysis. Seventy-five percent of the selected papers lacked validation against external datasets highlighting the generalizability issue. Also, interpretability of the AI decisions was identified as a central issue towards effective integration of AI in healthcare

An Optimal Policy for Patient Laboratory Tests in Intensive Care Units

Laboratory testing is an integral tool in the management of patient care in hospitals, particularly in intensive care units (ICUs). There exists an inherent trade-off in the selection and timing of lab tests between considerations of the expected utility in clinical decision-making of a given test at a specific time, and the associated cost or risk it poses to the patient. In this work, we introduce a framework that learns policies for ordering lab tests which optimizes for this trade-off. Our approach uses batch off-policy reinforcement learning with a composite reward function based on clinical imperatives, applied to data that include examples of clinicians ordering labs for patients. To this end, we develop and extend principles of Pareto optimality to improve the selection of actions based on multiple reward function components while respecting typical procedural considerations and prioritization of clinical goals in the ICU. Our experiments show that we can estimate a policy that reduces the frequency of lab tests and optimizes timing to minimize information redundancy. We also find that the estimated policies typically suggest ordering lab tests well ahead of critical onsets--such as mechanical ventilation or dialysis--that depend on the lab results. We evaluate our approach by quantifying how these policies may initiate earlier onset of treatment.Comment: The first two authors contributed equally to this work. Preprint of an article submitted for consideration in Pacific Symposium on Biocomputing copyright 2018 [copyright World Scientific Publishing Company] [https://psb.stanford.edu/