Predicting responders to prone positioning in mechanically ventilated patients with COVID-19 using machine learning

Background: For mechanically ventilated critically ill COVID-19 patients, prone positioning has quickly become an important treatment strategy, however, prone positioning is labor intensive and comes with potential adverse effects. Therefore, identifying which critically ill intubated COVID-19 patients will benefit may help allocate labor resources. Methods: From the multi-center Dutch Data Warehouse of COVID-19 ICU patients from 25 hospitals, we selected all 3619 episodes of prone positioning in 1142 invasively mechanically ventilated patients. We excluded episodes longer than 24 h. Berlin ARDS criteria were not formally documented. We used supervised machine learning algorithms Logistic Regression, Random Forest, Naive Bayes, K-Nearest Neighbors, Support Vector Machine and Extreme Gradient Boosting on readily available and clinically relevant features to predict success of prone positioning after 4 h (window of 1 to 7 h) based on various possible outcomes. These outcomes were defined as improvements of at least 10% in PaO2/FiO2 ratio, ventilatory ratio, respiratory system compliance, or mechanical power. Separate models were created for each of these outcomes. Re-supination within 4 h after pronation was labeled as failure. We also developed models using a 20 mmHg improvement cut-off for PaO2/FiO2 ratio and using a combined outcome parameter. For all models, we evaluated feature importance expressed as contribution to predictive performance based on their relative ranking. Results: The median duration of prone episodes was 17 h (11-20, median and IQR, N = 2632). Despite extensive modeling using a plethora of machine learning techniques and a large number of potentially clinically relevant features, discrimination between responders and non-responders remained poor with an area under the receiver operator characteristic curve of 0.62 for PaO2/FiO2 ratio using Logistic Regression, Random Forest and XGBoost. Feature importance was inconsistent between models for different outcomes. Notably, not even being a previous responder to prone positioning, or PEEP-levels before prone positioning, provided any meaningful contribution to predicting a successful next proning episode. Conclusions: In mechanically ventilated COVID-19 patients, predicting the success of prone positioning using clinically relevant and readily available parameters from electronic health records is currently not feasible. Given the current evidence base, a liberal approach to proning in all patients with severe COVID-19 ARDS is therefore justified and in particular regardless of previous results of proning. Keywords: Acute respiratory distress syndrome; COVID-19; Mechanical ventilation

Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy

Abstract: Purpose: Early clinical recognition of sepsis can be challenging. With the advancement of machine learning, promising real-time models to predict sepsis have emerged. We assessed their performance by carrying out a systematic review and meta-analysis. Methods: A systematic search was performed in PubMed, Embase.com and Scopus. Studies targeting sepsis, severe sepsis or septic shock in any hospital setting were eligible for inclusion. The index test was any supervised machine learning model for real-time prediction of these conditions. Quality of evidence was assessed using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) methodology, with a tailored Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) checklist to evaluate risk of bias. Models with a reported area under the curve of the receiver operating characteristic (AUROC) metric were meta-analyzed to identify strongest contributors to model performance. Results: After screening, a total of 28 papers were eligible for synthesis, from which 130 models were extracted. The majority of papers were developed in the intensive care unit (ICU, n = 15; 54%), followed by hospital wards (n = 7; 25%), the emergency department (ED, n = 4; 14%) and all of these settings (n = 2; 7%). For the prediction of sepsis, diagnostic test accuracy assessed by the AUROC ranged from 0.68–0.99 in the ICU, to 0.96–0.98 in-hospital and 0.87 to 0.97 in the ED. Varying sepsis definitions limit pooling of the performance across studies. Only three papers clinically implemented models with mixed results. In the multivariate analysis, temperature, lab values, and model type contributed most to model performance. Conclusion: This systematic review and meta-analysis show that on retrospective data, individual machine learning models can accurately predict sepsis onset ahead of time. Although they present alternatives to traditional scoring systems, between-study heterogeneity limits the assessment of pooled results. Systematic reporting and clinical implementation studies are needed to bridge the gap between bytes and bedside

Transatlantic transferability of a new reinforcement learning model for optimizing haemodynamic treatment for critically ill patients with sepsis

Introduction: In recent years, reinforcement learning (RL) has gained traction in the healthcare domain. In particular, RL methods have been explored for haemodynamic optimization of septic patients in the Intensive Care Unit. Most hospitals however, lack the data and expertise for model development, necessitating transfer of models developed using external datasets. This approach assumes model generalizability across different patient populations, the validity of which has not previously been tested. In addition, there is limited knowledge on safety and reliability. These challenges need to be addressed to further facilitate implementation of RL models in clinical practice. Method: We developed and validated a new reinforcement learning model for hemodynamic optimization in sepsis on the MIMIC intensive care database from the USA using a dueling double deep Q network. We then transferred this model to the European AmsterdamUMCdb intensive care database. T-Distributed Stochastic Neighbor Embedding and Sequential Organ Failure Assessment scores were used to explore the differences between the patient populations. We apply off-policy policy evaluation methods to quantify model performance. In addition, we introduce and apply a novel deep policy inspection to analyse how the optimal policy relates to the different phases of sepsis and sepsis treatment to provide interpretable insight in order to assess model safety and reliability. Results: The off-policy evaluation revealed that the optimal policy outperformed the physician policy on both datasets despite marked differences between the two patient populations and physician's policies. Our novel deep policy inspection method showed insightful results and unveiled that the model could initiate therapy adequately and adjust therapy intensity to illness severity and disease progression which indicated safe and reliable model behaviour. Compared to current physician behavior, the developed policy prefers a more liberal use of vasopressors with a more restrained use of fluid therapy in line with previous work. Conclusion: We created a reinforcement learning model for optimal bedside hemodynamic management and demonstrated model transferability between populations from the USA and Europe for the first time. We proposed new methods for deep policy inspection integrating expert domain knowledge. This is expected to facilitate progression to bedside clinical decision support for the treatment of critically ill patients

Right Dose, Right Now: Development of AutoKinetics for Real Time Model Informed Precision Antibiotic Dosing Decision Support at the Bedside of Critically Ill Patients

Introduction: Antibiotic dosing in critically ill patients is challenging because their pharmacokinetics (PK) are altered and may change rapidly with disease progression. Standard dosing frequently leads to inadequate PK exposure. Therapeutic drug monitoring (TDM) offers a potential solution but requires sampling and PK knowledge, which delays decision support. It is our philosophy that antibiotic dosing support should be directly available at the bedside through deep integration into the electronic health record (EHR) system. Therefore we developed AutoKinetics, a clinical decision support system (CDSS) for real time, model informed precision antibiotic dosing. Objective: To provide a detailed description of the design, development, validation, testing, and implementation of AutoKinetics. Methods: We created a development framework and used workflow analysis to facilitate integration into popular EHR systems. We used a development cycle to iteratively adjust and expand AutoKinetics functionalities. Furthermore, we performed a literature review to select and integrate pharmacokinetic models for five frequently prescribed antibiotics for sepsis. Finally, we tackled regulatory challenges, in particular those related to the Medical Device Regulation under the European regulatory framework. Results: We developed a SQL-based relational database as the backend of AutoKinetics. We developed a data loader to retrieve data in real time. We designed a clinical dosing algorithm to find a dose regimen to maintain antibiotic pharmacokinetic exposure within clinically relevant safety constraints. If needed, a loading dose is calculated to minimize the time until steady state is achieved. Finally, adaptive dosing using Bayesian estimation is applied if plasma levels are available. We implemented support for five extensively used antibiotics following model development, calibration, and validation. We integrated AutoKinetics into two popular EHRs (Metavision, Epic) and developed a user interface that provides textual and visual feedback to the physician. Conclusion: We successfully developed a CDSS for real time model informed precision antibiotic dosing at the bedside of the critically ill. This holds great promise for improving sepsis outcome. Therefore, we recently started the Right Dose Right Now multi-center randomized control trial to validate this concept in 420 patients with severe sepsis and septic shock

Clinically relevant pharmacokinetic knowledge on antibiotic dosing among intensive care professionals is insufficient: a cross-sectional study

BACKGROUND: Antibiotic exposure in intensive care patients with sepsis is frequently inadequate and is associated with poorer outcomes. Antibiotic dosing is challenging in the intensive care, as critically ill patients have altered and fluctuating antibiotic pharmacokinetics that make current one-size-fits-all regimens unsatisfactory. Real-time bedside dosing software is not available yet, and therapeutic drug monitoring is typically used for few antibiotic classes and only allows for delayed dosing adaptation. Thus, adequate and timely antibiotic dosing continues to rely largely on the level of pharmacokinetic expertise in the ICU. Therefore, we set out to assess the level of knowledge on antibiotic pharmacokinetics among these intensive care professionals. METHODS: In May 2018, we carried out a cross-sectional study by sending out an online survey on antibiotic dosing to more than 20,000 intensive care professionals. Questions were designed to cover relevant topics in pharmacokinetics related to intensive care antibiotic dosing. The preliminary pass mark was set by members of the examination committee for the European Diploma of Intensive Care using a modified Angoff approach. The final pass mark was corrected for clinical relevance as assessed for each question by international experts on pharmacokinetics. RESULTS: A total of 1448 respondents completed the survey. Most of the respondents were intensivists (927 respondents, 64%) from 97 countries. Nearly all questions were considered clinically relevant by pharmacokinetic experts. The pass mark corrected for clinical relevance was 52.8 out of 93.7 points. Pass rates were 42.5% for intensivists, 36.1% for fellows, 24.8% for residents, and 5.8% for nurses. Scores without correction for clinical relevance were worse, indicating that respondents perform better on more relevant topics. Correct answers and concise clinical background are provided. CONCLUSIONS: Clinically relevant pharmacokinetic knowledge on antibiotic dosing among intensive care professionals is insufficient. This should be addressed given the importance of adequate antibiotic exposure in critically ill patients with sepsis. Solutions include improved education, intensified pharmacy support, therapeutic drug monitoring, or the use of real-time bedside dosing software. Questions may provide useful for teaching purposes

Machine learning for the prediction of sepsis:a systematic review and meta-analysis of diagnostic test accuracy

Purpose: Early clinical recognition of sepsis can be challenging. With the advancement of machine learning, promising real-time models to predict sepsis have emerged. We assessed their performance by carrying out a systematic review and meta-analysis. Methods: A systematic search was performed in PubMed, Embase.com and Scopus. Studies targeting sepsis, severe sepsis or septic shock in any hospital setting were eligible for inclusion. The index test was any supervised machine learning model for real-time prediction of these conditions. Quality of evidence was assessed using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) methodology, with a tailored Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) checklist to evaluate risk of bias. Models with a reported area under the curve of the receiver operating characteristic (AUROC) metric were meta-analyzed to identify strongest contributors to model performance. Results: After screening, a total of 28 papers were eligible for synthesis, from which 130 models were extracted. The majority of papers were developed in the intensive care unit (ICU, n = 15; 54%), followed by hospital wards (n = 7; 25%), the emergency department (ED, n = 4; 14%) and all of these settings (n = 2; 7%). For the prediction of sepsis, diagnostic test accuracy assessed by the AUROC ranged from 0.68–0.99 in the ICU, to 0.96–0.98 in-hospital and 0.87 to 0.97 in the ED. Varying sepsis definitions limit pooling of the performance across studies. Only three papers clinically implemented models with mixed results. In the multivariate analysis, temperature, lab values, and model type contributed most to model performance. Conclusion: This systematic review and meta-analysis show that on retrospective data, individual machine learning models can accurately predict sepsis onset ahead of time. Although they present alternatives to traditional scoring systems, between-study heterogeneity limits the assessment of pooled results. Systematic reporting and clinical implementation studies are needed to bridge the gap between bytes and bedside

External evaluation of population pharmacokinetic models of vancomycin in large cohorts of intensive care unit patients

Dosing of vancomycin is often guided by therapeutic drug monitoring and population pharmacokinetic models in the intensive care unit (ICU). The validity of these models is crucial, as ICU patients have marked pharmacokinetic variability. Therefore, we set out to evaluate the predictive performance of published population pharmacokinetic models of vancomycin in ICU patients. The PubMed database was used to search for population pharmacokinetic models of vancomycin in adult ICU patients. The identified models were evaluated in two independent data sets which were collected from two large hospitals in the Netherlands (Amsterdam UMC, Location VUmc, and OLVG Oost). We also tested a one-compartment model with fixed values for clearance and volume of distribution, in which a clinical standard dosage regimen (SDR) was mimicked to assess its predictive performance. Prediction error was calculated to assess the predictive performance of the models. Six models plus the SDR model were evaluated. The model of Roberts et al. (J. A. Roberts, F. S. Taccone, A. A. Udy, J.-L. Vincent, F. Jacobs, and J. Lipman, Antimicrob Agents Chemother 55:2704–2709, 2011, https://doi.org/10.1128/AAC.01708-10) performed satisfactorily, with mean and median values of prediction error of 5.1% and 7.5%, respectively, for Amsterdam UMC, Location VUmc, patients, and 12.6% and 17.2% respectively, for OLVG Oost patients. The other models, including the SDR model, yielded high mean values (49.7% to 87.7%) and median values (56.1% to 66.1%) for both populations. In conclusion, only the model of Roberts et al. was able to validly predict the concentrations of vancomycin for our data, whereas other models and standard dosing were largely inadequate. Extensive evaluation should precede the adoption of any model in clinical practice for ICU patients

Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy

Abstract: Purpose: Early clinical recognition of sepsis can be challenging. With the advancement of machine learning, promising real-time models to predict sepsis have emerged. We assessed their performance by carrying out a systematic review and meta-analysis. Methods: A systematic search was performed in PubMed, Embase.com and Scopus. Studies targeting sepsis, severe sepsis or septic shock in any hospital setting were eligible for inclusion. The index test was any supervised machine learning model for real-time prediction of these conditions. Quality of evidence was assessed using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) methodology, with a tailored Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) checklist to evaluate risk of bias. Models with a reported area under the curve of the receiver operating characteristic (AUROC) metric were meta-analyzed to identify strongest contributors to model performance. Results: After screening, a total of 28 papers were eligible for synthesis, from which 130 models were extracted. The majority of papers were developed in the intensive care unit (ICU, n = 15; 54%), followed by hospital wards (n = 7; 25%), the emergency department (ED, n = 4; 14%) and all of these settings (n = 2; 7%). For the prediction of sepsis, diagnostic test accuracy assessed by the AUROC ranged from 0.68–0.99 in the ICU, to 0.96–0.98 in-hospital and 0.87 to 0.97 in the ED. Varying sepsis definitions limit pooling of the performance across studies. Only three papers clinically implemented models with mixed results. In the multivariate analysis, temperature, lab values, and model type contributed most to model performance. Conclusion: This systematic review and meta-analysis show that on retrospective data, individual machine learning models can accurately predict sepsis onset ahead of time. Although they present alternatives to traditional scoring systems, between-study heterogeneity limits the assessment of pooled results. Systematic reporting and clinical implementation studies are needed to bridge the gap between bytes and bedside