Yet Another ICU Benchmark: A Flexible Multi-Center Framework for Clinical ML

Medical applications of machine learning (ML) have experienced a surge in popularity in recent years. The intensive care unit (ICU) is a natural habitat for ML given the abundance of available data from electronic health records. Models have been proposed to address numerous ICU prediction tasks like the early detection of complications. While authors frequently report state-of-the-art performance, it is challenging to verify claims of superiority. Datasets and code are not always published, and cohort definitions, preprocessing pipelines, and training setups are difficult to reproduce. This work introduces Yet Another ICU Benchmark (YAIB), a modular framework that allows researchers to define reproducible and comparable clinical ML experiments; we offer an end-to-end solution from cohort definition to model evaluation. The framework natively supports most open-access ICU datasets (MIMIC III/IV, eICU, HiRID, AUMCdb) and is easily adaptable to future ICU datasets. Combined with a transparent preprocessing pipeline and extensible training code for multiple ML and deep learning models, YAIB enables unified model development. Our benchmark comes with five predefined established prediction tasks (mortality, acute kidney injury, sepsis, kidney function, and length of stay) developed in collaboration with clinicians. Adding further tasks is straightforward by design. Using YAIB, we demonstrate that the choice of dataset, cohort definition, and preprocessing have a major impact on the prediction performance - often more so than model class - indicating an urgent need for YAIB as a holistic benchmarking tool. We provide our work to the clinical ML community to accelerate method development and enable real-world clinical implementations. Software Repository: https://github.com/rvandewater/YAIB.Comment: Main benchmark: https://github.com/rvandewater/YAIB, Cohort generation: https://github.com/rvandewater/YAIB-cohorts, Models: https://github.com/rvandewater/YAIB-model

Machine learning in intensive care medicine: ready for take-off?

In 1986 the world was shaken by the Challenger space shuttle disaster. In the years that followed, the American National Aeronautics and Space Administration (NASA) called for a strategy change in space technology development [1]. Allowing technology to be developed without a specific space program in mind was central to the new strategy [2]. In order to evaluate resulting projects with no direct contribution to a space mission, NASA introduced the general concept of technology readiness levels (TRLs) [3]. These nine levels, adopted by many EU institutions, assess the maturity level of technology and estimate its readiness to fly

Sharing ICU Patient Data Responsibly Under the Society of Critical Care Medicine/European Society of Intensive Care Medicine Joint Data Science Collaboration: The Amsterdam University Medical Centers Database (AmsterdamUMCdb) Example.

OBJECTIVES: Critical care medicine is a natural environment for machine learning approaches to improve outcomes for critically ill patients as admissions to ICUs generate vast amounts of data. However, technical, legal, ethical, and privacy concerns have so far limited the critical care medicine community from making these data readily available. The Society of Critical Care Medicine and the European Society of Intensive Care Medicine have identified ICU patient data sharing as one of the priorities under their Joint Data Science Collaboration. To encourage ICUs worldwide to share their patient data responsibly, we now describe the development and release of Amsterdam University Medical Centers Database (AmsterdamUMCdb), the first freely available critical care database in full compliance with privacy laws from both the United States and Europe, as an example of the feasibility of sharing complex critical care data. SETTING: University hospital ICU. SUBJECTS: Data from ICU patients admitted between 2003 and 2016. INTERVENTIONS: We used a risk-based deidentification strategy to maintain data utility while preserving privacy. In addition, we implemented contractual and governance processes, and a communication strategy. Patient organizations, supporting hospitals, and experts on ethics and privacy audited these processes and the database. MEASUREMENTS AND MAIN RESULTS: AmsterdamUMCdb contains approximately 1 billion clinical data points from 23,106 admissions of 20,109 patients. The privacy audit concluded that reidentification is not reasonably likely, and AmsterdamUMCdb can therefore be considered as anonymous information, both in the context of the U.S. Health Insurance Portability and Accountability Act and the European General Data Protection Regulation. The ethics audit concluded that responsible data sharing imposes minimal burden, whereas the potential benefit is tremendous. CONCLUSIONS: Technical, legal, ethical, and privacy challenges related to responsible data sharing can be addressed using a multidisciplinary approach. A risk-based deidentification strategy, that complies with both U.S. and European privacy regulations, should be the preferred approach to releasing ICU patient data. This supports the shared Society of Critical Care Medicine and European Society of Intensive Care Medicine vision to improve critical care outcomes through scientific inquiry of vast and combined ICU datasets

Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy

Abstract: Purpose: Early clinical recognition of sepsis can be challenging. With the advancement of machine learning, promising real-time models to predict sepsis have emerged. We assessed their performance by carrying out a systematic review and meta-analysis. Methods: A systematic search was performed in PubMed, Embase.com and Scopus. Studies targeting sepsis, severe sepsis or septic shock in any hospital setting were eligible for inclusion. The index test was any supervised machine learning model for real-time prediction of these conditions. Quality of evidence was assessed using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) methodology, with a tailored Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) checklist to evaluate risk of bias. Models with a reported area under the curve of the receiver operating characteristic (AUROC) metric were meta-analyzed to identify strongest contributors to model performance. Results: After screening, a total of 28 papers were eligible for synthesis, from which 130 models were extracted. The majority of papers were developed in the intensive care unit (ICU, n = 15; 54%), followed by hospital wards (n = 7; 25%), the emergency department (ED, n = 4; 14%) and all of these settings (n = 2; 7%). For the prediction of sepsis, diagnostic test accuracy assessed by the AUROC ranged from 0.68–0.99 in the ICU, to 0.96–0.98 in-hospital and 0.87 to 0.97 in the ED. Varying sepsis definitions limit pooling of the performance across studies. Only three papers clinically implemented models with mixed results. In the multivariate analysis, temperature, lab values, and model type contributed most to model performance. Conclusion: This systematic review and meta-analysis show that on retrospective data, individual machine learning models can accurately predict sepsis onset ahead of time. Although they present alternatives to traditional scoring systems, between-study heterogeneity limits the assessment of pooled results. Systematic reporting and clinical implementation studies are needed to bridge the gap between bytes and bedside

Early mobilisation in critically ill COVID-19 patients: a subanalysis of the ESICM-initiated UNITE-COVID observational study

Background Early mobilisation (EM) is an intervention that may improve the outcome of critically ill patients. There is limited data on EM in COVID-19 patients and its use during the first pandemic wave. Methods This is a pre-planned subanalysis of the ESICM UNITE-COVID, an international multicenter observational study involving critically ill COVID-19 patients in the ICU between February 15th and May 15th, 2020. We analysed variables associated with the initiation of EM (within 72 h of ICU admission) and explored the impact of EM on mortality, ICU and hospital length of stay, as well as discharge location. Statistical analyses were done using (generalised) linear mixed-effect models and ANOVAs. Results Mobilisation data from 4190 patients from 280 ICUs in 45 countries were analysed. 1114 (26.6%) of these patients received mobilisation within 72 h after ICU admission; 3076 (73.4%) did not. In our analysis of factors associated with EM, mechanical ventilation at admission (OR 0.29; 95% CI 0.25, 0.35; p = 0.001), higher age (OR 0.99; 95% CI 0.98, 1.00; p ≤ 0.001), pre-existing asthma (OR 0.84; 95% CI 0.73, 0.98; p = 0.028), and pre-existing kidney disease (OR 0.84; 95% CI 0.71, 0.99; p = 0.036) were negatively associated with the initiation of EM. EM was associated with a higher chance of being discharged home (OR 1.31; 95% CI 1.08, 1.58; p = 0.007) but was not associated with length of stay in ICU (adj. difference 0.91 days; 95% CI − 0.47, 1.37, p = 0.34) and hospital (adj. difference 1.4 days; 95% CI − 0.62, 2.35, p = 0.24) or mortality (OR 0.88; 95% CI 0.7, 1.09, p = 0.24) when adjusted for covariates. Conclusions Our findings demonstrate that a quarter of COVID-19 patients received EM. There was no association found between EM in COVID-19 patients' ICU and hospital length of stay or mortality. However, EM in COVID-19 patients was associated with increased odds of being discharged home rather than to a care facility. Trial registration ClinicalTrials.gov: NCT04836065 (retrospectively registered April 8th 2021)

Application of the Sepsis-3 criteria to describe sepsis epidemiology in the Amsterdam UMCdb intensive care dataset.

INTRODUCTION: Sepsis is a major cause of morbidity and mortality worldwide. In the updated, 2016 Sepsis-3 criteria, sepsis is defined as life-threatening organ dysfunction caused by a dysregulated host response to infection, where organ dysfunction can be represented by an increase in the Sequential Organ Failure Assessment (SOFA) score of 2 points or more. We sought to apply the Sepsis-3 criteria to characterise the septic cohort in the Amsterdam University Medical Centres database (Amsterdam UMCdb). METHODS: We examined adult intensive care unit (ICU) admissions in the Amsterdam UMCdb, which contains de-identified data for patients admitted to a mixed surgical-medical ICU at a tertiary academic medical centre in the Netherlands. We operationalised the Sepsis-3 criteria, defining organ dysfunction as an increase in the SOFA score of 2 points or more, while infection was defined as a new course of antibiotics or an escalation in antibiotic therapy, with at least one antibiotic given intravenously. Patients with sepsis were determined to be in septic shock if they additionally required the use of vasopressors and had a lactate level >2 mmol/L. RESULTS: We identified 18,221 ICU admissions from 16,408 patients in our cohort. There were 6,312 unique sepsis episodes, of which 30.2% met the criteria for septic shock. A total of 4,911/6,312 sepsis (77.8%) episodes occurred on ICU admission. Forty-seven percent of emergency medical admissions and 36.7% of emergency surgical admissions were for sepsis. Overall, there was a 12.5% ICU mortality rate; patients with septic shock had a higher ICU mortality rate (38.4%) than those without shock (11.4%). CONCLUSIONS: We successfully operationalised the Sepsis-3 criteria to the Amsterdam UMCdb, allowing the characterization and comparison of sepsis epidemiology across different centres

Sepsis-3 criteria in AmsterdamUMCdb: open-source code implementation

Sepsis is a major healthcare problem with substantial mortality and a common reason for admission to the intensive care unit (ICU). For this reason, the management of sepsis is an important area of ICU research. A number of large-scale, freely-accessible ICU databases are available for observational research and the robust identification of septic patients in such data sets is crucial for research purposes, particularly for comparative studies between critical care sub-populations which may vary around the world. However, data structures are poorly standardised due to inevitable variances in clinical electronic health record system vendor and implementation as well as research database design choices. Robust and well-documented cohort selection (such as patients with sepsis) is crucial for reproducible research. In this work, we operationalise the Sepsis-3 definition on the AmsterdamUMCdb, a recently published large European ICU database, publishing open-access code for wider use by critical care researchers. This is the Author Accepted Manuscript, accepted after peer review by GigaByte

Application of the Sepsis-3 criteria to describe sepsis epidemiology in the Amsterdam UMCdb intensive care dataset.

Acknowledgements: A CC BY or equivalent licence is applied to the AAM arising from this submission. We would like to thank Professor Stephen J Eglen for supervision.INTRODUCTION: Sepsis is a major cause of morbidity and mortality worldwide. In the updated, 2016 Sepsis-3 criteria, sepsis is defined as life-threatening organ dysfunction caused by a dysregulated host response to infection, where organ dysfunction can be represented by an increase in the Sequential Organ Failure Assessment (SOFA) score of 2 points or more. We sought to apply the Sepsis-3 criteria to characterise the septic cohort in the Amsterdam University Medical Centres database (Amsterdam UMCdb). METHODS: We examined adult intensive care unit (ICU) admissions in the Amsterdam UMCdb, which contains de-identified data for patients admitted to a mixed surgical-medical ICU at a tertiary academic medical centre in the Netherlands. We operationalised the Sepsis-3 criteria, defining organ dysfunction as an increase in the SOFA score of 2 points or more, while infection was defined as a new course of antibiotics or an escalation in antibiotic therapy, with at least one antibiotic given intravenously. Patients with sepsis were determined to be in septic shock if they additionally required the use of vasopressors and had a lactate level >2 mmol/L. RESULTS: We identified 18,221 ICU admissions from 16,408 patients in our cohort. There were 6,312 unique sepsis episodes, of which 30.2% met the criteria for septic shock. A total of 4,911/6,312 sepsis (77.8%) episodes occurred on ICU admission. Forty-seven percent of emergency medical admissions and 36.7% of emergency surgical admissions were for sepsis. Overall, there was a 12.5% ICU mortality rate; patients with septic shock had a higher ICU mortality rate (38.4%) than those without shock (11.4%). CONCLUSIONS: We successfully operationalised the Sepsis-3 criteria to the Amsterdam UMCdb, allowing the characterization and comparison of sepsis epidemiology across different centres