Large expert-curated database for benchmarking document similarity detection in biomedical literature search

Document recommendation systems for locating relevant literature have mostly relied on methods developed a decade ago. This is largely due to the lack of a large offline gold-standard benchmark of relevant documents that cover a variety of research fields such that newly developed literature search techniques can be compared, improved and translated into practice. To overcome this bottleneck, we have established the RElevant LIterature SearcH consortium consisting of more than 1500 scientists from 84 countries, who have collectively annotated the relevance of over 180 000 PubMed-listed articles with regard to their respective seed (input) article/s. The majority of annotations were contributed by highly experienced, original authors of the seed articles. The collected data cover 76% of all unique PubMed Medical Subject Headings descriptors. No systematic biases were observed across different experience levels, research fields or time spent on annotations. More importantly, annotations of the same document pairs contributed by different scientists were highly concordant. We further show that the three representative baseline methods used to generate recommended articles for evaluation (Okapi Best Matching 25, Term Frequency-Inverse Document Frequency and PubMed Related Articles) had similar overall performances. Additionally, we found that these methods each tend to produce distinct collections of recommended articles, suggesting that a hybrid method may be required to completely capture all relevant articles. The established database server located at https://relishdb.ict.griffith.edu.au is freely available for the downloading of annotation data and the blind testing of new methods. We expect that this benchmark will be useful for stimulating the development of new powerful techniques for title and title/abstract-based search engines for relevant articles in biomedical research.Peer reviewe

Some Patients Are More Equal Than Others: Variation in Ventilator Settings for Coronavirus Disease 2019 Acute Respiratory Distress Syndrome

OBJECTIVES: As coronavirus disease 2019 is a novel disease, treatment strategies continue to be debated. This provides the intensive care community with a unique opportunity as the population of coronavirus disease 2019 patients requiring invasive mechanical ventilation is relatively homogeneous compared with other ICU populations. We hypothesize that the novelty of coronavirus disease 2019 and the uncertainty over its similarity with noncoronavirus disease 2019 acute respiratory distress syndrome resulted in substantial practice variation between hospitals during the first and second waves of coronavirus disease 2019 patients. DESIGN: Multicenter retrospective cohort study. SETTING: Twenty-five hospitals in the Netherlands from February 2020 to July 2020, and 14 hospitals from August 2020 to December 2020. PATIENTS: One thousand two hundred ninety-four critically ill intubated adult ICU patients with coronavirus disease 2019 were selected from the Dutch Data Warehouse. Patients intubated for less than 24 hours, transferred patients, and patients still admitted at the time of data extraction were excluded. MEASUREMENTS AND MAIN RESULTS: We aimed to estimate between-ICU practice variation in selected ventilation parameters (positive end-expiratory pressure, Fio2, set respiratory rate, tidal volume, minute volume, and percentage of time spent in a prone position) on days 1, 2, 3, and 7 of intubation, adjusted for patient characteristics as well as severity of illness based on Pao2/Fio2 ratio, pH, ventilatory ratio, and dynamic respiratory system compliance during controlled ventilation. Using multilevel linear mixed-effects modeling, we found significant (p ≤ 0.001) variation between ICUs in all ventilation parameters on days 1, 2, 3, and 7 of intubation for both waves. CONCLUSIONS: This is the first study to clearly demonstrate significant practice variation between ICUs related to mechanical ventilation parameters that are under direct control by intensivists. Their effect on clinical outcomes for both coronavirus disease 2019 and other critically ill mechanically ventilated patients could have widespread implications for the practice of intensive care medicine and should be investigated further by causal inference models and clinical trials

Risk factors for adverse outcomes during mechanical ventilation of 1152 COVID-19 patients: a multicenter machine learning study with highly granular data from the Dutch Data Warehouse

Background: The identification of risk factors for adverse outcomes and prolonged intensive care unit (ICU) stay in COVID-19 patients is essential for prognostication, determining treatment intensity, and resource allocation. Previous studies have determined risk factors on admission only, and included a limited number of predictors. Therefore, using data from the highly granular and multicenter Dutch Data Warehouse, we developed machine learning models to identify risk factors for ICU mortality, ventilator-free days and ICU-free days during the course of invasive mechanical ventilation (IMV) in COVID-19 patients. Methods: The DDW is a growing electronic health record database of critically ill COVID-19 patients in the Netherlands. All adult ICU patients on IMV were eligible for inclusion. Transfers, patients admitted for less than 24 h, and patients still admitted at time of data extraction were excluded. Predictors were selected based on the literature, and included medication dosage and fluid balance. Multiple algorithms were trained and validated on up to three sets of observations per patient on day 1, 7, and 14 using fivefold nested cross-validation, keeping observations from an individual patient in the same split. Results: A total of 1152 patients were included in the model. XGBoost models performed best for all outcomes and were used to calculate predictor importance. Using Shapley additive explanations (SHAP), age was the most important demographic risk factor for the outcomes upon start of IMV and throughout its course. The relative probability of death across age values is visualized in Partial Dependence Plots (PDPs), with an increase starting at 54 years. Besides age, acidaemia, low P/F-ratios and high driving pressures demonstrated a higher probability of death. The PDP for driving pressure showed a relative probability increase starting at 12 cmH2O. Conclusion: Age is the most important demographic risk factor of ICU mortality, ICU-free days and ventilator-free days throughout the course of invasive mechanical ventilation in critically ill COVID-19 patients. pH, P/F ratio, and driving pressure should be monitored closely over the course of mechanical ventilation as risk factors predictive of these outcomes

Some Patients Are More Equal Than Others:Variation in Ventilator Settings for Coronavirus Disease 2019 Acute Respiratory Distress Syndrome

OBJECTIVES: As coronavirus disease 2019 is a novel disease, treatment strategies continue to be debated. This provides the intensive care community with a unique opportunity as the population of coronavirus disease 2019 patients requiring invasive mechanical ventilation is relatively homogeneous compared with other ICU populations. We hypothesize that the novelty of coronavirus disease 2019 and the uncertainty over its similarity with noncoronavirus disease 2019 acute respiratory distress syndrome resulted in substantial practice variation between hospitals during the first and second waves of coronavirus disease 2019 patients. DESIGN: Multicenter retrospective cohort study. SETTING: Twenty-five hospitals in the Netherlands from February 2020 to July 2020, and 14 hospitals from August 2020 to December 2020. PATIENTS: One thousand two hundred ninety-four critically ill intubated adult ICU patients with coronavirus disease 2019 were selected from the Dutch Data Warehouse. Patients intubated for less than 24 hours, transferred patients, and patients still admitted at the time of data extraction were excluded. MEASUREMENTS AND MAIN RESULTS: We aimed to estimate between-ICU practice variation in selected ventilation parameters (positive end-expiratory pressure, Fio2, set respiratory rate, tidal volume, minute volume, and percentage of time spent in a prone position) on days 1, 2, 3, and 7 of intubation, adjusted for patient characteristics as well as severity of illness based on Pao2/Fio2 ratio, pH, ventilatory ratio, and dynamic respiratory system compliance during controlled ventilation. Using multilevel linear mixed-effects modeling, we found significant (p ≤ 0.001) variation between ICUs in all ventilation parameters on days 1, 2, 3, and 7 of intubation for both waves. CONCLUSIONS: This is the first study to clearly demonstrate significant practice variation between ICUs related to mechanical ventilation parameters that are under direct control by intensivists. Their effect on clinical outcomes for both coronavirus disease 2019 and other critically ill mechanically ventilated patients could have widespread implications for the practice of intensive care medicine and should be investigated further by causal inference models and clinical trials

Risk factors for adverse outcomes during mechanical ventilation of 1152 COVID-19 patients: a multicenter machine learning study with highly granular data from the Dutch Data Warehouse

Background: The identification of risk factors for adverse outcomes and prolonged intensive care unit (ICU) stay in COVID-19 patients is essential for prognostication, determining treatment intensity, and resource allocation. Previous studies have determined risk factors on admission only, and included a limited number of predictors. Therefore, using data from the highly granular and multicenter Dutch Data Warehouse, we developed machine learning models to identify risk factors for ICU mortality, ventilator-free days and ICU-free days during the course of invasive mechanical ventilation (IMV) in COVID-19 patients. Methods: The DDW is a growing electronic health record database of critically ill COVID-19 patients in the Netherlands. All adult ICU patients on IMV were eligible for inclusion. Transfers, patients admitted for less than 24 h, and patients still admitted at time of data extraction were excluded. Predictors were selected based on the literature, and included medication dosage and fluid balance. Multiple algorithms were trained and validated on up to three sets of observations per patient on day 1, 7, and 14 using fivefold nested cross-validation, keeping observations from an individual patient in the same split. Results: A total of 1152 patients were included in the model. XGBoost models performed best for all outcomes and were used to calculate predictor importance. Using Shapley additive explanations (SHAP), age was the most important demographic risk factor for the outcomes upon start of IMV and throughout its course. The relative probability of death across age values is visualized in Partial Dependence Plots (PDPs), with an increase starting at 54 years. Besides age, acidaemia, low P/F-ratios and high driving pressures demonstrated a higher probability of death. The PDP for driving pressure showed a relative probability increase starting at 12 cmH(2)O. Conclusion: Age is the most important demographic risk factor of ICU mortality, ICU-free days and ventilator-free days throughout the course of invasive mechanical ventilation in critically ill COVID-19 patients. pH, P/F ratio, and driving pressure should be monitored closely over the course of mechanical ventilation as risk factors predictive of these outcomes

Rapid Evaluation of Coronavirus Illness Severity (RECOILS) in intensive care: Development and validation of a prognostic tool for in-hospital mortality

Background The prediction of in-hospital mortality for ICU patients with COVID-19 is fundamental to treatment and resource allocation. The main purpose was to develop an easily implemented score for such prediction. Methods This was an observational, multicenter, development, and validation study on a national critical care dataset of COVID-19 patients. A systematic literature review was performed to determine variables possibly important for COVID-19 mortality prediction. Using a logistic multivariable model with a LASSO penalty, we developed the Rapid Evaluation of Coronavirus Illness Severity (RECOILS) score and compared its performance against published scores. Results Our development (validation) cohort consisted of 1480 (937) adult patients from 14 (11) Dutch ICUs admitted between March 2020 and April 2021. Median age was 65 (65) years, 31% (26%) died in hospital, 74% (72%) were males, average length of ICU stay was 7.83 (10.25) days and average length of hospital stay was 15.90 (19.92) days. Age, platelets, PaO2/FiO2 ratio, pH, blood urea nitrogen, temperature, PaCO2, Glasgow Coma Scale (GCS) score measured within +/−24 h of ICU admission were used to develop the score. The AUROC of RECOILS score was 0.75 (CI 0.71–0.78) which was higher than that of any previously reported predictive scores (0.68 [CI 0.64–0.71], 0.61 [CI 0.58–0.66], 0.67 [CI 0.63–0.70], 0.70 [CI 0.67–0.74] for ISARIC 4C Mortality Score, SOFA, SAPS-III, and age, respectively). Conclusions Using a large dataset from multiple Dutch ICUs, we developed a predictive score for mortality of COVID-19 patients admitted to ICU, which outperformed other predictive scores reported so far.ISSN:0001-5172ISSN:1399-657

Predicting responders to prone positioning in mechanically ventilated patients with COVID-19 using machine learning

Background: For mechanically ventilated critically ill COVID-19 patients, prone positioning has quickly become an important treatment strategy, however, prone positioning is labor intensive and comes with potential adverse effects. Therefore, identifying which critically ill intubated COVID-19 patients will benefit may help allocate labor resources. Methods: From the multi-center Dutch Data Warehouse of COVID-19 ICU patients from 25 hospitals, we selected all 3619 episodes of prone positioning in 1142 invasively mechanically ventilated patients. We excluded episodes longer than 24 h. Berlin ARDS criteria were not formally documented. We used supervised machine learning algorithms Logistic Regression, Random Forest, Naive Bayes, K-Nearest Neighbors, Support Vector Machine and Extreme Gradient Boosting on readily available and clinically relevant features to predict success of prone positioning after 4 h (window of 1 to 7 h) based on various possible outcomes. These outcomes were defined as improvements of at least 10% in PaO2/FiO2 ratio, ventilatory ratio, respiratory system compliance, or mechanical power. Separate models were created for each of these outcomes. Re-supination within 4 h after pronation was labeled as failure. We also developed models using a 20 mmHg improvement cut-off for PaO2/FiO2 ratio and using a combined outcome parameter. For all models, we evaluated feature importance expressed as contribution to predictive performance based on their relative ranking. Results: The median duration of prone episodes was 17 h (11–20, median and IQR, N = 2632). Despite extensive modeling using a plethora of machine learning techniques and a large number of potentially clinically relevant features, discrimination between responders and non-responders remained poor with an area under the receiver operator characteristic curve of 0.62 for PaO2/FiO2 ratio using Logistic Regression, Random Forest and XGBoost. Feature importance was inconsistent between models for different outcomes. Notably, not even being a previous responder to prone positioning, or PEEP-levels before prone positioning, provided any meaningful contribution to predicting a successful next proning episode. Conclusions: In mechanically ventilated COVID-19 patients, predicting the success of prone positioning using clinically relevant and readily available parameters from electronic health records is currently not feasible. Given the current evidence base, a liberal approach to proning in all patients with severe COVID-19 ARDS is therefore justified and in particular regardless of previous results of proning

Incidence, Risk Factors and Outcome of Suspected Central Venous Catheter-related Infections in Critically Ill COVID-19 Patients: A Multicenter Retrospective Cohort Study

BACKGROUND: Aims of this study were to investigate the prevalence and incidence of catheter-related infection, identify risk factors and determine the relation of catheter-related infection with mortality in critically ill COVID-19 patients. METHODS: This was a retrospective cohort study of central venous catheters (CVCs) in critically ill COVID-19 patients. Eligible CVC insertions required an indwelling time of at least 48 hours and were identified using a full-admission electronic healthrecord database. Risk factors were identified using logistic regression. Differences in survival rates at day 28 of follow-up were assessed using a log-rank test and proportional hazard model. RESULTS: In 538 patients, a total of 914 CVCs were included. Prevalence and incidence of suspected catheter-related infection were 7.9% and 9.4 infections per 1000 catheter indwelling days, respectively. Prone ventilation for more than five days was associated with increased risk of suspected catheter-related infection; odds ratio: 5.05 (95 CI: 2.12 - 11.0). Risk of death was significantly higher in patients with suspected catheter-related infection; hazard ratio: 1.78 (95% CI: 1.25 - 2.53). CONCLUSION: This study shows that in critically ill patients with COVID-19 prevalence and incidence of suspected catheter-related infection is high, prone ventilation is a risk factor and mortality is higher in case of catheter-related infection

Assess and validate predictive performance of models for in-hospital mortality in COVID-19 patients: A retrospective cohort study in the Netherlands comparing the value of registry data with high-granular electronic health records

Purpose : To assess, validate and compare the predictive performance of models for in-hospital mortality of COVID-19 patients admitted to the intensive care unit (ICU) over two different waves of infections. Our models were built with high-granular Electronic Health Records (EHR) data versus less-granular registry data. Methods : Observational study of all COVID-19 patients admitted to 19 Dutch ICUs participating in both the national quality registry National Intensive Care Evaluation (NICE) and the EHR-based Dutch Data Warehouse (hereafter EHR). Multiple models were developed on data from the first 24 h of ICU admissions from February to June 2020 (first COVID-19 wave) and validated on prospective patients admitted to the same ICUs between July and December 2020 (second COVID-19 wave). We assessed model discrimination, calibration, and the degree of relatedness between development and validation population. Coefficients were used to identify relevant risk factors. Results : A total of 1533 patients from the EHR and 1563 from the registry were included. With high granular EHR data, the average AUROC was 0.69 (standard deviation of 0.05) for the internal validation, and the AUROC was 0.75 for the temporal validation. The registry model achieved an average AUROC of 0.76 (standard deviation of 0.05) in the internal validation and 0.77 in the temporal validation. In the EHR data, age, and respiratory-system related variables were the most important risk factors identified. In the NICE registry data, age and chronic respiratory insufficiency were the most important risk factors. Conclusion : In our study, prognostic models built on less-granular but readily-available registry data had similar performance to models built on high-granular EHR data and showed similar transportability to a prospective COVID-19 population. Future research is needed to verify whether this finding can be confirmed for upcoming waves