Datathons and Software to Promote Reproducible Research

Background: Datathons facilitate collaboration between clinicians, statisticians, and data scientists in order to answer important clinical questions. Previous datathons have resulted in numerous publications of interest to the critical care community and serve as a viable model for interdisciplinary collaboration. Objective: We report on an open-source software called Chatto that was created by members of our group, in the context of the second international Critical Care Datathon, held in September 2015. Methods: Datathon participants formed teams to discuss potential research questions and the methods required to address them. They were provided with the Chatto suite of tools to facilitate their teamwork. Each multidisciplinary team spent the next 2 days with clinicians working alongside data scientists to write code, extract and analyze data, and reformulate their queries in real time as needed. All projects were then presented on the last day of the datathon to a panel of judges that consisted of clinicians and scientists. Results: Use of Chatto was particularly effective in the datathon setting, enabling teams to reduce the time spent configuring their research environments to just a few minutes—a process that would normally take hours to days. Chatto continued to serve as a useful research tool after the conclusion of the datathon. Conclusions: This suite of tools fulfills two purposes: (1) facilitation of interdisciplinary teamwork through archiving and version control of datasets, analytical code, and team discussions, and (2) advancement of research reproducibility by functioning postpublication as an online environment in which independent investigators can rerun or modify analyses with relative ease. With the introduction of Chatto, we hope to solve a variety of challenges presented by collaborative data mining projects while improving research reproducibility

The association between the neutrophil-to-lymphocyte ratio and mortality in critical illness: an observational cohort study

Introduction The neutrophil-to-lymphocyte ratio (NLR) is a biological marker that has been shown to be associated with outcomes in patients with a number of different malignancies. The objective of this study was to assess the relationship between NLR and mortality in a population of adult critically ill patients. Methods We performed an observational cohort study of unselected intensive care unit (ICU) patients based on records in a large clinical database. We computed individual patient NLR and categorized patients by quartile of this ratio. The association of NLR quartiles and 28-day mortality was assessed using multivariable logistic regression. Secondary outcomes included mortality in the ICU, in-hospital mortality and 1-year mortality. An a priori subgroup analysis of patients with versus without sepsis was performed to assess any differences in the relationship between the NLR and outcomes in these cohorts. Results A total of 5,056 patients were included. Their 28-day mortality rate was 19%. The median age of the cohort was 65 years, and 47% were female. The median NLR for the entire cohort was 8.9 (interquartile range, 4.99 to 16.21). Following multivariable adjustments, there was a stepwise increase in mortality with increasing quartiles of NLR (first quartile: reference category; second quartile odds ratio (OR) = 1.32; 95% confidence interval (CI), 1.03 to 1.71; third quartile OR = 1.43; 95% CI, 1.12 to 1.83; 4th quartile OR = 1.71; 95% CI, 1.35 to 2.16). A similar stepwise relationship was identified in the subgroup of patients who presented without sepsis. The NLR was not associated with 28-day mortality in patients with sepsis. Increasing quartile of NLR was statistically significantly associated with secondary outcome. Conclusion The NLR is associated with outcomes in unselected critically ill patients. In patients with sepsis, there was no statistically significant relationship between NLR and mortality. Further investigation is required to increase understanding of the pathophysiology of this relationship and to validate these findings with data collected prospectively.National Institutes of Health (U.S.) (Grant R01 EB017205-01A1

The effect of age and clinical circumstances on the outcome of red blood cell transfusion in critically ill patients

Introduction: Whether red blood cell (RBC) transfusion is beneficial remains controversial. In both retrospective and prospective evaluations, transfusion has been associated with adverse, neutral, or protective effects. These varying results likely stem from a complex interplay between transfusion, patient characteristics, and clinical context. The objective was to test whether age, comorbidities, and clinical context modulate the effect of transfusion on survival. Methods: By using the multiparameter intelligent monitoring in intensive care II database (v. 2.6), a retrospective analysis of 9,809 critically ill patients, we evaluated the effect of RBC transfusion on 30-day and 1-year mortality. Propensity score modeling and logistic regression adjusted for known confounding and assessed the independent effect of transfusion on 30-day and 1-year mortality. Sensitivity analysis was performed by using 3,164 transfused and non-transfused pairs, matched according the previously validated propensity model for RBC transfusion. Results: RBC transfusion did not affect 30-day or 1-year mortality in the overall cohort. Patients younger than 55 years had increased odds of mortality (OR, 1.71; P < 0.01) with transfusion. Patients older than 75 years had lower odds of 30-day and 1-year mortality (OR, 0.70; P < 0.01) with transfusion. Transfusion was associated with worse outcome among patients undergoing cardiac surgery (OR, 2.1; P < 0.01). The propensity-matched population corroborated findings identified by regression adjustment. Conclusion: A complex relation exists between RBC transfusion and clinical outcome. Our results show that transfusion is associated with improved outcomes in some cohorts and worse outcome in others, depending on comorbidities and patient characteristics. As such, future investigations and clinical decisions evaluating the value of transfusion should account for variations in baseline characteristics and clinical context. Electronic supplementary material The online version of this article (doi:10.1186/s13054-014-0487-z) contains supplementary material, which is available to authorized users

Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States

Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. Starting in April 2020, the US COVID-19 Forecast Hub (https://covid19forecasthub.org/) collected, disseminated, and synthesized tens of millions of specific predictions from more than 90 different academic, industry, and independent research groups. A multimodel ensemble forecast that combined predictions from dozens of groups every week provided the most consistently accurate probabilistic forecasts of incident deaths due to COVID-19 at the state and national level from April 2020 through October 2021. The performance of 27 individual models that submitted complete forecasts of COVID-19 deaths consistently throughout this year showed high variability in forecast skill across time, geospatial units, and forecast horizons. Two-thirds of the models evaluated showed better accuracy than a naïve baseline model. Forecast accuracy degraded as models made predictions further into the future, with probabilistic error at a 20-wk horizon three to five times larger than when predicting at a 1-wk horizon. This project underscores the role that collaboration and active coordination between governmental public-health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks

The United States COVID-19 Forecast Hub dataset

Academic researchers, government agencies, industry groups, and individuals have produced forecasts at an unprecedented scale during the COVID-19 pandemic. To leverage these forecasts, the United States Centers for Disease Control and Prevention (CDC) partnered with an academic research lab at the University of Massachusetts Amherst to create the US COVID-19 Forecast Hub. Launched in April 2020, the Forecast Hub is a dataset with point and probabilistic forecasts of incident cases, incident hospitalizations, incident deaths, and cumulative deaths due to COVID-19 at county, state, and national, levels in the United States. Included forecasts represent a variety of modeling approaches, data sources, and assumptions regarding the spread of COVID-19. The goal of this dataset is to establish a standardized and comparable set of short-term forecasts from modeling teams. These data can be used to develop ensemble models, communicate forecasts to the public, create visualizations, compare models, and inform policies regarding COVID-19 mitigation. These open-source data are available via download from GitHub, through an online API, and through R packages

“Yes, but will it work for my patients?” Driving clinically relevant research with benchmark datasets

Benchmark datasets have a powerful normative influence: by determining how the real world is represented in data, they define which problems will first be solved by algorithms built using the datasets and, by extension, who these algorithms will work for. It is desirable for these datasets to serve four functions: (1) enabling the creation of clinically relevant algorithms; (2) facilitating like-for-like comparison of algorithmic performance; (3) ensuring reproducibility of algorithms; (4) asserting a normative influence on the clinical domains and diversity of patients that will potentially benefit from technological advances. Without benchmark datasets that satisfy these functions, it is impossible to address two perennial concerns of clinicians experienced in computational research: “the data scientists just go where the data is rather than where the needs are,” and, “yes, but will this work for my patients?” If algorithms are to be developed and applied for the care of patients, then it is prudent for the research community to create benchmark datasets proactively, across specialties. As yet, best practice in this area has not been defined. Broadly speaking, efforts will include design of the dataset; compliance and contracting issues relating to the sharing of sensitive data; enabling access and reuse; and planning for translation of algorithms to the clinical environment. If a deliberate and systematic approach is not followed, not only will the considerable benefits of clinical algorithms fail to be realized, but the potential harms may be regressively incurred across existing gradients of social inequity.National Institutes of Health (Grant R01 EV017205

A Systematic Review of ‘Fair’ AI Model Development for Image Classification and Prediction

Abstract Purpose The new challenge in Artificial Intelligence (AI) is to understand the limitations of models to reduce potential harm. Particularly, unknown disparities based on demographic factors could encrypt currently existing inequalities worsening patient care for some groups. Methods Following PRISMA guidelines, we present a systematic review of ‘fair’ deep learning modeling techniques for natural and medical image applications which were published between year 2011 to 2021. Our search used Covidence review management software and incorporates articles from PubMed, IEEE, and ACM search engines and three reviewers independently review the manuscripts. Results Inter-rater agreement was 0.89 and conflicts were resolved by obtaining consensus between three reviewers. Our search initially retrieved 692 studies but after careful screening, our review included 22 manuscripts that carried four prevailing themes; ‘fair’ training dataset generation (4/22), representation learning (10/22), model disparity across institutions (5/22) and model fairness with respect to patient demographics (3/22). We benchmark the current literature regarding fairness in AI-based image analysis and highlighted the existing challenges. We observe that often discussion regarding fairness are limited to analyzing existing bias without further establishing methodologies to overcome model disparities. Conclusion Based on the current research trends, exploration of adversarial learning for demographic/camera/institution agnostic models is an important direction to minimize disparity gaps for imaging. Privacy preserving approaches also present encouraging performance for both natural and medical image domain

A Clinical Database-Driven Approach to Decision Support: Predicting Mortality Among Patients with Acute Kidney Injury

In exploring an approach to decision support based on information extracted from a clinical database, we developed mortality prediction models of intensive care unit (ICU) patients who had acute kidney injury (AKI) and compared them against the Simplified Acute Physiology Score (SAPS). We used MIMIC, a public de-identified database of ICU patients admitted to Beth Israel Deaconess Medical Center, and identified 1400 patients with an ICD9 diagnosis of AKI and who had an ICU stay ≥3 days. Multivariate regression models were built using the SAPS variables from the first 72 hours of ICU admission. All the models developed on the training set performed better than SAPS (AUC = 0.64, Hosmer-Lemeshow p <0.001) on an unseen test set; the best model had an AUC = 0.74 and Hosmer-Lemeshow p = 0.53. These findings suggest that local customized modeling might provide more accurate predictions. This could be the first step towards an envisioned individualized point-of-care probabilistic modeling using one's clinical database