Assessing optimal methods for transferring machine learning models to low-volume and imbalanced clinical datasets: experiences from predicting outcomes of Danish trauma patients

IntroductionAccurately predicting patient outcomes is crucial for improving healthcare delivery, but large-scale risk prediction models are often developed and tested on specific datasets where clinical parameters and outcomes may not fully reflect local clinical settings. Where this is the case, whether to opt for de-novo training of prediction models on local datasets, direct porting of externally trained models, or a transfer learning approach is not well studied, and constitutes the focus of this study. Using the clinical challenge of predicting mortality and hospital length of stay on a Danish trauma dataset, we hypothesized that a transfer learning approach of models trained on large external datasets would provide optimal prediction results compared to de-novo training on sparse but local datasets or directly porting externally trained models.MethodsUsing an external dataset of trauma patients from the US Trauma Quality Improvement Program (TQIP) and a local dataset aggregated from the Danish Trauma Database (DTD) enriched with Electronic Health Record data, we tested a range of model-level approaches focused on predicting trauma mortality and hospital length of stay on DTD data. Modeling approaches included de-novo training of models on DTD data, direct porting of models trained on TQIP data to the DTD, and a transfer learning approach by training a model on TQIP data with subsequent transfer and retraining on DTD data. Furthermore, data-level approaches, including mixed dataset training and methods countering imbalanced outcomes (e.g., low mortality rates), were also tested.ResultsUsing a neural network trained on a mixed dataset consisting of a subset of TQIP and DTD, with class weighting and transfer learning (retraining on DTD), we achieved excellent results in predicting mortality, with a ROC-AUC of 0.988 and an F2-score of 0.866. The best-performing models for predicting long-term hospitalization were trained only on local data, achieving an ROC-AUC of 0.890 and an F1-score of 0.897, although only marginally better than alternative approaches.ConclusionOur results suggest that when assessing the optimal modeling approach, it is important to have domain knowledge of how incidence rates and workflows compare between hospital systems and datasets where models are trained. Including data from other health-care systems is particularly beneficial when outcomes are suffering from class imbalance and low incidence. Scenarios where outcomes are not directly comparable are best addressed through either de-novo local training or a transfer learning approach

The role of point of care ultrasound in prehospital critical care: a systematic review

Abstract Background In 2011, the role of Point of Care Ultrasound (POCUS) was defined as one of the top five research priorities in physician-provided prehospital critical care and future research topics were proposed; the feasibility of prehospital POCUS, changes in patient management induced by POCUS and education of providers. This systematic review aimed to assess these three topics by including studies examining all kinds of prehospital patients undergoing all kinds of prehospital POCUS examinations and studies examining any kind of POCUS education in prehospital critical care providers. Methods and results By a systematic literature search in MEDLINE, EMBASE, and Cochrane databases, we identified and screened titles and abstracts of 3264 studies published from 2012 to 2017. Of these, 65 studies were read in full-text for assessment of eligibility and 27 studies were ultimately included and assessed for quality by SIGN-50 checklists. No studies compared patient outcome with and without prehospital POCUS. Four studies of acceptable quality demonstrated feasibility and changes in patient management in trauma. Two studies of acceptable quality demonstrated feasibility and changes in patient management in breathing difficulties. Four studies of acceptable quality demonstrated feasibility, outcome prediction and changes in patient management in cardiac arrest, but also that POCUS may prolong pauses in compressions. Two studies of acceptable quality demonstrated that short (few hours) teaching sessions are sufficient for obtaining simple interpretation skills, but not image acquisition skills. Three studies of acceptable quality demonstrated that longer one- or two-day courses including hands-on training are sufficient for learning simple, but not advanced, image acquisition skills. Three studies of acceptable quality demonstrated that systematic educational programs including supervised examinations are sufficient for learning advanced image acquisition skills in healthy volunteers, but that more than 50 clinical examinations are required for expertise in a clinical setting. Conclusion Prehospital POCUS is feasible and changes patient management in trauma, breathing difficulties and cardiac arrest, but it is unknown if this improves outcome. Expertise in POCUS requires extensive training by a combination of theory, hands-on training and a substantial amount of clinical examinations – a large part of these needs to be supervised

Effect of ultrasound training of physicians working in the prehospital setting

BACKGROUND: Advances in technology have made ultrasound (US) devices smaller and portable, hence accessible for prehospital care providers. This study aims to evaluate the effect of a four-hour, hands-on US training course for physicians working in the prehospital setting. The primary outcome measure was US performance assessed by the total score in a modified version of the Objective Structured Assessment of Ultrasound Skills scale (mOSAUS). METHODS: Prehospital physicians participated in a four-hour US course consisting of both hands-on training and e-learning including a pre- and a post-learning test. Prior to the hands-on training a pre-training test was applied comprising of five videos in which the participants should identify pathology and a five-minute US examination of a healthy volunteer portraying to be a shocked patient after a blunt torso trauma. Following the pre-training test, the participants received a four-hour, hands-on US training course which was concluded with a post-training test. The US examinations and screen output from the US equipment were recorded for subsequent assessment. Two blinded raters assessed the videos using the mOSAUS. RESULTS: Forty participants completed the study. A significant improvement was identified in e-learning performance and US performance, (37.5 (SD: 10.0)) vs. (51.3 (SD: 5.9) p = < 0.0001), total US performance score (15.3 (IQR: 12.0-17.5) vs. 17.5 (IQR: 14.5-21.0), p = < 0.0001) and in each of the five assessment elements of the mOSAUS. CONCLUSION: In the prehospital physicians assessed, we found significant improvements in the ability to perform US examinations after completing a four-hour, hands-on US training course