Exploring objective measures for assessing team performance in healthcare: an interview study.

INTRODUCTION Effective teamwork plays a critical role in achieving high-performance outcomes in healthcare. Consequently, conducting a comprehensive assessment of team performance is essential for providing meaningful feedback during team trainings and enabling comparisons in scientific studies. However, traditional methods like self-reports or behavior observations have limitations such as susceptibility to bias or being resource consuming. To overcome these limitations and gain a more comprehensive understanding of team processes and performance, the assessment of objective measures, such as physiological parameters, can be valuable. These objective measures can complement traditional methods and provide a more holistic view of team performance. The aim of this study was to explore the potential of the use of objective measures for evaluating team performance for research and training purposes. For this, experts in the field of research and medical simulation training were interviewed to gather their opinions, ideas, and concerns regarding this novel approach. METHODS A total of 34 medical and research experts participated in this exploratory qualitative study, engaging in semi-structured interviews. During the interview, experts were asked for (a) their opinion on measuring team performance with objective measures, (b) their ideas concerning potential objective measures suitable for measuring team performance of healthcare teams, and (c) their concerns regarding the use of objective measures for evaluating team performance. During data analysis responses were categorized per question. RESULTS The findings from the 34 interviews revealed a predominantly positive reception of the idea of utilizing objective measures for evaluating team performance. However, the experts reported limited experience in actively incorporating objective measures into their training and research. Nevertheless, they identified various potential objective measures, including acoustical, visual, physiological, and endocrinological measures and a time layer. Concerns were raised regarding feasibility, complexity, cost, and privacy issues associated with the use of objective measures. DISCUSSION The study highlights the opportunities and challenges associated with employing objective measures to assess healthcare team performance. It particularly emphasizes the concerns expressed by medical simulation experts and team researchers, providing valuable insights for developers, trainers, researchers, and healthcare professionals involved in the design, planning or utilization of objective measures in team training or research

Barriers to seeking emergency care during the COVID-19 pandemic may lead to higher morbidity and mortality – a retrospective study from a Swiss university hospital

AIMS OF THE STUDY While COVID-19 significantly overburdens emergency rooms (ERs) and hospitals in affected areas, ERs elsewhere report a marked decrease in patient numbers. This study aimed to investigate the assumption that patients with urgent problems currently avoid the ER. METHODS Electronic health records from the ER of a large Swiss university hospital were extracted for three periods: first, the awareness phase (ap) from the publication of the national government’s initiative “How to protect ourselves” on 1 March 2020 to the lockdown of the country on 16 March; second, the mitigation phase (mp) from 16–30 March; finally, patients presenting in March 2019 were used as a control group. We compared parameters including a critical illness as the discharge diagnosis (e.g., myocardial infarction, stroke, sepsis and ER death) using logistic and linear regression, as well as 15-day bootstrapped means and 95% confidence intervals for the control group. RESULTS In the three periods, a total of 7143 patients were treated. We found a 24.9% (42.5%) significant decline in the number of patients presenting during the ap (mp). Patients presenting during the mp were more likely to be critically ill. There was an increase of 233% and 367% (ap and mp, respectively) of ER deaths (none related to COVID-19) compared to the control period. Apart from polytrauma (increase of 5% in the mp), all other critical illnesses as discharge diagnosis showed a lower incidence in descriptive analysis. Significantly more patients died in the ER in both the ap and mp. CONCLUSIONS Barriers to seeking emergency care during COVID-19 pandemic may lead to higher morbidity and mortality. Healthcare authorities and hospitals must ensure low barriers to treatment and business as usual for all patients

Use of epinephrine in anaphylaxis: a retrospective cohort study at a Swiss university emergency department.

AIMS OF THE STUDY Anaphylaxis is a medical emergency and requires prompt treatment to prevent life-threatening conditions. Epinephrine, considered as the first-line drug, is often not administered. We aimed first to analyse the use of epinephrine in patients with anaphylaxis in the emergency department of a university hospital and secondly to identify factors that influence the use of epinephrine. METHODS We performed a retrospective analysis of all patients admitted with moderate or severe anaphylaxis to the emergency department between 1 January 2013 and 31 December 2018. Patient characteristics and treatment information were extracted from the electronic medical database of the emergency department. RESULTS A total of 531 (0.2%) patients with moderate or severe anaphylaxis out of 260,485 patients admitted to the emergency department were included. Epinephrine was administered in 252 patients (47.3%). In a multivariate logistic regression, cardiovascular (Odds Ratio [OR] = 2.94, CI 1.96-4.46, p <0.001) and respiratory symptoms (OR = 3.14, CI 1.95-5.14, p<0.001) were associated with increased likelihood of epinephrine administration, in contrast to integumentary symptoms (OR = 0.98, CI 0.54-1.81, p = 0.961) and gastrointestinal symptoms (OR = 0.62, CI 0.39-1.00, p = 0.053). CONCLUSIONS Less than half of the patients with moderate and severe anaphylaxis received epinephrine according to guidelines. In particular, gastrointestinal symptoms seem to be misrecognised as serious symptoms of anaphylaxis. Training of the emergency medical services and emergency department medical staff and further awareness are crucial to increase the administration rate of epinephrine in anaphylaxis

Is language an issue? Accuracy of the German computerized diagnostic decision support system ISABEL and cross-validation with the English counterpart.

OBJECTIVES Existing computerized diagnostic decision support tools (CDDS) accurately return possible differential diagnoses (DDx) based on the clinical information provided. The German versions of the CDDS tools for clinicians (Isabel Pro) and patients (Isabel Symptom Checker) from ISABEL Healthcare have not been validated yet. METHODS We entered clinical features of 50 patient vignettes taken from an emergency medical text book and 50 real cases with a confirmed diagnosis derived from the electronic health record (EHR) of a large academic Swiss emergency room into the German versions of Isabel Pro and Isabel Symptom Checker. We analysed the proportion of DDx lists that included the correct diagnosis. RESULTS Isabel Pro and Symptom Checker provided the correct diagnosis in 82 and 71 % of the cases, respectively. Overall, the correct diagnosis was ranked in 71 , 61 and 37 % of the cases within the top 20, 10 and 3 of the provided DDx when using Isabel Pro. In general, accuracy was higher with vignettes than ED cases, i.e. listed the correct diagnosis more often (non-significant) and ranked the diagnosis significantly more often within the top 20, 10 and 3. On average, 38 ± 4.5 DDx were provided by Isabel Pro and Symptom Checker. CONCLUSIONS The German versions of Isabel achieved a somewhat lower accuracy compared to previous studies of the English version. The accuracy decreases substantially when the position in the suggested DDx list is taken into account. Whether Isabel Pro is accurate enough to improve diagnostic quality in clinical ED routine needs further investigation

a prospective observational study

Introduction Emergency rooms (ERs) generally assign a preliminary diagnosis to patients, who are then hospitalised and may subsequently experience a change in their lead diagnosis (cDx). In ERs, the cDx rate varies from around 15% to more than 50%. Among the most frequent reasons for diagnostic errors are cognitive slips, which mostly result from faulty data synthesis. Furthermore, physicians have been repeatedly found to be poor self-assessors and to be overconfident in the quality of their diagnosis, which limits their ability to improve. Therefore, some of the clinically most relevant research questions concern how diagnostic decisions are made, what determines their quality and what can be done to improve them. Research that addresses these questions is, however, still rare. In particular, field studies that allow for generalising findings from controlled experimental settings are lacking. The ER, with its high throughput and its many simultaneous visits, is perfectly suited for the study of factors contributing to diagnostic error. With this study, we aim to identify factors that allow prediction of an ER's diagnostic performance. Knowledge of these factors as well as of their relative importance allows for the development of organisational, medical and educational strategies to improve the diagnostic performance of ERs. Methods and analysis We will conduct a field study by collecting diagnostic decision data, physician confidence and a number of influencing factors in a real-world setting to model real-world diagnostic decisions and investigate the adequacy, validity and informativeness of physician confidence in these decisions. We will specifically collect data on patient, physician and encounter factors as predictors of the dependent variables. Statistical methods will include analysis of variance and a linear mixed-effects model. Ethics and dissemination The Bern ethics committee approved the study under KEK Number 197/15. Results will be published in peer-reviewed scientific medical journals. Authorship will be determined according to ICMJE guidelines. Trial registration number The study protocol Version 1.0 from 17 May 2015 is registered in the Inselspital Research Database Information System (IRDIS) and with the IRB (‘Kantonale Ethikkomission’) Bern under KEK Number 197/15

A validity argument for progress testing: Examining the relation between growth trajectories obtained by progress tests and national licensing examinations using a latent growth curve approach

Background: Progress testing is a longitudinal assessment that aims at tracking students' development of knowledge. This approach is used in many medical schools internationally. Although progress tests are longitudinal in nature, and their focus and use of developmental aspects is a key advantage, individual students' learning trajectories themselves play, to date, only a minor role in the use of the information obtained through progress testing. Methods: We investigate in how far between-person differences in initial levels of performance and within-person rate of growth can be regarded as distinct components of students' development and analyze the extent to which these two components are related to performances on national licensing examinations using a latent growth curve model. Results: Both, higher initial levels of performances and steepness of growth are positively related to long-term outcomes as measured by performance on national licensing examinations. We interpret these findings as evidence for progress tests' suitability to monitor students' growth of knowledge across the course of medical training. Conclusions: This study indicates that individual development as obtained by formative progress tests is related to performance in high-stakes assessments. Future studies may put more focus on the use of between-persons differences in growth of knowledge

No knowledge gap in human physiology after remote teaching for second year medical students throughout the Covid-19 pandemic

Abstract The COVID-19 pandemic had a disruptive effect on higher education. A critical question is whether these changes affected students’ learning outcomes. Knowledge gaps have consequences for future learning and may—in health professionals' education—also pose a threat to patient safety. Current research has shortcomings and does not allow for clear-cut interpretation. Our context is instruction in human physiology in an undergraduate medical program from high stakes end of term examinations. The sequence of imposed measures to slow the COVID-19 pandemic created a natural experiment, allowing for comparisons in performance during in-person versus remote instruction. In a two-factorial design, mode of instruction (in-person vs. remote) and mode of assessment (in-person vs. remote) were analyzed using both basic (non-parametric statistics, T-tests) and advanced statistical methods (linear mixed-effects model; resampling techniques). Test results from a total of N = 1095 s-year medical students were included in the study. We did not find empirical evidence of knowledge gaps; rather, students received comparable or higher scores during remote teaching. We interpret these findings as empirical evidence that both students and teachers adapted to pandemic disruption in a way that did not lead to knowledge gaps. We conclude that highly motivated students had no reduction in academic achievement. Moreover, we have developed an accessible digital exam system for secure, fair, and effective assessments which is sufficiently defensible for making pass/fail decisions

Progress testing anytime and anywhere - Does a mobile-learning approach enhance the utility of a large-scale formative assessment tool?

Background The widespread use of mobile devices among students favors the use of mobile learning scenarios at universities. In this study, we explore whether a time- and location-independent variant of a formative progress test has an impact on the students' acceptance, its validity and reliability and if there is a difference in response processes between the two exam conditions. Methods Students were randomly assigned to two groups of which one took the test free of local or temporal fixations, while the other group took the test at the local testing center under usual examination conditions. Beside the generated test data, such as test score, time-on-test, and semester status, students also evaluated the settings. Results While there was no significant effect on the test score between the two groups, students in the mobile group spent more time on the test and were more likely to use the help of books or online resources. The results of the evaluation show that the acceptability among students is increased by a mobile version of the formative progress test. Conclusions The results suggest that the acceptance and motivation to participate in formative tests is enhanced by lifting local and temporal restrictions. The mobile version nonetheless does not have an impact on the students' performance

Why assessment in medical education needs a solid foundation in modern test theory

Despite the frequent use of state-of-the-art psychometric models in the field of medical education, there is a growing body of literature that questions their usefulness in the assessment of medical competence. Essentially, a number of authors raised doubt about the appropriateness of psychometric models as a guiding framework to secure and refine current approaches to the assessment of medical competence. In addition, an intriguing phenomenon known as case specificity is specific to the controversy on the use of psychometric models for the assessment of medical competence. Broadly speaking, case specificity is the finding of instability of performances across clinical cases, tasks, or problems. As stability of performances is, generally speaking, a central assumption in psychometric models, case specificity may limit their applicability. This has probably fueled critiques of the field of psychometrics with a substantial amount of potential empirical evidence. This article aimed to explain the fundamental ideas employed in psychometric theory, and how they might be problematic in the context of assessing medical competence. We further aimed to show why and how some critiques do not hold for the field of psychometrics as a whole, but rather only for specific psychometric approaches. Hence, we highlight approaches that, from our perspective, seem to offer promising possibilities when applied in the assessment of medical competence. In conclusion, we advocate for a more differentiated view on psychometric models and their usage