Exploring the introduction of entrustment rating scales in an existing objective structured clinical examination

Background: The concept of EPAs is increasingly applied to assess trainees’ workplace performance by means of entrustment ratings. OSCEs assess performance in a simulated setting, and it is unclear whether entrustment ratings can be integrated into these exams. This study explores the introduction of an entrustment rating scale into an existing OSCE. Methods: A 6-point entrustment scale was added to the standard ratings in an OSCE administered prior to students’ final clerkship year in an undergraduate medical programme. Standard OSCE ratings assess clinical and communication skills. Assessors (n = 54) rated students’ performance (n = 227) on a diverse set of clinical tasks and evaluated the addition of entrustment scales to OSCEs. Descriptive and inferential statistics were calculated for analyses. Results: Student performance varied across the stations, as reflected in both the standard OSCE ratings and the added entrustment ratings. Students received generally high standard OSCE ratings, whereas entrustment ratings were more widely distributed. All students passed the OSCE, and only a small proportion of students did not reach the expected pass threshold of 60% on the standard ratings in the single stations. The proportion of students who did not reach the expected entrustment level in the respective stations was noticeably higher. Both the clinical and communication skill ratings were related to the entrustment rating in most OSCE stations. A majority of the assessors positively evaluated the addition of entrustment ratings into the OSCE. Discussion: The findings provide an empirical basis to broaden our understanding of the potential use of entrustment ratings in existing OSCEs. They provide directions for future, more specific studies. The ratings might be used for formative feedback on students’ readiness for workplace practice

Effects of COVID-19 Pandemic on Progress Test Performance in German-Speaking Countries

Background. The COVID-19 pandemic has been the source of many challenges for medical students worldwide. The authors examined short-term effects on the knowledge gain of medical students in German-speaking countries. Methods. The development of the knowledge gain of medical students during the pandemic was measured by comparing the outcomes of shared questions within Berlin Progress Test (PT) pairs. The PT is a formative test of 200 multiple choice questions at the graduate level, which provides feedback to students on knowledge and knowledge gain during their course of study. It is provided to about 11,000 students in Germany and Austria around the beginning of each semester. We analyzed three successive test pairs: PT36-PT41 (both conducted before the pandemic), PT37-PT42 (PT37 took place before the pandemic; PT42 was conducted from April 2020 onwards), and PT38-PT43 (PT38 was administered before the pandemic; PT43 started in November 2020). The authors used mixed-effect regression models and compared the absolute variations in the percentage of correct answers per subject. Results. The most recent test of each PT pair showed a higher mean score compared to the previous test in the same pair (PT36-PT41 : 2.53 (95% CI: 1.31-3.75), PT37-PT42 : 3.72 (2.57-4.88), and PT38-PT43 : 5.66 (4.63-6.69)). Analogously, an increase in the share of correct answers was observed for most medical disciplines, with Epidemiology showing the most remarkable upsurge. Conclusions. Overall, PT performance improved during the pandemic, which we take as an indication that the sudden shift to online learning did not have a negative effect on the knowledge gain of students. We consider that these results may be helpful in advancing innovative approaches to medical education

Communication skills training in undergraduate medical education at Charité - Universitätsmedizin Berlin.

Objective: The objective of this article is a description of the longitudinal communication curriculum in the Model Medicine Curriculum (MSM) at Charité - Universitätsmedizin Berlin. The authors describe the planning and integration of the curriculum into the study program, outline how communicative competence is taught and evaluated in the MSM, and identify which challenges need to be mastered in the process. Project description: Starting with the introduction of the MSM in 2010, students have been spending 102 class hours, spread out over seven semesters, practicing social and communicative competences in the interactive small group format "Communication, Interaction & Teamwork (KIT)". The course contents are closely linked to the topics covered each semester and increase in complexity over the course of their studies. The contents are selected by the KIT planning group whose members continually check the curriculum's timeliness and determine any changes. Students as well as instructors have opportunities for evaluating KIT throughout, and their evaluations are taken into consideration as KIT continues to be updated. Instructors from different disciplines teach KIT courses. They participate in mandatory didactic trainings that prepare them to teach KIT. During their 4th and 9th semesters, respectively, students take summative exams that test their communicative competence. Results: According to the semester evaluations by students and instructors, students participating in KIT improved their conversation management skills (students: M=2.2, SD=1.1, instructors: M=1.9, SD=0.7, on a scale of 1-5). In addition, students and graduates rate KIT to be (very) relevant, consider the degree to which it is taught in the MSM to be (very) high, and consider KIT to be a meaningful part of the curriculum. Students taking the summative exams in their 4th and 9th semesters achieve a mean score of 75.9%, respectively 76.9%, in the purely communicative stations and 82.6%, respectively 83.3%, in the global evaluation of communicative competence in clinical-practical stations. Discussion: Survey and exam results alike indicate that the communication training is well accepted by students and instructors and that the training led to an improvement in general and specific communicative skills. Due to a lack of control groups or a pre-post design, it has thus far not been possible to unequivocally demonstrate a causal relationship between communicative competence trainings and good test results. Quality control measures, such as trainings for instructors and regular course evaluations, have been designed to address any challenges in the implementation of the communication curriculum at the faculty level. Conclusion: Building on the experience with the Charité's Reformed Medical Curriculum, a longitudinal, competence-based communication curriculum was integrated into the MSM's overall curriculum. This measure remedied a gap in the medical training that many graduates of regular study programs had previously bemoaned (Jansen 2010 [1])

Das Visitengespräch als Lerninhalt im Medizinstudium

Fragestellung/Zielsetzung: Das Visitengespräch ist für Diagnostik, Prognoseeinschätzung und Therapieentscheidungen in interprofessionellen Stationsteams unerlässlich. Die Erwartungen des Stationsteams an das Gespräch unterscheiden sich oft grundlegend von denen der behandelten Person. Während für das interprofessionelle Versorgungsteam diagnostische Informationen und Therapieplanung im Vordergrund stehen, ist für die behandelte Person meist ihre emotionale Situation und die Bewältigung der Krankheitsfolgen zentral. Auch Störungen im Ablauf (z.B. Telefonanrufe, Piepser) lassen sich nicht immer vermeiden. Nicht zuletzt aus diesen Gründen muss die ärztlich tätige Person, welche die Visite leitet, eine hohe Gesprächsführungskompetenz zeigen, damit die Ergebnisse der Visite für alle Beteiligten befriedigend sind („verhinderter Dialog“). Die Studierenden im Modellstudiengang der Charité üben die Durchführung von Stationsvisiten in einem Kommunikationstraining. Das Training wird seit dem WS 2018/19 im fünften Semester durchgeführt und umfasst zwei Unterrichtseinheiten, in denen zunächst der idealtypische Ablauf der Visite und der Umgang mit Störungen thematisiert werden. Anschließend werden Rollenspiele durchgeführt und ausgewertet. Methoden: Das Kommunikationstraining wird nach Abschluss von den Studierenden online anhand standardisierter Items auf einer fünfstufigen Likert-Skala (stimme voll zu, stimme zu, teils/teils, stimme nicht zu, stimme gar nicht zu) evaluiert. Ergebnisse: Im WS 2018/19 und im SoSe 2019 evaluierten sowohl die Studierenden als auch die Lehrenden das Kommunikationstraining jeweils am Semesterende (fünfstufige Likert-Skala). Die Studierenden stimmten mit 54% voll oder teilweise der Aussage zu: „Ich habe durch das Training gelernt, ein Visitengespräch zu führen.“, 29% gaben Unentschiedenheit an und 17% stimmten nicht oder gar nicht zu. Die Lehrenden stimmten der Aussage „Die Studierenden haben durch das Training gelernt, ein Visitengespräch zu führen“ zu 61% voll oder teilweise zu, 30% gaben an, unentschieden zu sein und 9% stimmten nicht zu. Die Rücklaufquoten betrugen für die Studierenden 23% (WS 2018/19) und 16% (SoSe 2019) sowie für die Lehrenden 41% (WS 2018/19) und 24% (SoSe 2019). Diskussion: Nur etwa die Hälfte der Studierenden und etwa zwei Drittel der Dozierenden berichteten, die Kompetenz, ein Visitengespräch zu führen, durch das Training erworben zu haben. Diese Zustimmungswerte könnten darauf zurückzuführen sein, dass sich eine derart komplexe Gesprächsführungskompetenz nicht allein in einem Training von zwei Unterrichtseinheiten erlernen lässt. Es wird deswegen in höheren Semestern weitere Möglichkeiten im Studierendenunterricht geben, Visitengespräche zu reflektieren. So kann im Sinne einer Lernspirale diese Gesprächsführungskompetenz weiter verbessert werden. Take Home Message: Ein Training zur Durchführung von Visitengesprächen wurde realisiert. Studierende und Dozierende berichteten einen Kompentenzzuwachs

Discovering unknown response patterns in progress test data to improve the estimation of student performance

Abstract Background The Progress Test Medizin (PTM) is a 200-question formative test that is administered to approximately 11,000 students at medical universities (Germany, Austria, Switzerland) each term. Students receive feedback on their knowledge (development) mostly in comparison to their own cohort. In this study, we use the data of the PTM to find groups with similar response patterns. Methods We performed k-means clustering with a dataset of 5,444 students, selected cluster number k = 5, and answers as features. Subsequently, the data was passed to XGBoost with the cluster assignment as target enabling the identification of cluster-relevant questions for each cluster with SHAP. Clusters were examined by total scores, response patterns, and confidence level. Relevant questions were evaluated for difficulty index, discriminatory index, and competence levels. Results Three of the five clusters can be seen as “performance” clusters: cluster 0 (n = 761) consisted predominantly of students close to graduation. Relevant questions tend to be difficult, but students answered confidently and correctly. Students in cluster 1 (n = 1,357) were advanced, cluster 3 (n = 1,453) consisted mainly of beginners. Relevant questions for these clusters were rather easy. The number of guessed answers increased. There were two “drop-out” clusters: students in cluster 2 (n = 384) dropped out of the test about halfway through after initially performing well; cluster 4 (n = 1,489) included students from the first semesters as well as “non-serious” students both with mostly incorrect guesses or no answers. Conclusion Clusters placed performance in the context of participating universities. Relevant questions served as good cluster separators and further supported our “performance” cluster groupings