Exploring the introduction of entrustment rating scales in an existing objective structured clinical examination

Background: The concept of EPAs is increasingly applied to assess trainees’ workplace performance by means of entrustment ratings. OSCEs assess performance in a simulated setting, and it is unclear whether entrustment ratings can be integrated into these exams. This study explores the introduction of an entrustment rating scale into an existing OSCE. Methods: A 6-point entrustment scale was added to the standard ratings in an OSCE administered prior to students’ final clerkship year in an undergraduate medical programme. Standard OSCE ratings assess clinical and communication skills. Assessors (n = 54) rated students’ performance (n = 227) on a diverse set of clinical tasks and evaluated the addition of entrustment scales to OSCEs. Descriptive and inferential statistics were calculated for analyses. Results: Student performance varied across the stations, as reflected in both the standard OSCE ratings and the added entrustment ratings. Students received generally high standard OSCE ratings, whereas entrustment ratings were more widely distributed. All students passed the OSCE, and only a small proportion of students did not reach the expected pass threshold of 60% on the standard ratings in the single stations. The proportion of students who did not reach the expected entrustment level in the respective stations was noticeably higher. Both the clinical and communication skill ratings were related to the entrustment rating in most OSCE stations. A majority of the assessors positively evaluated the addition of entrustment ratings into the OSCE. Discussion: The findings provide an empirical basis to broaden our understanding of the potential use of entrustment ratings in existing OSCEs. They provide directions for future, more specific studies. The ratings might be used for formative feedback on students’ readiness for workplace practice

Prüfung von Baypamune® im Infektionsmodell der kaninen oralen Papillomatose am Hund

Canine Oral Papillomavirus (COPV) induces warts on the oral mucosa in domestic dogs and other canids. The canine oral papillomatosis (COP) is a well established animal model of mucosal papillomatosis. While the regression of a current infection is mediated by cellular immunity, humoral immunity does prevent reinfection. Question of the present study was, if unspecific stimulation of the immune system with the inducer of paramunity Baypamune® during the period of growth would have an influence on the course of canine oral papillomatosis. Clinical criteria for this purpose were period of growth, period of regression and the size and morphology of Papillomas at the end of growth. Additionally ALAT and ASAT were quantified in order to rule out deterioration of liver cells. PCV and leucocytes where monitored and a differentiation was performed. In a second part of the study, L1-antibodies and IFNg, TNF-a as well as IL-18 were determined by the Institute of Virology of the Faculty of Veterinary Medicine of the University of Leipzig. At the end of the study, biopsies of mucosa were taken for detection of viral genome. In a pre-study, three Labrador Retrievers at 14 weeks of age were challenged with 15 μl of virus suspension (40 μg of COPV-L1 Protein/ml) per site by scarification of oral mucosa. Papillomas developed at all challenged sites. These were removed during the period of growth and handed over to the Institute of Virology of the Faculty of Summary 70 Veterinary Medicine of the University of Leipzig for preparation of the suspension used in the main study. In the main study, 13 Labrador Retrievers at 14 weeks of age were challenged with 10 μl of virus suspension (40 μg of COPV-L1 Protein/ml) per site by scarification of three areas of the left oral mucosa. All animals developed at least at one site warts, totally 88 % of the challenged sites showed papillomas. 44 Days after infection, 12 dogs were divided into two groups which did not differ in time of incubation and size of the papillomas. Over a period of four weeks, one group received a total of five dosis of Baypamune®, the other group five dosis of placebo. There was no statistically significant difference between the groups regarding the period of growth, period of regression, size and morphology of the papillomas. The administration of Baypamune® during the period of growth had no effect on the clinical course of COP in this trial. However, time of application of the substances should be considered critically, since papillomas of five animals had been in regression before the first application of Baypamune® and placebo had taken place. No deterioration of liver cells could be ascertained. There was no difference between the groups regarding packed cell volume, white blood count and differentiation throughout the study. Parameters of unspecific immunity determined by the Institute of Virology, IFN-γ, TNF-a und IL-18, delivered no evaluable results. However, four animals of the placebogroup showed Papilloma-DNA in the Biopsies taken, but none of the animals, that received Baypamune®, being this a possible indicator for stimulation of cellular immunity. Anti-L1-Antibodies rose earlier in the Baypamune®group than in the placebogroup. In conclusion therapeutic efficacy of Baypamune® in dogs with present COP could not be shown in this trial, making future investigation necessary

Effects of COVID-19 Pandemic on Progress Test Performance in German-Speaking Countries

Background. The COVID-19 pandemic has been the source of many challenges for medical students worldwide. The authors examined short-term effects on the knowledge gain of medical students in German-speaking countries. Methods. The development of the knowledge gain of medical students during the pandemic was measured by comparing the outcomes of shared questions within Berlin Progress Test (PT) pairs. The PT is a formative test of 200 multiple choice questions at the graduate level, which provides feedback to students on knowledge and knowledge gain during their course of study. It is provided to about 11,000 students in Germany and Austria around the beginning of each semester. We analyzed three successive test pairs: PT36-PT41 (both conducted before the pandemic), PT37-PT42 (PT37 took place before the pandemic; PT42 was conducted from April 2020 onwards), and PT38-PT43 (PT38 was administered before the pandemic; PT43 started in November 2020). The authors used mixed-effect regression models and compared the absolute variations in the percentage of correct answers per subject. Results. The most recent test of each PT pair showed a higher mean score compared to the previous test in the same pair (PT36-PT41 : 2.53 (95% CI: 1.31-3.75), PT37-PT42 : 3.72 (2.57-4.88), and PT38-PT43 : 5.66 (4.63-6.69)). Analogously, an increase in the share of correct answers was observed for most medical disciplines, with Epidemiology showing the most remarkable upsurge. Conclusions. Overall, PT performance improved during the pandemic, which we take as an indication that the sudden shift to online learning did not have a negative effect on the knowledge gain of students. We consider that these results may be helpful in advancing innovative approaches to medical education

Prüfung von Baypamune® im Infektionsmodell der kaninen oralen Papillomatose am Hund

Canine Oral Papillomavirus (COPV) induces warts on the oral mucosa in domestic dogs and other canids. The canine oral papillomatosis (COP) is a well established animal model of mucosal papillomatosis. While the regression of a current infection is mediated by cellular immunity, humoral immunity does prevent reinfection. Question of the present study was, if unspecific stimulation of the immune system with the inducer of paramunity Baypamune® during the period of growth would have an influence on the course of canine oral papillomatosis. Clinical criteria for this purpose were period of growth, period of regression and the size and morphology of Papillomas at the end of growth. Additionally ALAT and ASAT were quantified in order to rule out deterioration of liver cells. PCV and leucocytes where monitored and a differentiation was performed. In a second part of the study, L1-antibodies and IFNg, TNF-a as well as IL-18 were determined by the Institute of Virology of the Faculty of Veterinary Medicine of the University of Leipzig. At the end of the study, biopsies of mucosa were taken for detection of viral genome. In a pre-study, three Labrador Retrievers at 14 weeks of age were challenged with 15 μl of virus suspension (40 μg of COPV-L1 Protein/ml) per site by scarification of oral mucosa. Papillomas developed at all challenged sites. These were removed during the period of growth and handed over to the Institute of Virology of the Faculty of Summary 70 Veterinary Medicine of the University of Leipzig for preparation of the suspension used in the main study. In the main study, 13 Labrador Retrievers at 14 weeks of age were challenged with 10 μl of virus suspension (40 μg of COPV-L1 Protein/ml) per site by scarification of three areas of the left oral mucosa. All animals developed at least at one site warts, totally 88 % of the challenged sites showed papillomas. 44 Days after infection, 12 dogs were divided into two groups which did not differ in time of incubation and size of the papillomas. Over a period of four weeks, one group received a total of five dosis of Baypamune®, the other group five dosis of placebo. There was no statistically significant difference between the groups regarding the period of growth, period of regression, size and morphology of the papillomas. The administration of Baypamune® during the period of growth had no effect on the clinical course of COP in this trial. However, time of application of the substances should be considered critically, since papillomas of five animals had been in regression before the first application of Baypamune® and placebo had taken place. No deterioration of liver cells could be ascertained. There was no difference between the groups regarding packed cell volume, white blood count and differentiation throughout the study. Parameters of unspecific immunity determined by the Institute of Virology, IFN-γ, TNF-a und IL-18, delivered no evaluable results. However, four animals of the placebogroup showed Papilloma-DNA in the Biopsies taken, but none of the animals, that received Baypamune®, being this a possible indicator for stimulation of cellular immunity. Anti-L1-Antibodies rose earlier in the Baypamune®group than in the placebogroup. In conclusion therapeutic efficacy of Baypamune® in dogs with present COP could not be shown in this trial, making future investigation necessary

Prüfung von Baypamune® im Infektionsmodell der kaninen oralen Papillomatose am Hund

Canine Oral Papillomavirus (COPV) induces warts on the oral mucosa in domestic dogs and other canids. The canine oral papillomatosis (COP) is a well established animal model of mucosal papillomatosis. While the regression of a current infection is mediated by cellular immunity, humoral immunity does prevent reinfection. Question of the present study was, if unspecific stimulation of the immune system with the inducer of paramunity Baypamune® during the period of growth would have an influence on the course of canine oral papillomatosis. Clinical criteria for this purpose were period of growth, period of regression and the size and morphology of Papillomas at the end of growth. Additionally ALAT and ASAT were quantified in order to rule out deterioration of liver cells. PCV and leucocytes where monitored and a differentiation was performed. In a second part of the study, L1-antibodies and IFNg, TNF-a as well as IL-18 were determined by the Institute of Virology of the Faculty of Veterinary Medicine of the University of Leipzig. At the end of the study, biopsies of mucosa were taken for detection of viral genome. In a pre-study, three Labrador Retrievers at 14 weeks of age were challenged with 15 μl of virus suspension (40 μg of COPV-L1 Protein/ml) per site by scarification of oral mucosa. Papillomas developed at all challenged sites. These were removed during the period of growth and handed over to the Institute of Virology of the Faculty of Summary 70 Veterinary Medicine of the University of Leipzig for preparation of the suspension used in the main study. In the main study, 13 Labrador Retrievers at 14 weeks of age were challenged with 10 μl of virus suspension (40 μg of COPV-L1 Protein/ml) per site by scarification of three areas of the left oral mucosa. All animals developed at least at one site warts, totally 88 % of the challenged sites showed papillomas. 44 Days after infection, 12 dogs were divided into two groups which did not differ in time of incubation and size of the papillomas. Over a period of four weeks, one group received a total of five dosis of Baypamune®, the other group five dosis of placebo. There was no statistically significant difference between the groups regarding the period of growth, period of regression, size and morphology of the papillomas. The administration of Baypamune® during the period of growth had no effect on the clinical course of COP in this trial. However, time of application of the substances should be considered critically, since papillomas of five animals had been in regression before the first application of Baypamune® and placebo had taken place. No deterioration of liver cells could be ascertained. There was no difference between the groups regarding packed cell volume, white blood count and differentiation throughout the study. Parameters of unspecific immunity determined by the Institute of Virology, IFN-γ, TNF-a und IL-18, delivered no evaluable results. However, four animals of the placebogroup showed Papilloma-DNA in the Biopsies taken, but none of the animals, that received Baypamune®, being this a possible indicator for stimulation of cellular immunity. Anti-L1-Antibodies rose earlier in the Baypamune®group than in the placebogroup. In conclusion therapeutic efficacy of Baypamune® in dogs with present COP could not be shown in this trial, making future investigation necessary

Digital examination formats - on the changes in the examination culture and examination practice in modular courses of studies

Die neue Diskussion um standardisierte IT-gestützte Prüfungen an Hochschulen verweist auf grundsätzliche Fragen des Zusammenhangs von Lehre und Prüfung. In der Logik des Bologna-Prozesses liegt die Orientierung der Prüfung an einem in Kompetenzen beschriebenen Absolventenprofil. Vergleichend wird die Entwicklung im medizinischen Bereich herangezogen, wo die zu prüfenden Wissensbereiche nicht weniger komplex als im erziehungswissenschaftlichen Bereich sind. Der Spezialfall standardisierter Großklausuren wird näher untersucht. (DIPF/Orig.)The recent debate on standardized IT-supported examinations at universities refers to fundamental questions concerning the relation between teaching and examining. It is within the logic of the Bologna Process that examinations are oriented by a graduate profile described through competences. The development within the medical field is used as a comparative example, - the areas of knowledge to be examined being hardly any less complex than those of the educational-scientific field. The special case of standardized comprehensive written examinations is investigated more closely. (DIPF/Orig.

Der PTM (Progress Test Medizin) zur Förderung von studentischer Selbstreflexion: Entwicklung einer Best Practice Feedback-Routine

Problemstellung/Lernziele/Motivation der WS-AutorInnen: Progress Testing ist international wie auch im deutschsprachigen Raum weit verbreitet. Als formatives Prüfungsformat soll Progress Testing v.a. Informationen bereitstellen, die den spontanen Wissensstand der Lernenden widerspiegeln. Das regelmäßige Reflektieren des eigenen, aktuellen Wissensstands ermöglicht eine Leistungssteigerung bei Prüfungen. Der Feedback-Charakter von Progress Testing als ein Hauptmerkmal dieser Prüfungsform hat motivierenden Einfluss auf die Lernsteuerung der Studierenden und einen positiven Einfluss auf die Weiterentwicklung der Lehre. Die Übersetzung individueller Progress Test Leistungen in individuelle Lernziele ist hierbei ein organisatorisches wie didaktisches Nadelöhr. In diesem Workshop kann sich die deutschsprachige Progress Test Gemeinschaft austauschen und gemeinsam diskutieren, wie der formative Nutzen des Progress Test ressourcenschonend optimiert werden kann. Ablauf Workshop: Einführung und Vorstellung Allgemeine Grundlagen des PTM Kriterien zur Förderung von studentischer Selbstreflexion Entwicklung einer Best Practice Feedback-Routine Anwendung von Feedback im eigenen Unterricht Abschlussrunde und Feedback Zielgruppe: Alle an Lehre und Prüfung Interessierten sind herzlich willkommen. Vorwissen und/oder Vorerfahrung mit Progress Testing ist von Vorteil. Vorbereitung/Mitzubringen: Keine Vorbereitung notwendig. Kenntnis davon, wie an der eigenen Einrichtung mit formativem Assessment umgegangen und den Lernenden Feedback gegeben wird, ist von Vorteil

ChatGPT in medical school: how successful is AI in progress testing?

Friederichs H, Friederichs WJ, März M. ChatGPT in medical school: how successful is AI in progress testing? Medical Education Online. 2023;28(1).Background As generative artificial intelligence (AI), ChatGPT provides easy access to a wide range of information, including factual knowledge in the field of medicine. Given that knowledge acquisition is a basic determinant of physicians’ performance, teaching and testing different levels of medical knowledge is a central task of medical schools. To measure the factual knowledge level of the ChatGPT responses, we compared the performance of ChatGPT with that of medical students in a progress test. Methods A total of 400 multiple-choice questions (MCQs) from the progress test in German-speaking countries were entered into ChatGPT’s user interface to obtain the percentage of correctly answered questions. We calculated the correlations of the correctness of ChatGPT responses with behavior in terms of response time, word count, and difficulty of a progress test question. Results Of the 395 responses evaluated, 65.5% of the progress test questions answered by ChatGPT were correct. On average, ChatGPT required 22.8 s (SD 17.5) for a complete response, containing 36.2 (SD 28.1) words. There was no correlation between the time used and word count with the accuracy of the ChatGPT response (correlation coefficient for time rho = −0.08, 95% CI [−0.18, 0.02], t(393) = −1.55, p = 0.121; for word count rho = −0.03, 95% CI [−0.13, 0.07], t(393) = −0.54, p = 0.592). There was a significant correlation between the difficulty index of the MCQs and the accuracy of the ChatGPT response (correlation coefficient for difficulty: rho = 0.16, 95% CI [0.06, 0.25], t(393) = 3.19, p = 0.002). Conclusion ChatGPT was able to correctly answer two-thirds of all MCQs at the German state licensing exam level in Progress Test Medicine and outperformed almost all medical students in years 1–3. The ChatGPT answers can be compared with the performance of medical students in the second half of their studies

Rechtliche Aspekte zu generativer Künstlicher Intelligenz und Large-Language-Modellen in Prüfungen und Abschlussarbeiten

The high performance of generative artificial intelligence (AI) and large language models (LLM) in examination contexts has triggered an intense debate about their applications, effects and risks. What legal aspects need to be considered when using LLM in teaching and assessment? What possibilities do language models offer? Statutes and laws are used to assess the use of LLM:- University statutes, state higher education laws, licensing regulations for doctors- Copyright Act (UrhG)- General Data Protection Regulation (DGPR)- AI Regulation (EU AI Act)LLM and AI offer opportunities but require clear university frameworks. These should define legitimate uses and areas where use is prohibited. Cheating and plagiarism violate good scientific practice and copyright laws. Cheating is difficult to detect. Plagiarism by AI is possible. Users of the products are responsible. LLM are effective tools for generating exam questions. Nevertheless, careful review is necessary as even apparently high-quality products may contain errors. However, the risk of copyright infringement with AI-generated exam questions is low, as copyright law allows up to 15% of protected works to be used for teaching and exams. The grading of exam content is subject to higher education laws and regulations and the GDPR. Exclusively computer-based assessment without human review is not permitted. For high-risk applications in education, the EU's AI Regulation will apply in the future. When dealing with LLM in assessments, evaluation criteria for existing assessments can be adapted, as can assessment programmes, e.g. to reduce the motivation to cheat. LLM can also become the subject of the examination themselves. Teachers should undergo further training in AI and consider LLM as an addition.Die hohe Leistungsfähigkeit von generativer Künstlicher Intelligenz (KI) und großen Sprachmodellen (LLM) in Prüfungskontexten hat eine intensive Debatte über ihre Anwendungen, Auswirkungen und Risiken ausgelöst. Welche rechtlichen Aspekte sind beim Einsatz von LLM in Lehre und Prüfungen zu berücksichtigen? Welche Chancen bieten Sprachmodelle? Für die rechtliche Bewertung des Einsatzes von LLM finden Satzungen und Gesetze Anwendung:- Universitäre Satzungen, Hochschulgesetze der Länder, Approbationsordnung für Ärzte- Urheberrechtsgesetz (UrhG)- Datenschutz-Grundverordnung (DS-GVO)- KI-Verordnung (KI-VO)LLM und KI bieten Chancen, erfordern aber klare universitäre Rahmenbedingungen. Diese sollten den legitimen Einsatz und die Bereiche, in denen die Nutzung untersagt ist, definieren. Täuschungen und Plagiate verstoßen gegen die wissenschaftliche Praxis und das UrhG. Eine Täuschung ist schwer nachzuweisen. Plagiate durch KI sind möglich. Nutzer*innen der Produkte sind in der Verantwortung.LLM sind effektive Tools zur Generierung von Prüfungsfragen. Dennoch ist ein sorgfältiges Review notwendig, da selbst qualitativ hochwertig scheinende Produkte Fehler enthalten können. Das Risiko von Urheberrechtsverletzungen bei KI-generierten Prüfungsaufgaben ist hingegen gering, da das Urheberrecht den Einsatz geschützter Werke für Lehre und Prüfungen bis zu 15% erlaubt.Die Bewertung von Prüfungsinhalten unterliegt Hochschulgesetzen und -ordnungen und der DSGVO. Eine ausschließlich computergestützte Bewertung ohne menschliche Überprüfung ist nicht zulässig. Für Hochrisiko-Anwendungen in der beruflichen Lehre findet künftig die KI-VO der EU Anwendung.Im Umgang mit LLM in Prüfungen können Bewertungskriterien bestehender Prüfungen angepasst werden, aber auch Prüfungsprogramme, z.B. um die Täuschungsmotivation zu reduzieren. LLM können zudem selbst Gegenstand der Prüfung werden. Lehrende sollten sich in KI weiterbilden und LLM als Ergänzung betrachten