Expertise in performance assessment: assessors' perspectives

The recent rise of interest among the medical education community in individual faculty making subjective judgments about medical trainee performance appears to be directly related to the introduction of notions of integrated competency-based education and assessment for learning. Although it is known that assessor expertise plays an important role in performance assessment, the roles played by different factors remain to be unraveled. We therefore conducted an exploratory study with the aim of building a preliminary model to gain a better understanding of assessor expertise. Using a grounded theory approach, we conducted seventeen semi-structured interviews with individual faculty members who differed in professional background and assessment experience. The interviews focused on participants' perceptions of how they arrived at judgments about student performance. The analysis resulted in three categories and three recurring themes within these categories: the categories assessor characteristics, assessors' perceptions of the assessment tasks, and the assessment context, and the themes perceived challenges, coping strategies, and personal development. Central to understanding the key processes in performance assessment appear to be the dynamic interrelatedness of the different factors and the developmental nature of the processes. The results are supported by literature from the field of expertise development and in line with findings from social cognition research. The conceptual framework has implications for faculty development and the design of programs of assessmen

Refreshing medical students' intravenous-cannulation skills: a blinded observer three-arm randomised comparison of mental imagery, part-task trainer simulation and written instructions.

INTRODUCTION Intravenous cannulation is a core competence in medicine, but is considered challenging to learn. This study investigates the effectiveness of three educational strategies used to refresh the intravenous cannulation skills of first-year medical students: mental imagery, part-task trainer simulation and written instructions. MATERIALS AND METHODS In this single-centre randomised controlled trial, first-year medical students were assigned to one of three different refresher tutorials on intravenous cannulation. Six months after their compulsory 4 hour instructor-led intravenous-cannulation course, each student was randomised to a 6 min self-learning tutorial: a mental imagery audioguide session, hands-on intravenous cannulation on a part-task trainer or reading written instructions.Immediately after the refresher tutorials, trained evaluators who were blinded to the randomised group assessed the students' performance. Each evaluator completed a 15-item standardised checklist in an Objective Structured Clinical Examination (OSCE) station for intravenous cannulation. We performed a descriptive analysis of the data and a one-way analysis of variance. Additionally, we investigated the influence of previous intravenous cannulation experience on the total OSCE score. RESULTS AND DISCUSSION On analysing the 309 students' results, we did not find differences in the total rating of the performance (in percentage) between the three groups at the OSCE station (mental imagery group: 72.0%±17.9%; part-task trainer group: 74.4%±15.6%; written instructions group: 69.9%±16.6%, p=0.158). Multiple linear regression showed a small but statistically significant effect of students' previous intravenous cannulation experience on OSCE performance. With the same outcome, written instructions and mental imagery had a better return on effort, compared with resource-intensive hands-on training with part-task trainers. CONCLUSION A single, short refresher seems to have a limited effect on intravenous-cannulation skills in first-year medical students. Less resource-intensive interventions, such as written instructions or mental imagery, are effective compared with hands-on part-task trainer simulation for refreshing this simple but important skill. TRIAL REGISTRATION NUMBER AEARCTR-0008043

From threat to challenge-Improving medical students' stress response and communication skills performance through the combination of stress arousal reappraisal and preparatory worked example-based learning when breaking bad news to simulated patients: study protocol for a randomized controlled trial.

BACKGROUND Breaking bad news (BBN; e.g., delivering a cancer diagnosis) is perceived as one of the most demanding communication tasks in the medical field and associated with high levels of stress. Physicians' increased stress in BBN encounters can negatively impact their communication performance, and in the long term, patient-related health outcomes. Although a growing body of literature acknowledges the stressful nature of BBN, little has been done to address this issue. Therefore, there is a need for appropriate tools to help physicians cope with their stress response, so that they can perform BBN at their best. In the present study, we implement the biopsychosocial model of challenge and threat as theoretical framework. According to this model, the balance between perceived situational demands and perceived coping resources determines whether a stressful performance situation, such as BBN, is experienced as challenge (resources > demands) or threat (resources < demands). Using two interventions, we aim to support medical students in shifting towards challenge-oriented stress responses and improved communication performance: (1) stress arousal reappraisal (SAR), which guides individuals to reinterpret their stress arousal as an adaptive and beneficial response for task performance; (2) worked examples (WE), which demonstrate how to BBN in a step-by-step manner, offering structure and promoting skill acquisition. METHODS In a randomized controlled trial with a 2 (SAR vs. control) x 2 (WE vs. control) between-subjects design, we will determine the effects of both interventions on stress response and BBN skills performance in N = 200 third-year medical students during a simulated BBN encounter. To identify students' stress responses, we will assess their perceived coping resources and task demands, record their cardiovascular activity, and measure salivary parameters before, during, and after BBN encounters. Three trained raters will independently score students' BBN skills performances. DISCUSSION Findings will provide unique insights into the psychophysiology of medical students who are tasked with BBN. Parameters can be understood more comprehensively from the challenge and threat perspective and linked to performance outcomes. If proven effective, the evaluated interventions could be incorporated into the curriculum of medical students and facilitate BBN skills acquisition. TRIAL REGISTRATION ClinicalTrials.gov (NCT05037318), September 8, 2021

Physicians’ attitudes toward, use of, and perceived barriers to clinical guidelines: a survey among Swiss physicians

BACKGROUND: Little is known about the attitudes toward, use of, and perceived barriers to clinical guidelines in Switzerland, a country with no national guideline agency. Moreover, there is no available data on the objective assessment of guideline knowledge in Switzerland. Therefore, we conducted a study at a large university's Department of General Internal Medicine in Switzerland to assess physicians' attitudes toward, use of, perceived barriers to, and knowledge of clinical guidelines. PARTICIPANTS AND METHODS: Ninety-six physicians (residents, n=78, and attendings, n=18) were invited to take part in a survey. Attitudes toward, self-reported use of, and barriers hindering adherence to the clinical guidelines were assessed using established scales and frameworks. Knowledge of the guidelines was objectively tested in a written assessment comprising of 14 multiple-choice and 3 short answer case-based questions. RESULTS: Fifty-five participants completed the survey (residents, n=42, and attendings, n=13; overall response rate 57%). Of these, 50 took part in the knowledge assessment (residents, n=37, and attendings, n=13; overall response rate 52%). Attitudes toward guidelines were favorable. They were considered to be a convenient source of advice (94% agreement), good educational tools (89% agreement), and likely to improve patient quality of care (91% agreement). Self-reported use of guidelines was limited, with only one-third reporting using guidelines often or very often. The main barriers to guideline adherence were identified as lack of guideline awareness and familiarity, applicability of existing guidelines to multimorbid patients, unfavorable guideline factors, and lack of time as well as inertia toward changing previous practice. In the assessment of guideline knowledge, the scores were rather modest (mean ± standard deviation: 60.5%±12.7% correct answers). CONCLUSION: In general, this study found favorable physician attitudes toward clinical guidelines. However, several barriers hindering guideline implementation were identified. The importance of improving guideline implementation was supported by modest results in a guideline knowledge test

Swiss general internal medicine board examination: quantitative effects of publicly available and unavailable questions on question difficulty and test performance.

BACKGROUND Formerly, a substantial number of the 120 multiple-choice questions of the Swiss Society of General Internal Medicine (SSGIM) board examination were derived from publicly available MKSAP questions (Medical Knowledge Self-Assessment Program®). The possibility to memorise publicly available questions may unduly influence the candidates' examination performance. Therefore, the examination board raised concerns that the examination did not meet the objective of evaluating the application of knowledge. The society decided to develop new, "Helvetic" questions to improve the examination. The aim of the present study was to quantitatively assess the degree of difficulty of the Helvetic questions (HQ) compared with publicly available and unavailable MKSAP questions and to investigate whether the degree of difficulty of MKSAP questions changed over time as their status changed from publicly available to unavailable. METHODS The November 2019 examination consisted of 40 Helvetic questions, 40 publicly available questions from MKSAP edition 17 (MKSAP-17) and 40 questions from MKSAP-15/16, which were no longer publicly available at the time of the examination. An one factorial univariate analysis of variance (ANOVA) examined question difficulty (lower values mean higher difficulty) between these three question sets. A repeated ANOVA compared the difficulty of MKSAP-15/16 questions in the November 2019 examination with the difficulty of the exact same questions from former examinations, when these questions belonged to the publicly available MKSAP edition. The publicly available MKSAP-17 and the publicly unavailable Helvetic questions served as control. RESULTS The analysis of the November 2019 exam showed a significant difference in average item difficulty between Helvetic and MKSAP-17 questions (71% vs 86%, p <0.001) and between MKSAP-15/16 and MKSAP-17 questions (70% vs 86%, p <0.001). There was no significant difference in item difficulty between Helvetic and MKSAP-15/16 questions (71% vs 70%, p = 0.993). The repeated measures ANOVA on question use and the three question categories showed a significant interaction (p <0.001, partial eta-squared = 0.422). The change in the availability of MKSAP-15/16 questions had a strong effect on difficulty. Questions became on average 21.9% more difficult when they were no longer publicly available. In contrast, the difficulty of the MKSAP-17 and Helvetic questions did not change significantly across administrations. DISCUSSION This study provides the quantitative evidence that the public availability of questions has a decisive influence on question difficulty and thus on SSGIM board examination performance. Reducing the number of publicly available questions in the examination by introducing confidential, high-quality Helvetic questions contributes to the validity of the board examination by addressing higher order cognitive skills and making rote-learning strategies less effective

Identity matters - perceptions of inter-professional feedback in the context of workplace-based assessment in Diabetology training: a qualitative study

Background Inter-professional collaboration is acknowledged as essential for quality patient-care. However, little is known about receptiveness to inter-professional feedback in the postgraduate training. This study explores, in light of social identity theory, the perceptions of residents, supervising physicians and allied health care professionals regarding inter-professional feedback in the context of workplace-based assessment. Methods For 6 months, residents in Diabetology at the University Hospital of Bern performed formative workplace-based assessments under direct observation of a supervising physician and an allied health care professional. Feedback from both observers was given to the resident after every assessment. Subsequently, focus group discussions were conducted to collect the participants’ perceptions of inter- and intra-professional feedback. Transcripts were analyzed qualitatively using a thematic analysis approach. Results We identified four main themes: (1) Identity and hierarchy; (2) Interdependence of feedback source and feedback content; (3) Impact on collaboration and patient-care; (4) Logistical and organizational requirements. While different social identities are the source of inter-professional hierarchies, they did not impede the receptiveness to feedback. Perceived trustworthiness of the feedback was attributed with more importance than professional affiliations, whereas intra-professional hierarchies between physicians led to the perception of a more summative nature of the feedback and rather impeded receptiveness. According to the participants, inter-professional feedback raised awareness of the working reality of other team members and had a positive impact on communication between the different professional groups. Moreover, participants reported positive response from patients regarding the inter-professional collaboration they experienced. Considerable organizational effort is required to enable the parallel observation of a resident’s consultation by a supervising physician and an allied health care professional. Conclusions Feedback from allied health care professionals can be a valuable learning resource for residents, given its role outside the sometimes conflicting area of intra-professional hierarchies. Inter-professional feedback in the context of workplace-based assessment carries the potential to strengthen collaboration between the different professional groups

Training von Clinical Reasoning mit einfachen Lehrmethoden im klinischen Alltag

Im klinischen Alltag lässt sich oft beobachten, dass Assistenzärztinnen und -ärzte (AÄ) in einer ausführenden Rolle verhaftet bleiben und selbst eher wenig zum Diagnose- und Managementprozess in der Patientenbetreuung beitragen [1, 2]