Medical specialty certification exams studied according to the Ottawa Quality Criteria: a systematic review.

BACKGROUND Medical specialty certification exams are high-stakes summative assessments used to determine which doctors have the necessary skills, knowledge, and attitudes to treat patients independently. Such exams are crucial for patient safety, candidates' career progression and accountability to the public, yet vary significantly among medical specialties and countries. It is therefore of paramount importance that the quality of specialty certification exams is studied in the scientific literature. METHODS In this systematic literature review we used the PICOS framework and searched for papers concerning medical specialty certification exams published in English between 2000 and 2020 in seven databases using a diverse set of search term variations. Papers were screened by two researchers independently and scored regarding their methodological quality and relevance to this review. Finally, they were categorized by country, medical specialty and the following seven Ottawa Criteria of good assessment: validity, reliability, equivalence, feasibility, acceptability, catalytic and educational effect. RESULTS After removal of duplicates, 2852 papers were screened for inclusion, of which 66 met all relevant criteria. Over 43 different exams and more than 28 different specialties from 18 jurisdictions were studied. Around 77% of all eligible papers were based in English-speaking countries, with 55% of publications centered on just the UK and USA. General Practice was the most frequently studied specialty among certification exams with the UK General Practice exam having been particularly broadly analyzed. Papers received an average of 4.2/6 points on the quality score. Eligible studies analyzed 2.1/7 Ottawa Criteria on average, with the most frequently studied criteria being reliability, validity, and acceptability. CONCLUSIONS The present systematic review shows a growing number of studies analyzing medical specialty certification exams over time, encompassing a wider range of medical specialties, countries, and Ottawa Criteria. Due to their reliance on multiple assessment methods and data-points, aspects of programmatic assessment suggest a promising way forward in the development of medical specialty certification exams which fulfill all seven Ottawa Criteria. Further research is needed to confirm these results, particularly analyses of examinations held outside the Anglosphere as well as studies analyzing entire certification exams or comparing multiple examination methods

Team leadership assessment after advanced life support courses comparing real teams vs. simulated teams.

AIM Effective team leadership is essential during cardiopulmonary resuscitation (CPR) and is taught during international advanced life support (ALS) courses. This study compared the judgement of team leadership during summative assessments after those courses using different validated assessment tools while comparing two different summative assessment methods. METHODS After ALS courses, twenty videos of simulated team assessments and 20 videos of real team assessments were evaluated and compared. Simulated team assessment used an instructor miming a whole team, whereas real team assessment used course participants as a team that acted on the team leader's commands. Three examiners individually evaluated each video on four different validated team leadership assessment tools and on the original European Resuscitation Council's (ERC) scenario test assessment form which does not assess leadership. The primary outcome was the average performance summary score between all three examiners for each assessment method. RESULTS The average performance summary score for each of the four assessment tools was significantly higher for real team assessments compared to simulated team assessments (all p-values < 0.01). The summary score of the ERC's scenario test assessment form was comparable between both assessment methods (p = 0.569), meaning that participants of both assessments performed equally. CONCLUSION Team leadership performance is rated significantly higher in real team summative assessments after ALS courses compared to simulated team assessments by four leadership assessment tools but not by the standard ERC's scenario test assessment form. These results suggest that summative assessments in ALS courses should integrate real team assessments, and a new assessment tool including an assessment of leadership skills needs to be developed

Assessment of Middle Ear Anatomy Teaching Methodologies Using Microscopy versus Endoscopy: A Randomized Comparative Study.

Teaching methodologies for the anatomy of the middle ear have not been investigated greatly due to the middle ear's highly complex structure and hidden location inside of the temporal bone. The aim of this randomized study was to quantitatively compare the suitability of using microscope- and endoscope-based methods for teaching the anatomy of the middle ear. We hypothesize that the endoscopic approach will be more efficient compared to the microscopic approach. To answer the study questions, 33 sixth-year medical students, residents and otorhinolaryngology specialists were randomized either into the endoscopy or the microscopy group. Their anatomical knowledge was assessed using a structured anatomical knowledge test before and after each session. Each participant received tutoring on a human cadaveric specimen using one of the two methods. They then performed a hands-on dissection. After 2-4 weeks, the same educational curriculum was repeated using the other technique. The mean gains in anatomical knowledge for the specialists, residents, and medical students were +19.0%, +34.6%, and +23.4%, respectively. Multivariate analyses identified a statistically significant increase in performance for the endoscopic method compared to the microscopic technique (P < 0.001). For the recall of anatomical structures during dissection, the endoscopic method outperformed the microscopic technique independently of the randomization or the prior training level of the attendees (P < 0.001). In conclusion, the endoscopic approach to middle ear anatomy education is associated to an improved gain in knowledge as compared to the microscopic approach. The participants subjectively preferred the endoscope for educational purposes

Trainee doctors' preparedness for clinical work in geriatric psychiatry: A survey on 18 preliminary entrustable professional activities.

BACKGROUND Research concerning transitions from one rotation to another during medical specialist training is scarce. This study examined trainee doctors' perceived preparedness for core clinical activities, trainee doctors' preparedness levels, and general perceptions of medical specialist training in geriatric psychiatry. METHOD Swiss trainee doctors in geriatric psychiatry were surveyed about their perceived preparedness for 18 preliminary entrustable professional activities (EPAs), curricular support, and general perceptions of their medical specialist training. Closed questions were analysed using descriptive statistics, while open questions were subjected to content analysis. RESULTS The participants comprised 48 trainee doctors (30.4% response rate) who differed in their educational experience (years of residency and specialism) and clinical subspecialisation goals. Trainee doctors felt adequately prepared for most EPAs but less prepared for some, including electroconvulsive therapy, psychotherapy, and treating older adults in the home environment or residential facilities. Despite the trainee doctors' diversity, they did not differ significantly in perceived preparedness for most EPAs. The most often offered suggestions for improving geriatric psychiatry training were intensified clinical supervision and a structured induction programme. CONCLUSION Trainee doctors reported that they felt sufficiently prepared for most EPAs, regardless of their backgrounds and professional goals. However, several professional activities in geriatric psychiatry warrant further training. Our findings indicate the need for a higher intensity of clinical supervision (e.g. more direct observation and specific feedback), the introduction of structured induction programmes (e.g. orientation week), and specific teachings (e.g. on neurocognitive assessment)

Cost-effectiveness of peer role play and standardized patients in undergraduate communication training

Background: The few studies directly comparing the methodological approach of peer role play (RP) and standardized patients (SP) for the delivery of communication skills all suggest that both methods are effective. In this study we calculated the costs of both methods (given comparable outcomes) and are the first to generate a differential cost-effectiveness analysis of both methods. Methods: Medical students in their prefinal year were randomly assigned to one of two groups receiving communication training in Pediatrics either with RP (N = 34) or 19 individually trained SP (N = 35). In an OSCE with standardized patients using the Calgary-Cambridge Referenced Observation Guide both groups achieved comparable high scores (results published). In this study, corresponding costs were assessed as man-hours resulting from hours of work of SP and tutors. A cost-effectiveness analysis was performed. Results: Cost-effectiveness analysis revealed a major advantage for RP as compared to SP (112 vs. 172 man hours; cost effectiveness ratio .74 vs. .45) at comparable performance levels after training with both methods. Conclusions: While both peer role play and training with standardized patients have their value in medical curricula, RP has a major advantage in terms of cost-effectiveness. This could be taken into account in future decisions