Aldeen, A.

Branzetti, J.

Hartman, N.

Quattromani, E.

Wheaton, N.

Williamson, K.

eScholarship - University of California

UC IrvineWestern Journal of Emergency Medicine: Integrating Emergency Care with Population HealthTitleFaculty Prediction of In-training Examination Scores of Emergency Medicine Residents: A Multi-center StudyPermalinkhttps://escholarship.org/uc/item/9vp8969jJournalWestern Journal of Emergency Medicine: Integrating Emergency Care with Population Health, 16(4.1)ISSN1936-900XAuthorsWilliamson, K.Aldeen, A.Quattromani, E.et al.Publication Date2015Copyright InformationCopyright 2015 by the author(s).This work is made available under the terms of a Creative Commons Attribution License, available at https://creativecommons.org/licenses/by/4.0/eScholarship.org Powered by the California Digital LibraryUniversity of CaliforniaWestern Journal of Emergency Medicine S12 Volume XVI, Supplement : July 2015CORD Abstracts Issue 2015 was not seen with the paper form. Storage of complete data allows students and preceptors to reflect on the contents of the MSF sessions at a later date.23Enhancing Resident Engagement and Knowledge Retention through Curricular ModificationsBavolek R, Wagner J / Washington University in St. Louis School of Medicine, St. Louis, MO Background: Emergency medicine is a specialty filled with individuals ill suited to sit in a lecture hall for long periods. Despite this, 5 subsequent hours of 50 minute blocks are commonly used to meet Residency Review Committee conference requirements. These prolonged sessions test the patience of the audience and impact knowledge retention.Educational Objectives: At Washington University School of Medicine, we’ve implemented changes to combat boredom while enhancing retention.Curricular Design: We addressed the issue of fatigue and boredom when sitting through 5 hours of lecture by dividing lectures into 25 minute slots rather than the traditional 50 minutes. This rapid-fire lecture style minimizes lapses in attention, which studies have shown occur roughly every 10-15 minutes. One challenge has been preventing lecturers from delivering 50 minutes of content in only 25. Additionally, the audience must get adequate breaks, as most people cannot maintain continuous attention for more than about 45 minutes. We addressed this obstacle by using a timer that runs continuously during conference, giving the speaker a 5 minute warning followed by an alert that their lecture time is over. This dramatically improves the ease in which a long-winded lecturer can be curtailed. Finally, we implemented spaced-repetition. This is a well-proven method of enhancing knowledge retention. The key points are repeated to the residents 3 times. First during the lecture itself, second as a rapid-fire summary of the entire day of content at the end of conference, and a third time as a faculty run review the following week.Impact: As this is a cutting-edge curricular change, we have little evidence to its effectiveness. While we don’t have internal evidence, there is a large body of educational literature to support these initiatives. Conference feedback and evaluations improved dramatically over the past six months. Our hope is that inservice scores will reflect the benefit of our new curricular design.24Exploratory Factor Analysis of Patient Ability to Differentiate Individual Core Competencies During Evaluation of Resident Clinical PerformanceBackground: Patient evaluation of resident performance has been included as part of 360 degree evaluations by the Residency Review Committee of Emergency Medicine. Despite their use in most residency programs, little research has been done to evaluate the metrics of patient evaluations.Objectives: We sought to determine the ability of emergency department (ED) patients to differentiate individual core competencies when asked to evaluate resident clinical performance.Methods: This prospective observational study was conducted at an urban ED with a postgraduate year 1- 3 emergency medicine residency program comprised of 30 residents. Each resident was evaluated by approximately 10 patients over a 2 month period on a competency-based evaluation questionnaire. The questionnaire was administered to patients by a trained research assistant and resident performance on 8 competency based items was rated on a fixed 9 point scale. Surveyed patients were selected randomly by the research assistant during clinical shifts from the patient log without resident knowledge. Pearson correlation coefficients across each resident’s score for the competency based questions were analyzed in a correlation matrix.Results: During the 2 month period of the study 286 patients evaluated 29 residents yielding an average of 9.8 evaluations per resident. To determine whether patients were able to separate and rank residents on the individual competencies we compared Pearson correlation coefficients across each resident’s score for the competency based questions. The resulting correlation matrix yielded 28 combinations. The patient rankings for all of these scores were highly correlated. The correlations ranged from 0.78 to 0.97 and all were significant at p<0.001.Conclusion: When patients evaluate resident performance using a competency based form, the results obtained across multiple competency based questions are highly correlated. Patients do not discriminate well between individual competency based constructs when performing clinical evaluations of residents.25Faculty Prediction of In-training Examination Scores of Emergency Medicine Residents: A Multi-center StudyRyan J, Barlas D / New York Hospital Queens, Flushing, NY Williamson K, Aldeen A, Quattromani E, Hartman N, Wheaton N, Branzetti J / Advocate Christ Medical Center, Oak Lawn, IL; Presence St. Joseph’s Medical Center, Joliet, IL; St. Louis University, St. Louis, MO; Wake Forest University, Winston Salem, NC; University of Iowa, Iowa City, IA; University of Washington, Seattle, WA Background: The Emergency Medicine In-Training Examination (EMITE) is one of the few valid tools for medical knowledge assessment in use by emergency Volume XVI, Supplement : July 2015 S13 Western Journal of Emergency Medicine CORD Abstracts Issue 2015medicine (EM) residencies. Review of EMITE performance helps programs identify resident weaknesses in core content knowledge, but its use as a formative assessment tool is limited by infrequent administration. If EM faculty could accurately predict residents’ EMITE scores, then residents with medical knowledge deficiencies could be identified earlier, providing time to institute remediation.Objectives: To conduct a multicenter trial to evaluate the ability of EM faculty to predict EMITE scores.Methods: This was a prospective, multicenter trial involving five EM residencies. Institutional review board approval was obtained by all sites. EM faculty were asked to predict 2014 EMITE scores of their residents using an online survey instrument. The primary outcome was prediction accuracy (the proportion of predictions within 6% of the actual score). The secondary outcome was prediction precision (the mean deviation of predictions from the actual scores). We also assessed faculty background variables, including years of experience, educational leadership status, and clinical hours worked.Results: 111 faculty physicians participated, rendering 3,219 predictions for 147 residents. The mean prediction accuracy was 60% (95% CI:[57.5-62.6%]) and the mean prediction precision was 6.3% (95% CI:[6.0-6.6%]). Prediction accuracy was not significantly different between educational leaders (63.9%, 95% CI:[60.4-67.4]) and non-educational leaders (58.4%, 95% CI:[44.2-61.6]) and there was no correlation with other faculty background variables. Only eight participants predicted scores with high accuracy (>80%).Conclusion: In this multicenter study, EM faculty possess only moderate accuracy at predicting resident EMITE scores. This finding calls into question the ability of faculty to accurately assess a standardized marker of resident medical knowledge.26Feasability of Improving Bedside Teaching through Targeted Simulation-Based Education for FacultyElahi N, Aghera A, Marshall J, Gillett B / Maimonides Medical Center, Brooklyn, NY Background: Long lasting learning is maximized when educational activities are paired with matched clinical bedside teaching. Conversely, lack of bedside teaching for a given topic likely impairs any educational initiative to close knowledge gaps. Managing ventilators in the emergency department is an example of a topic with potential asynchrony of formal education and practical bedside teaching.Objectives: To determine if targeted simulation-based faculty education could enhance bedside teaching in the domain of mechanical ventilation.Methods: First, a needs analysis was performed at an urban community academic hospital, asking emergency medicine residents to rate the frequency of bedside teaching when caring for patients who require mechanical ventilation on a rating scale of 1 to 4 (1=never, 4=always). A prospective cohort study was then done on 27 out of 44 faculty members who participated in a one-hour advanced simulation-based mechanical ventilation course. Faculty self-rated their pre- and post-course competency of ventilator management on a novice to expert Dreyfus scale from 1 to 5. They also rated their current frequency of bedside teaching regarding ventilator management and their anticipated frequency of teaching after completing the course on a rating scale from 1 to 4 (1=never, 4=always).Results: 33 of 48 residents responded to the needs analysis survey, with average and median ratings of 1.52 and 1, respectively. Before and after the course, average faculty self-ratings of competency on the Dreyfus Scale improved from 2.7 to 3.6 (p<0.001), with median ratings improving from 3 to 4. Average ratings of current and anticipated frequency of bedside teaching improved from 2.3 to 3.1 (p<0.001), with median ratings improving from 2 to 3.Conclusion: Emergency medicine residents report a low frequency of bedside teaching related to mechanical ventilation. Targeted simulation-based education for faculty has the potential to significantly improve the frequency of bedside teaching of this topic.27Get Feedback Now: How to Best Use Your Residency Management Software to Increase the Response Rate and Quality of Conference EvaluationsShin N, Simcoe R, Sule H, Massone R, Hall R /Thomas Jefferson University, Philadelphia, PA; Rutgers, University of Medicine and Dentistry (UMDNJ)-New Jersey Medical School, Newark, NJ Introduction: Feedback is a vital ingredient for successful post-graduate medical education. It is required by the Accreditation Council for Graduate Medical Education for assessment and improvement of key aspects of the residency program, one of which is the mandatory weekly conferences. Audience feedback allows for presenters and residency leadership to make adjustments to future content to better meet the needs of the residents. It is most useful when the collective feedback is numerous, timely, and organized. Educational Objectives: Optimize collection of feedback survey forms using a residency management software, handheld technology (tablet/smartphone), internet access, and protected time following presentations. Curricular Design: A prospective cohort study of emergency medicine residents and teaching faculty at an academic hospital was conducted. Evaluations of weekly residency conferences were collected from 8/7/14-11/20/14 using the “Conference Survey” function within the New Innovations? 

Faculty Prediction of In-training Examination Scores of Emergency Medicine Residents: A Multi-center Study

Abstract

Similar works

Full text

Available Versions

eScholarship - University of California