Features and uses of high-fidelity medical simulations that lead to effective learning: A BEME systematic review.

Review date: 1969 to 2003, 34 years. Background and context: Simulations are now in widespread use in medical education and medical personnel evaluation. Outcomes research on the use and effectiveness of simulation technology in medical education is scattered, inconsistent and varies widely in methodological rigor and substantive focus. Objectives: Review and synthesize existing evidence in educational science that addresses the question, 'What are the features and uses of high-fidelity medical simulations that lead to most effective learning?'. Search strategy: The search covered five literature databases (ERIC, MEDLINE, PsycINFO, Web of Science and Timelit) and employed 91 single search terms and concepts and their Boolean combinations. Hand searching, Internet searches and attention to the 'grey literature' were also used. The aim was to perform the most thorough literature search possible of peer-reviewed publications and reports in the unpublished literature that have been judged for academic quality. Inclusion and exclusion criteria: Four screening criteria were used to reduce the initial pool of 670 journal articles to a focused set of 109 studies: (a) elimination of review articles in favor of empirical studies; (b) use of a simulator as an educational assessment or intervention with learner outcomes measured quantitatively; (c) comparative research, either experimental or quasi-experimental; and (d) research that involves simulation as an educational intervention. Data extraction: Data were extracted systematically from the 109 eligible journal articles by independent coders. Each coder used a standardized data extraction protocol. Data synthesis: Qualitative data synthesis and tabular presentation of research methods and outcomes were used. Heterogeneity of research designs, educational interventions, outcome measures and timeframe precluded data synthesis using meta-analysis. Headline results: Coding accuracy for features of the journal articles is high. The extant quality of the published research is generally weak. The weight of the best available evidence suggests that high-fidelity medical simulations facilitate learning under the right conditions. These include the following: providing feedback-51 (47%) journal articles reported that educational feedback is the most important feature of simulation-based medical education; repetitive practice-43 (39%) journal articles identified repetitive practice as a key feature involving the use of high-fidelity simulations in medical education; curriculum integration-27 (25%) journal articles cited integration of simulation-based exercises into the standard medical school or postgraduate educational curriculum as an essential feature of their effective use; range of difficulty level-15 (14%) journal articles address the importance of the range of task difficulty level as an important variable in simulation-based medical education; multiple learning strategies-11 (10%) journal articles identified the adaptability of high-fidelity simulations to multiple learning strategies as an important factor in their educational effectiveness; capture clinical variation-11 (10%) journal articles cited simulators that capture a wide variety of clinical conditions as more useful than those with a narrow range; controlled environment-10 (9%) journal articles emphasized the importance of using high-fidelity simulations in a controlled environment where learners can make, detect and correct errors without adverse consequences; individualized learning-10 (9%) journal articles highlighted the importance of having reproducible, standardized educational experiences where learners are active participants, not passive bystanders; defined outcomes-seven (6%) journal articles cited the importance of having clearly stated goals with tangible outcome measures that will more likely lead to learners mastering skills; simulator validity-four (3%) journal articles provided evidence for the direct correlation of simulation validity with effective learning. Conclusions: While research in this field needs improvement in terms of rigor and quality, high-fidelity medical simulations are educationally effective and simulation-based education complements medical education in patient care settings

A report on the piloting of a novel computer-based medical case simulation for teaching and formative assessment of diagnostic laboratory testing

Objectives: Insufficient attention has been given to how information from computer-based clinical case simulations is presented, collected, and scored. Research is needed on how best to design such simulations to acquire valid performance assessment data that can act as useful feedback for educational applications. This report describes a study of a new simulation format with design features aimed at improving both its formative assessment feedback and educational function. Methods: Case simulation software (LabCAPS) was developed to target a highly focused and well-defined measurement goal with a response format that allowed objective scoring. Data from an eight-case computer-based performance assessment administered in a pilot study to 13 second-year medical students was analyzed using classical test theory and generalizability analysis. In addition, a similar analysis was conducted on an administration in a less controlled setting, but to a much large sample (n=143), within a clinical course that utilized two random case subsets from a library of 18 cases. Results: Classical test theory case-level item analysis of the pilot assessment yielded an average case discrimination of 0.37, and all eight cases were positively discriminating (range=0.11–0.56). Classical test theory coefficient alpha and the decision study showed the eight-case performance assessment to have an observed reliability of σ=G=0.70. The decision study further demonstrated that a G=0.80 could be attained with approximately 3 h and 15 min of testing. The less-controlled educational application within a large medical class produced a somewhat lower reliability for eight cases (G=0.53). Students gave high ratings to the logic of the simulation interface, its educational value, and to the fidelity of the tasks. Conclusions: LabCAPS software shows the potential to provide formative assessment of medical students’ skill at diagnostic test ordering and to provide valid feedback to learners. The perceived fidelity of the performance tasks and the statistical reliability findings support the validity of using the automated scores for formative assessment and learning. LabCAPS cases appear well designed for use as a scored assignment, for stimulating discussions in small group educational settings, for self-assessment, and for independent learning. Extension of the more highly controlled pilot assessment study with a larger sample will be needed to confirm its reliability in other assessment applications

Development of the Uncertainty Communication Checklist: A Patient-Centered Approach to Patient Discharge From the Emergency Department

Clear communication with patients upon emergency department (ED) discharge is important for patient safety during the transition to outpatient care. Over one-third of patients are discharged from the ED with diagnostic uncertainty, yet there is no established approach for effective discharge communication in this scenario. From 2017 to 2019, the authors developed the Uncertainty Communication Checklist for use in simulation-based training and assessment of emergency physician communication skills when discharging patients with diagnostic uncertainty. This development process followed the established 12-step Checklist Development Checklist framework and integrated patient feedback into 6 of the 12 steps. Patient input was included as it has potential to improve patient-centeredness of checklists related to assessment of clinical performance. Focus group patient participants from 2 clinical sites were included: Thomas Jefferson University Hospital, Philadelphia, PA, and Northwestern University Hospital, Chicago, Illinois. The authors developed a preliminary instrument based on existing checklists, clinical experience, literature review, and input from an expert panel comprising health care professionals and patient advocates. They then refined the instrument based on feedback from 2 waves of patient focus groups, resulting in a final 21-item checklist. The checklist items assess if uncertainty was addressed in each step of the discharge communication, including the following major categories: introduction, test results/ED summary, no/uncertain diagnosis, next steps/follow-up, home care, reasons to return, and general communication skills. Patient input influenced both what items were included and the wording of items in the final checklist. This patient-centered, systematic approach to checklist development is built upon the rigor of the Checklist Development Checklist and provides an illustration of how to integrate patient feedback into the design of assessment tools when appropriate

Simulation-based mastery learning compared to standard education for discussing diagnostic uncertainty with patients in the emergency department: a randomized controlled trial.

BACKGROUND: Diagnostic uncertainty occurs frequently in emergency medical care, with more than one-third of patients leaving the emergency department (ED) without a clear diagnosis. Despite this frequency, ED providers are not adequately trained on how to discuss diagnostic uncertainty with these patients, who often leave the ED confused and concerned. To address this training need, we developed the Uncertainty Communication Education Module (UCEM) to teach physicians how to discuss diagnostic uncertainty. The purpose of the study is to evaluate the effectiveness of the UCEM in improving physician communications. METHODS: The trial is a multicenter, two-arm randomized controlled trial designed to teach communication skills using simulation-based mastery learning (SBML). Resident emergency physicians from two training programs will be randomly assigned to immediate or delayed receipt of the two-part UCEM intervention after completing a baseline standardized patient encounter. The two UCEM components are: 1) a web-based interactive module, and 2) a smart-phone-based game. Both formats teach and reinforce communication skills for patient cases involving diagnostic uncertainty. Following baseline testing, participants in the immediate intervention arm will complete a remote deliberate practice session via a video platform and subsequently return for a second study visit to assess if they have achieved mastery. Participants in the delayed intervention arm will receive access to UCEM and remote deliberate practice after the second study visit. The primary outcome of interest is the proportion of residents in the immediate intervention arm who achieve mastery at the second study visit. DISCUSSION: Patients\u27 understanding of the care they received has implications for care quality, safety, and patient satisfaction, especially when they are discharged without a definitive diagnosis. Developing a patient-centered diagnostic uncertainty communication strategy will improve safety of acute care discharges. Although use of SBML is a resource intensive educational approach, this trial has been deliberately designed to have a low-resource, scalable intervention that would allow for widespread dissemination and uptake. TRIAL REGISTRATION: The trial was registered at clinicaltrials.gov (NCT04021771). Registration date: July 16, 2019

Looking to the Future

This chapter discusses a growing awareness that traditional methods of clinical education and evaluation in the health professions are becoming obsolete. It presents a new unifying educational framework for health professions individuals and teams that incorporates the science of mastery learning and deliberate practice, to better prepare health professionals for patient care responsibilities. The chapter aims to advance a roadmap for best practices research in health professions education and evaluation that addresses five facets: research opportunities, research methods, translational science, implementation science, and scholarship by individuals and teams. The research methods used by investigators in health professions education in scientific studies should be matched to the focused question being addressed. Evaluation best practices in the health professions will be identified, refined, disseminated worldwide, and adapted to local conditions as a result of scholarship and publication by individuals, and increasingly by teams

Timed Manual Performance In a Community Elderly Population

The extent of functional disability measured by performance of a series of simple manual tasks was determined in 1,286 elderly people, aged 66 to 98, who resided in central North Carolina. Missing data left 1,106 subjects for analysis. Poor manual performance was defined by either of two criteria: (1) inability to complete all the test items, or (2) taking longer than 350 seconds to complete the test. Using the first criterion, 113 persons (10.2%) were poor performers. When the two writing items were omitted from the analysis, 28 persons (2.4%) were in the poor performance group. Of the 993 persons who completed all items, 59 persons (5.9%) performed poorly. Using both criteria, the extent of functional disability was 16.1%; excluding the writing items, the extent of poor performance was 8.3%. Compared to those who performed well, those who performed poorly were older, poorer, less educated, more likely to be black, and less healthy. Because manual ability correlates highly with functional dependency and identifies those at risk for increased care needs, the magnitude of manual dysfunction noted in this survey has important implications for care providers and policy‐makers. © 1990 The American Geriatrics Societ