An overview of realist evaluation for simulation-based education

This article describes the key features of realist (realistic) evaluation and illustrates their application using, as an example, a simulation-based course for final year medical students. The use of simulation-based education (SBE) is increasing and so too is the evidence supporting its value as a powerful technique which can lead to substantial educational benefits. Accompanying these changes is a call for research into its use to be more theory-driven and to investigate both 'Did it work?' and as importantly 'Why did it work (or not)?' An evaluation methodology that is capable of answering both questions is realist evaluation. Realist evaluation is an emerging methodology that is suited to evaluating complex interventions such as SBE. The realist philosophy positions itself between positivist and constructivist paradigms and seeks to answer the question 'What works for whom, in what circumstances and why?' In seeking to answer this question, realist evaluation sets out to identify three fundamental components of an intervention, namely context, mechanism and outcome. Educational programmes work (successful outcomes) when theory-driven interventions (mechanisms) are applied to groups under appropriate conditions (context). Realist research uses a mixed methods (qualitative and quantitative) approach to gathering data in order to test the proposed context-mechanism-outcome (CMO) configurations of the intervention under investigation. Realist evaluation offers a valuable methodology for researchers investigating interventions utilising simulation-based education. By investigating and understanding the context, mechanisms and outcomes of SBE interventions, realist evaluation can provide the deeper level of understanding being called for

Choice of outcomes and measurement instruments in randomised trials on eLearning in medical education: A systematic mapping review protocol

Background: There will be a lack of 18 million healthcare workers by 2030. Multiplying the number of well-trained healthcare workers through innovative ways such as eLearning is highly recommended in solving this shortage. However, high heterogeneity of learning outcomes in eLearning systematic reviews reveals a lack of consistency and agreement on core learning outcomes in eLearning for medical education. In addition, there seems to be a lack of validity evidence for measurement instruments used in these trials. This undermines the credibility of these outcome measures and affects the ability to draw accurate and meaningful conclusions. The aim of this research is to address this issue by determining the choice of outcomes, measurement instruments and the prevalence of measurement instruments with validity evidence in randomised trials on eLearning for pre-registration medical education. Methods: We will conduct a systematic mapping and review to identify the types of outcomes, the kinds of measurement instruments and the prevalence of validity evidence among measurement instruments in eLearning randomised controlled trials (RCTs) in pre-registration medical education. The search period will be from January 1990 until August 2017. We will consider studies on eLearning for health professionals' education. Two reviewers will extract and manage data independently from the included studies. Data will be analysed and synthesised according to the aim of the review. Discussion: Appropriate choice of outcomes and measurement tools is essential for ensuring high-quality research in the field of eLearning and eHealth. The results of this study could have positive implications for other eHealth interventions, including (1) improving quality and credibility of eLearning research, (2) enhancing the quality of digital medical education and (3) informing researchers, academics and curriculum developers about the types of outcomes and validity evidence for measurement instruments used in eLearning studies. The protocol aspires to assist in the advancement of the eLearning research field as well as in the development of high-quality healthcare professionals' digital education. Systematic review registration: PROSPERO CRD4201706842

Using a conceptual framework during learning attenuates the loss of expert-type knowledge structure

BACKGROUND: During evolution from novice to expert, knowledge structure develops into an abridged network organized around pathophysiological concepts. The objectives of this study were to examine the change in knowledge structure in medical students in one year and to investigate the association between the use of a conceptual framework (diagnostic scheme) and long-term knowledge structure. METHODS: Medical students' knowledge structure of metabolic alkalosis was studied after instruction and one year later using concept-sorting. Knowledge structure was labeled 'expert-type' if students shared ≥ 2 concepts with experts and 'novice-type' if they shared < 2 concepts. Conditional logistic regression was used to study the association between short-term knowledge structure, the use of a diagnostic scheme and long-term knowledge structure. RESULTS: Thirty-four medical students completed the concept-sorting task on both occasions. Twenty-four used a diagnostic scheme for metabolic alkalosis. Short-term knowledge structure was not a correlate of long-term knowledge structure, whereas use of a diagnostic scheme was associated with increased odds of expert-type long-term knowledge structure (odds ratio 12.6 [1.4, 116.0], p = 0.02). There was an interaction between short-term knowledge structure and the use of a diagnostic scheme. In the group who did not use a diagnostic scheme the number of students changing from expert-type to novice-type was greater than vice versa (p = 0.046). There was no significant change in the group that used the diagnostic scheme (p = 0.6). CONCLUSION: The use of a diagnostic scheme by students may attenuate the loss of expert-type knowledge structure

Instructor feedback versus no instructor feedback on performance in a laparoscopic virtual reality simulator: a randomized educational trial

<p>Abstract</p> <p>Background</p> <p>Several studies have found a positive effect on the learning curve as well as the improvement of basic psychomotor skills in the operating room after virtual reality training. Despite this, the majority of surgical and gynecological departments encounter hurdles when implementing this form of training. This is mainly due to lack of knowledge concerning the time and human resources needed to train novice surgeons to an adequate level. The purpose of this trial is to investigate the impact of instructor feedback regarding time, repetitions and self-perception when training complex operational tasks on a virtual reality simulator.</p> <p>Methods/Design</p> <p>The study population consists of medical students on their 4^thto 6^thyear without prior laparoscopic experience. The study is conducted in a skills laboratory at a centralized university hospital. Based on a sample size estimation 98 participants will be randomized to an intervention group or a control group. Both groups have to achieve a predefined proficiency level when conducting a laparoscopic salpingectomy using a surgical virtual reality simulator. The intervention group receives standardized instructor feedback of 10 to 12 min a maximum of three times. The control group receives no instructor feedback. Both groups receive the automated feedback generated by the virtual reality simulator. The study follows the CONSORT Statement for randomized trials. Main outcome measures are time and repetitions to reach the predefined proficiency level on the simulator. We include focus on potential sex differences, computer gaming experience and self-perception.</p> <p>Discussion</p> <p>The findings will contribute to a better understanding of optimal training methods in surgical education.</p> <p>Trial Registration</p> <p><a href="http://www.clinicaltrials.gov/ct2/show/NCT01497782">NCT01497782</a></p

Comparison of the sensitivity of the UKCAT and A levels to sociodemographic characteristics: a national study

Background: The UK Clinical Aptitude Test (UKCAT) was introduced to facilitate widening participation in medical and dental education in the UK by providing universities with a continuous variable to aid selection; one that might be less sensitive to the sociodemographic background of candidates compared to traditional measures of educational attainment. Initial research suggested that males, candidates from more advantaged socioeconomic backgrounds and those who attended independent or grammar schools performed better on the test. The introduction of the A* grade at A level permits more detailed analysis of the relationship between UKCAT scores, secondary educational attainment and sociodemographic variables. Thus, our aim was to further assess whether the UKCAT is likely to add incremental value over A level (predicted or actual) attainment in the selection process. Methods: Data relating to UKCAT and A level performance from 8,180 candidates applying to medicine in 2009 who had complete information relating to six key sociodemographic variables were analysed. A series of regression analyses were conducted in order to evaluate the ability of sociodemographic status to predict performance on two outcome measures: A level ‘best of three’ tariff score; and the UKCAT scores. Results: In this sample A level attainment was independently and positively predicted by four sociodemographic variables (independent/grammar schooling, White ethnicity, age and professional social class background). These variables also independently and positively predicted UKCAT scores. There was a suggestion that UKCAT scores were less sensitive to educational background compared to A level attainment. In contrast to A level attainment, UKCAT score was independently and positively predicted by having English as a first language and male sex. Conclusions: Our findings are consistent with a previous report; most of the sociodemographic factors that predict A level attainment also predict UKCAT performance. However, compared to A levels, males and those speaking English as a first language perform better on UKCAT. Our findings suggest that UKCAT scores may be more influenced by sex and less sensitive to school type compared to A levels. These factors must be considered by institutions utilising the UKCAT as a component of the medical and dental school selection process