A report on the piloting of a novel computer-based medical case simulation for teaching and formative assessment of diagnostic laboratory testing

Objectives: Insufficient attention has been given to how information from computer-based clinical case simulations is presented, collected, and scored. Research is needed on how best to design such simulations to acquire valid performance assessment data that can act as useful feedback for educational applications. This report describes a study of a new simulation format with design features aimed at improving both its formative assessment feedback and educational function. Methods: Case simulation software (LabCAPS) was developed to target a highly focused and well-defined measurement goal with a response format that allowed objective scoring. Data from an eight-case computer-based performance assessment administered in a pilot study to 13 second-year medical students was analyzed using classical test theory and generalizability analysis. In addition, a similar analysis was conducted on an administration in a less controlled setting, but to a much large sample (n=143), within a clinical course that utilized two random case subsets from a library of 18 cases. Results: Classical test theory case-level item analysis of the pilot assessment yielded an average case discrimination of 0.37, and all eight cases were positively discriminating (range=0.11–0.56). Classical test theory coefficient alpha and the decision study showed the eight-case performance assessment to have an observed reliability of σ=G=0.70. The decision study further demonstrated that a G=0.80 could be attained with approximately 3 h and 15 min of testing. The less-controlled educational application within a large medical class produced a somewhat lower reliability for eight cases (G=0.53). Students gave high ratings to the logic of the simulation interface, its educational value, and to the fidelity of the tasks. Conclusions: LabCAPS software shows the potential to provide formative assessment of medical students’ skill at diagnostic test ordering and to provide valid feedback to learners. The perceived fidelity of the performance tasks and the statistical reliability findings support the validity of using the automated scores for formative assessment and learning. LabCAPS cases appear well designed for use as a scored assignment, for stimulating discussions in small group educational settings, for self-assessment, and for independent learning. Extension of the more highly controlled pilot assessment study with a larger sample will be needed to confirm its reliability in other assessment applications

Using item response theory to explore the psychometric properties of extended matching questions examination in undergraduate medical education

BACKGROUND: As assessment has been shown to direct learning, it is critical that the examinations developed to test clinical competence in medical undergraduates are valid and reliable. The use of extended matching questions (EMQ) has been advocated to overcome some of the criticisms of using multiple-choice questions to test factual and applied knowledge. METHODS: We analysed the results from the Extended Matching Questions Examination taken by 4th year undergraduate medical students in the academic year 2001 to 2002. Rasch analysis was used to examine whether the set of questions used in the examination mapped on to a unidimensional scale, the degree of difficulty of questions within and between the various medical and surgical specialties and the pattern of responses within individual questions to assess the impact of the distractor options. RESULTS: Analysis of a subset of items and of the full examination demonstrated internal construct validity and the absence of bias on the majority of questions. Three main patterns of response selection were identified. CONCLUSION: Modern psychometric methods based upon the work of Rasch provide a useful approach to the calibration and analysis of EMQ undergraduate medical assessments. The approach allows for a formal test of the unidimensionality of the questions and thus the validity of the summed score. Given the metric calibration which follows fit to the model, it also allows for the establishment of items banks to facilitate continuity and equity in exam standards

Does Medical Students' Preference of Test Format (Computer-based vs. Paper-based) have an Influence on Performance?

<p>Abstract</p> <p>Background</p> <p>Computer-based examinations (CBE) ensure higher efficiency with respect to producibility and assessment compared to paper-based examinations (PBE). However, students often have objections against CBE and are afraid of getting poorer results in a CBE.</p> <p>The aims of this study were (1) to assess the readiness and the objections of students to a CBE vs. PBE (2) to examine the acceptance and satisfaction with the CBE on a voluntary basis, and (3) to compare the results of the examinations, which were conducted in different formats.</p> <p>Methods</p> <p>Fifth year medical students were introduced to an examination-player and were free to choose their format for the test. The reason behind the choice of the format as well as the satisfaction with the choice was evaluated after the test with a questionnaire. Additionally, the expected and achieved examination results were measured.</p> <p>Results</p> <p>Out of 98 students, 36 voluntarily chose a CBE (37%), 62 students chose a PBE (63%). Both groups did not differ concerning sex, computer-experience, their achieved examination results of the test, and their satisfaction with the chosen format. Reasons for the students' objections against CBE include the possibility for outlines or written notices, a better overview, additional noise from the keyboard or missing habits normally present in a paper based exam. The students with the CBE tended to judge their examination to be more clear and understandable. Moreover, they saw their results to be independent of the format.</p> <p>Conclusions</p> <p>Voluntary computer-based examinations lead to equal test scores compared to a paper-based format.</p

Comparison of the sensitivity of the UKCAT and A levels to sociodemographic characteristics: a national study

Background: The UK Clinical Aptitude Test (UKCAT) was introduced to facilitate widening participation in medical and dental education in the UK by providing universities with a continuous variable to aid selection; one that might be less sensitive to the sociodemographic background of candidates compared to traditional measures of educational attainment. Initial research suggested that males, candidates from more advantaged socioeconomic backgrounds and those who attended independent or grammar schools performed better on the test. The introduction of the A* grade at A level permits more detailed analysis of the relationship between UKCAT scores, secondary educational attainment and sociodemographic variables. Thus, our aim was to further assess whether the UKCAT is likely to add incremental value over A level (predicted or actual) attainment in the selection process. Methods: Data relating to UKCAT and A level performance from 8,180 candidates applying to medicine in 2009 who had complete information relating to six key sociodemographic variables were analysed. A series of regression analyses were conducted in order to evaluate the ability of sociodemographic status to predict performance on two outcome measures: A level ‘best of three’ tariff score; and the UKCAT scores. Results: In this sample A level attainment was independently and positively predicted by four sociodemographic variables (independent/grammar schooling, White ethnicity, age and professional social class background). These variables also independently and positively predicted UKCAT scores. There was a suggestion that UKCAT scores were less sensitive to educational background compared to A level attainment. In contrast to A level attainment, UKCAT score was independently and positively predicted by having English as a first language and male sex. Conclusions: Our findings are consistent with a previous report; most of the sociodemographic factors that predict A level attainment also predict UKCAT performance. However, compared to A levels, males and those speaking English as a first language perform better on UKCAT. Our findings suggest that UKCAT scores may be more influenced by sex and less sensitive to school type compared to A levels. These factors must be considered by institutions utilising the UKCAT as a component of the medical and dental school selection process

Workplace-based assessment: effects of rater expertise

Traditional psychometric approaches towards assessment tend to focus exclusively on quantitative properties of assessment outcomes. This may limit more meaningful educational approaches towards workplace-based assessment (WBA). Cognition-based models of WBA argue that assessment outcomes are determined by cognitive processes by raters which are very similar to reasoning, judgment and decision making in professional domains such as medicine. The present study explores cognitive processes that underlie judgment and decision making by raters when observing performance in the clinical workplace. It specifically focuses on how differences in rating experience influence information processing by raters. Verbal protocol analysis was used to investigate how experienced and non-experienced raters select and use observational data to arrive at judgments and decisions about trainees’ performance in the clinical workplace. Differences between experienced and non-experienced raters were assessed with respect to time spent on information analysis and representation of trainee performance; performance scores; and information processing––using qualitative-based quantitative analysis of verbal data. Results showed expert-novice differences in time needed for representation of trainee performance, depending on complexity of the rating task. Experts paid more attention to situation-specific cues in the assessment context and they generated (significantly) more interpretations and fewer literal descriptions of observed behaviors. There were no significant differences in rating scores. Overall, our findings seemed to be consistent with other findings on expertise research, supporting theories underlying cognition-based models of assessment in the clinical workplace. Implications for WBA are discussed

Ebola: translational science considerations

We are currently in the midst of the most aggressive and fulminating outbreak of Ebola-related disease, commonly referred to as “Ebola”, ever recorded. In less than a year, the Ebola virus (EBOV, Zaire ebolavirus species) has infected over 10,000 people, indiscriminately of gender or age, with a fatality rate of about 50%. Whereas at its onset this Ebola outbreak was limited to three countries in West Africa (Guinea, where it was first reported in late March 2014, Liberia, where it has been most rampant in its capital city, Monrovia and other metropolitan cities, and Sierra Leone), cases were later reported in Nigeria, Mali and Senegal, as well as in Western Europe (i.e., Madrid, Spain) and the US (i.e., Dallas, Texas; New York City) by late October 2014. World and US health agencies declared that the current Ebola virus disease (EVD) outbreak has a strong likelihood of growing exponentially across the world before an effective vaccine, treatment or cure can be developed, tested, validated and distributed widely. In the meantime, the spread of the disease may rapidly evolve from an epidemics to a full-blown pandemic. The scientific and healthcare communities actively research and define an emerging kaleidoscope of knowledge about critical translational research parameters, including the virology of EBOV, the molecular biomarkers of the pathological manifestations of EVD, putative central nervous system involvement in EVD, and the cellular immune surveillance to EBOV, patient-centered anthropological and societal parameters of EVD, as well as translational effectiveness about novel putative patient-targeted vaccine and pharmaceutical interventions, which hold strong promise, if not hope, to curb this and future Ebola outbreaks. This work reviews and discusses the principal known facts about EBOV and EVD, and certain among the most interesting ongoing or future avenues of research in the field, including vaccination programs for the wild animal vectors of the virus and the disease from global translational science perspective

Synchronous chaos and broad band gamma rhythm in a minimal multi-layer model of primary visual cortex

Visually induced neuronal activity in V1 displays a marked gamma-band component which is modulated by stimulus properties. It has been argued that synchronized oscillations contribute to these gamma-band activity [... however,] even when oscillations are observed, they undergo temporal decorrelation over very few cycles. This is not easily accounted for in previous network modeling of gamma oscillations. We argue here that interactions between cortical layers can be responsible for this fast decorrelation. We study a model of a V1 hypercolumn, embedding a simplified description of the multi-layered structure of the cortex. When the stimulus contrast is low, the induced activity is only weakly synchronous and the network resonates transiently without developing collective oscillations. When the contrast is high, on the other hand, the induced activity undergoes synchronous oscillations with an irregular spatiotemporal structure expressing a synchronous chaotic state. As a consequence the population activity undergoes fast temporal decorrelation, with concomitant rapid damping of the oscillations in LFPs autocorrelograms and peak broadening in LFPs power spectra. [...] Finally, we argue that the mechanism underlying the emergence of synchronous chaos in our model is in fact very general. It stems from the fact that gamma oscillations induced by local delayed inhibition tend to develop chaos when coupled by sufficiently strong excitation.Comment: 49 pages, 11 figures, 7 table