Are we failing to “build on the scientific basis of medicine?”

In this article, we question if and why the Canadian national medical education organizations have failed to introduce or promote changes that compel or encourage Canadian medical schools to heed the recommendation from the Future of Medical Education in Canada report to "build on the scientific basis of medicine." We end by offering suggestions on how these organizations could help Canadian medical schools build in the scientific basis of medicine

The impact of two multiple-choice question formats on the problem-solving strategies used by novices and experts

BACKGROUND: Pencil-and-paper examination formats, and specifically the standard, five-option multiple-choice question, have often been questioned as a means for assessing higher-order clinical reasoning or problem solving. This study firstly investigated whether two paper formats with differing number of alternatives (standard five-option and extended-matching questions) can test problem-solving abilities. Secondly, the impact of the alternatives number on psychometrics and problem-solving strategies was examined. METHODS: Think-aloud protocols were collected to determine the problem-solving strategy used by experts and non-experts in answering Gastroenterology questions, across the two pencil-and-paper formats. RESULTS: The two formats demonstrated equal ability in testing problem-solving abilities, while the number of alternatives did not significantly impact psychometrics or problem-solving strategies utilized. CONCLUSIONS: These results support the notion that well-constructed multiple-choice questions can in fact test higher order clinical reasoning. Furthermore, it can be concluded that in testing clinical reasoning, the question stem, or content, remains more important than the number of alternatives

Comparison of Student Performance on Internally Prepared Clerkship Examinations and NBME Subject Examinations

Background: This pilot study compared performance of University of Calgary students on internal clerkship examinations with corresponding National Board of Medical Examiners (NBME) subject examinations.Methods: Between April and October 2007, students completed internal and NBME subject examinations following six mandatory rotations. Local faculty within each discipline set the minimum performance level (MPL) for internal examinations. Two methods of standard setting were considered for NBME exams and a sensitivity analysis was performed. Corresponding internal and NBME examination scores were compared using McNemar’s discordant pair analysis.Results: A significant and unexpected difference in failure rate between internal and external examinations was found in all clerkships. 1.4% of students were below the MPL for internal examinations and 27.3% (modified Angoff) or 25.9% (mean Hofstee compromise) (p<0.0001 for both) for the NBME. The proportion of students below MPL for internal examinations was also below the lower limit of the Hofstee compromise (14.4%).Conclusion: Possible explanations include leniency bias in internal standard setting, discrepant content validity between local curriculum and NBME examinations, difference in student perception of examinations, and performance bias due to unfamiliar units

Ego Identity Status of Medical Students in Clerkship

Background: Medical students encounter a variety of experiences that have an impact on their emerging professional identity. Clerkship, in particular, presents opportunities for students to consider their career options and decide upon a career path. The process of developing their professional identity begins well before clerkship, however. Anecdotal evidence suggests that interests in medicine begin as early as childhood. This study retrospectively examines the decision-making process clerks make in choosing medicine as a career.Methods: A total of 76 clerks (36 male, 34 female, 6 not reported) responded to four open-ended and two follow-up questions that measure career interests and pursuits. Questions addressed when and how students developed interests in medicine and alternate careers before beginning medical school. An additional eight closed questions drawn from the Ego Status Extended Objective Measure of Ego Identity Status II (EOM-EIS-II) were administered. Content analyses and inter-rater reliability analyses were conducted to classify students according to Marcia’s1  four ego identity statuses.Results: Having obtained high inter-rater consistency (Cohen’s Kappa coefficient of 0.92), responses to the open-ended questions resulted in the classification of three identity statuses. In total, 49.3% of students were in the ‘achieved’ (high exploration and commitment to choices) status and 48.1% were in the ‘foreclosed’ (low exploration but high commitment to choices) status. A small percentage (1.3%) of students were in the ‘moratorium’category (high exploration but low commitment to choices), while none of the students were in the ‘diffused’ (low exploration and low commitment to choices) category.Conclusions: With approximately half of the students demonstrating a ‘foreclosed’ status, this study reveals that despite exposure to a variety of careers when attending university, only half of the students had seriously pursued a career outside of medicine. The majority of students, moreover, developed an interest in medicine before adulthood, and did so independently from parental influence

Does the Medical College Admission Test (MCAT) predict licensing examination performance in the Canadian context?

Background: Research on the predictive validity of the Medical College Admissions Test (MCAT) on licensing examination performance is varied in its conclusions, with only a few studies examining this relationship in a Canadian context. We assessed the predictive validity of the MCAT on successful performance on the Medical Council of Canada Qualifying Examination (MCCQE) Part 1 by students attending the Cumming School of Medicine.   Methods: Prospective data were collected on MCAT score and sub-section scores, MCCQE decision, multiple mini interview (MMI) performance, gender, and age. The cohort was divided into a derivation cohort (2013 and 2014) and validation cohort (2015 and 2016). Students were dichotomized into pass or fail on MCCQE. Multiple logistic regression in which our dependent variable was MCCQE Part I examination success at the first attempt was used, and potential explanatory variables were age, gender, MCAT total score, and sub-scores for the biological sciences (MCAT-BS), physical sciences, and verbal reasoning, GPA, and MMI ratings.Results: For the derivation cohort MCAT-BS was associated with success on the MCCQE Part I. The odds ratio for this association of 1.37 (95% confidence interval [1.01, 1.85], p = 0.04). When we applied the MCAT-BS to our validation cohort the odds ratio of MCCQE Part I examination success was 1.42 [1.10, 1.83], p = 0.007) and the area under the ROC curve was 0.66 [0.54, 0.79]).Conclusion: The MCAT-BS predicted successful performance on the MCCQE Part 1 Examination in the Canadian setting

Experienced physicians benefit from analyzing initial diagnostic hypotheses

Background: Most incorrect diagnoses involve at least one cognitive error, of which premature closure is the most prevalent. While metacognitive strategies can mitigate premature closure in inexperienced learners, these are rarely studied in experienced physicians. Our objective here was to evaluate the effect of analytic information processing on diagnostic performance of nephrologists and nephrology residents. Methods: We asked nine nephrologists and six nephrology residents at the University of Calgary and Glasgow University to diagnose ten nephrology cases. We provided presenting features along with contextual information, after which we asked for an initial diagnosis. We then primed participants to use either hypothetico-deductive reasoning or scheme-inductive reasoning to analyze the remaining case data and generate a final diagnosis. Results: After analyzing initial hypotheses, both nephrologists and residents improved the accuracy of final diagnoses (31.1% vs. 65.6%, p < 0.001, and 40.0% vs. 70.0%, p < 0.001, respectively). We found a significant interaction between experience and analytic processing strategy (p = 0.002): nephrology residents had significantly increased odds of diagnostic success when using scheme-inductive reasoning (odds ratio [95% confidence interval] 5.69 [1.59, 20.33], p = 0.007), whereas the performance of experienced nephrologists did not differ between strategies (odds ratio 0.57 [0.23, 1.39], p = 0.2). Discussion: Experienced nephrologists and nephrology residents can improve their performance by analyzing initial diagnostic hypotheses. The explanation of the interaction between experience and the effect of different reasoning strategies is unclear, but may relate to preferences in reasoning strategy, or the changes in knowledge structure with experience

A prospective randomized trial of content expertise versus process expertise in small group teaching

Article deposited according to agreement with BMC, December 6, 2010.YesFunding provided by the Open Access Authors Fund

Can standardized patients replace physicians as OSCE examiners?

BACKGROUND: To reduce inter-rater variability in evaluations and the demand on physician time, standardized patients (SP) are being used as examiners in OSCEs. There is concern that SP have insufficient training to provide valid evaluation of student competence and/or provide feedback on clinical skills. It is also unknown if SP ratings predict student competence in other areas. The objectives of this study were: to examine student attitudes towards SP examiners; to compare SP and physician evaluations of competence; and to compare predictive validity of these scores, using performance on the multiple choice questions examination (MCQE) as the outcome variable. METHODS: This was a cross-sectional study of third-year medical students undergoing an OSCE during the Internal Medicine clerkship rotation. Fifty-two students rotated through 8 stations (6 physician, 2 SP examiners). Statistical tests used were Pearson's correlation coefficient, two-sample t-test, effect size calculation, and multiple linear regression. RESULTS: Most students reported that SP stations were less stressful, that SP were as good as physicians in giving feedback, and that SP were sufficiently trained to judge clinical skills. SP scored students higher than physicians (mean 90.4% +/- 8.9 vs. 82.2% +/- 3.7, d = 1.5, p < 0.001) and there was a weak correlation between the SP and physician scores (coefficient 0.4, p = 0.003). Physician scores were predictive of summative MCQE scores (regression coefficient = 0.88 [0.15, 1.61], P = 0.019) but there was no relationship between SP scores and summative MCQE scores (regression coefficient = -0.23, P = 0.133). CONCLUSION: These results suggest that SP examiners are acceptable to medical students, SP rate students higher than physicians and, unlike physician scores, SP scores are not related to other measures of competence

Involvement in teaching improves learning in medical students: a randomized cross-over study

<p>Abstract</p> <p>Background</p> <p>Peer-assisted learning has many purported benefits including preparing students as educators, improving communication skills and reducing faculty teaching burden. But comparatively little is known about the effects of teaching on learning outcomes of peer educators in medical education.</p> <p>Methods</p> <p>One hundred and thirty-five first year medical students were randomly allocated to 11 small groups for the Gastroenterology/Hematology Course at the University of Calgary. For each of 22 sessions, two students were randomly selected from each group to be peer educators. Students were surveyed to estimate time spent preparing as peer educator versus group member. Students completed an end-of-course 94 question multiple choice exam. A paired t-test was used to compare performance on clinical presentations for which students were peer educators to those for which they were not.</p> <p>Results</p> <p>Preparation time increased from a mean (SD) of 36 (33) minutes baseline to 99 (60) minutes when peer educators (Cohen's <it>d </it>= 1.3; p < 0.001). The mean score (SD) for clinical presentations in which students were peer educators was 80.7% (11.8) compared to77.6% (6.9) for those which they were not (<it>d </it>= 0.33; <it>p </it>< 0.01).</p> <p>Conclusion</p> <p>Our results suggest that involvement in teaching small group sessions improves medical students' knowledge acquisition and retention.</p