Do OSCE assessments reflect clinical maturation of medical students? An evaluation of progression in core knowledge and competency domains

Introduction/Background Application of knowledge in clinical settings can be assessed using several different methods. However, constructing an effective evaluation system for an educational program is a challenging task. Purpose/Objectives In this study, we aimed to evaluate our medical curriculum by investigating medical students' progressive acquisition of core clinical competencies in their clinical years, using domain-based and behaviourally anchored checklist and global rating scales in the OSCE format. Issues/Questions for exploration/Methodology Three collaboratively developed OSCE cases were administered in the 2012 end of year OSCEs to all 3rd, 4th and 5th year medical students. These OSCEs assessed core clinical competencies Year of study effect on checklist and global scores as well as competency scores were analysed using the multivariate analysis of variance. Results Irrespective of year of study, the results showed similar patterns in student performance across the three OSCE stations. However, performance was significantly affected by year of study (p<0.01) and case difficulty (p<0.001). Discussion There was a progressive increase in performance, with the more senior students obtaining higher checklist and global scores. They also demonstrated better performance in more complex clinical skills such as diagnostic skills and investigation plan. Conclusions The use of domain-based and behaviourally anchored checklist and global rating scales enhance the validity of the OSCE in measuring progressive clinical competence. It also enables identification of problem areas and provides feedback to faculty on the efficacy of the curriculum

Research: Why aren’t more medical students doing it?

Background Many medical schools in Australia and Internationally struggle to engage medical students along a research pathway. Aims The aim of this study was to identify medical student’s confidence, attitudes and perceived obstacles to participating in research during medical school. Methods A cross-sectional survey was carried out of undergraduate medical students in years 1–5 attending the University of Tasmania. Results Of the 237 students who responded to the survey (response rate of 41.9 per cent) the majority (70.3 per cent) agreed that research was a useful experience and expressed a desire to be involved in research (60.1 per cent). Women were generally less confident than men in their ability to conduct research while research experience was significantly associated with an increase in confidence in conducting research. Frequently endorsed reasons by students for not undertaking an Honours year were a desire to not delay graduation by a year for financial (79 per cent) and employment reasons (71 per cent) and social concerns regarding integrating with a different year group (69 per cent). Additional concerns included a lack of time and motivation. Conclusion The difficulties inherent for students in undertaking research including low confidence, lack of time, financial constraints and pressure to get out into the workforce should help to inform medical educators in developing solutions to encourage student participation in research

Inter-rater reliability: comparison of checklist and global scoring for OSCEs

Objective Structured Clinical Examinations (OSCEs) have been used globally in evaluating clinical competence in the education of health professionals. Despite the objective intent of OSCEs, scoring methods used by examiners have been a potential source of measurement error affecting the precision with which test scores are determined. In this study, we investigated the differences in the inter-rater reliabilities of objective checklist and subjective global rating scores of examiners (who were exposed to an online training program to standardise scoring techniques) across two medical schools. Examiners' perceptions of the e-scoring program were also investigated. Two Australian universities shared three OSCE stations in their end-of-year undergraduate medical OSCEs. The scenarios were video-taped and used for on-line examiner training prior to actual exams. Examiner ratings of performance at both sites were analysed using generalisability theory. A single facet, all random persons by raters design [PxR] was used to measure inter-rater reliability for each station, separate for checklist scores and global ratings. The resulting variance components were pooled across stations and examination sites. Decision studies were used to measure reliability estimates. There was no significant mean score difference between examination sites. Variation in examinee ability accounted for 68.3% of the total variance in checklist scores and 90.2% in global ratings. Rater contribution was 1.4% & 0% of the total variance in checklist score and global rating respectively, reflecting high inter-rater reliability of the scores provided by co-examiners across the two schools. Score variance due to interaction and residual error was larger for checklist scores (30.3% vs 9.7%) than for global ratings. Reproducibility coefficients for global ratings were higher than for checklist scores. Survey results showed that the e-scoring package facilitated consensus on scoring techniques. This approach to examiner training also allowed examiners to calibrate the OSCEs in their own time. This study revealed that inter-rater reliability was higher for global ratings than for checklist scores, thus providing further evidence for the reliability of subjective examiner ratings

The predictive validity of the UMAT : a multi-institutional study

Background and Aims: This multi-centre study aimed to determine the predictive validity of the Undergraduate Medicine and Health Sciences Admission Test (UMAT) and other selection tools (ATAR/GPA, Interview) on medical student academic performance, to inform best practice in undergraduate medical student selection. Methods: Eleven of twelve universities across Australia and New Zealand participated. The dataset consists of N=10471 medical students commencing from 2006-2012. Admissions scores (UMAT, ATAR/GPA and Interview score), sex and age were used to predict academic performance during medical school. Two transition points were identified as outcome variables using knowledge- and clinical-based assessment results as well as total year scores. These were the transition from campus-based to hospital-based training (T1), and between the penultimate to pre-internship year (T2). An overall course total was also included (T3). Results: Hierarchical Multiple Regressions were used to analyse the predictive validity of the selection tools. ATAR/GPA was the strongest predictor of higher performance at the transition years (T1 &T2) and overall during the course (T3). UMAT Section 1 (Problem Solving and Logical Reasoning) was a significant predictor of increased performance at all three points. UMAT Section 2 (Understanding People) was a significant predictor at T1 and T2. UMAT Section 3 (Non-Verbal Reasoning) was a significant negative predictor at T2 and T3. Interview score was a predictor of higher performance in the hospital-based years, and of overall course performance. Conclusions / Recommendations: The presentation reports the preliminary analysis of this comprehensive dataset. Additional analyses using advanced statistical modelling techniques are underway to further determine the predictive validity of selection tools on medical student academic performance

Ivermectin increases striatal cholinergic activity to facilitate dopamine terminal function

Abstract Ivermectin (IVM) is a commonly prescribed antiparasitic treatment with pharmacological effects on invertebrate glutamate ion channels resulting in paralysis and death of invertebrates. However, it can also act as a modulator of some vertebrate ion channels and has shown promise in facilitating L-DOPA treatment in preclinical models of Parkinson’s disease. The pharmacological effects of IVM on dopamine terminal function were tested, focusing on the role of two of IVM’s potential targets: purinergic P2X4 and nicotinic acetylcholine receptors. Ivermectin enhanced electrochemical detection of dorsal striatum dopamine release. Although striatal P2X4 receptors were observed, IVM effects on dopamine release were not blocked by P2X4 receptor inactivation. In contrast, IVM attenuated nicotine effects on dopamine release, and antagonizing nicotinic receptors prevented IVM effects on dopamine release. IVM also enhanced striatal cholinergic interneuron firing. L-DOPA enhances dopamine release by increasing vesicular content. L-DOPA and IVM co-application further enhanced release but resulted in a reduction in the ratio between high and low frequency stimulations, suggesting that IVM is enhancing release largely through changes in terminal excitability and not vesicular content. Thus, IVM is increasing striatal dopamine release through enhanced cholinergic activity on dopamine terminals

White-nose syndrome is associated with increased replication of a naturally persisting coronaviruses in bats

Spillover of viruses from bats to other animals may be associated with increased contact between them, as well as increased shedding of viruses by bats. Here, we tested the prediction that little brown bats (Myotis lucifugus) co-infected with the M. lucifugus coronavirus (Myl-CoV) and with Pseudogymnoascus destructans (Pd), the fungus that causes bat white-nose syndrome (WNS), exhibit different disease severity, viral shedding and molecular responses than bats infected with only Myl-CoV or only P. destructans. We took advantage of the natural persistence of Myl-CoV in bats that were experimentally inoculated with P. destructans in a previous study. Here, we show that the intestines of virus-infected bats that were also infected with fungus contained on average 60-fold more viral RNA than bats with virus alone. Increased viral RNA in the intestines correlated with the severity of fungus-related pathology. Additionally, the intestines of bats infected with fungus exhibited different expression of mitogen-activated protein kinase pathway and cytokine related transcripts, irrespective of viral presence. Levels of coronavirus antibodies were also higher in fungal-infected bats. Our results suggest that the systemic effects of WNS may down-regulate anti-viral responses in bats persistently infected with M. lucifugus coronavirus and increase the potential of virus shedding

White-nose syndrome is associated with increased replication of a naturally persisting coronaviruses in bats

Abstract Spillover of viruses from bats to other animals may be associated with increased contact between them, as well as increased shedding of viruses by bats. Here, we tested the prediction that little brown bats (Myotis lucifugus) co-infected with the M. lucifugus coronavirus (Myl-CoV) and with Pseudogymnoascus destructans (Pd), the fungus that causes bat white-nose syndrome (WNS), exhibit different disease severity, viral shedding and molecular responses than bats infected with only Myl-CoV or only P. destructans. We took advantage of the natural persistence of Myl-CoV in bats that were experimentally inoculated with P. destructans in a previous study. Here, we show that the intestines of virus-infected bats that were also infected with fungus contained on average 60-fold more viral RNA than bats with virus alone. Increased viral RNA in the intestines correlated with the severity of fungus-related pathology. Additionally, the intestines of bats infected with fungus exhibited different expression of mitogen-activated protein kinase pathway and cytokine related transcripts, irrespective of viral presence. Levels of coronavirus antibodies were also higher in fungal-infected bats. Our results suggest that the systemic effects of WNS may down-regulate anti-viral responses in bats persistently infected with M. lucifugus coronavirus and increase the potential of virus shedding