Can standardized patients replace physicians as OSCE examiners?

BACKGROUND: To reduce inter-rater variability in evaluations and the demand on physician time, standardized patients (SP) are being used as examiners in OSCEs. There is concern that SP have insufficient training to provide valid evaluation of student competence and/or provide feedback on clinical skills. It is also unknown if SP ratings predict student competence in other areas. The objectives of this study were: to examine student attitudes towards SP examiners; to compare SP and physician evaluations of competence; and to compare predictive validity of these scores, using performance on the multiple choice questions examination (MCQE) as the outcome variable. METHODS: This was a cross-sectional study of third-year medical students undergoing an OSCE during the Internal Medicine clerkship rotation. Fifty-two students rotated through 8 stations (6 physician, 2 SP examiners). Statistical tests used were Pearson's correlation coefficient, two-sample t-test, effect size calculation, and multiple linear regression. RESULTS: Most students reported that SP stations were less stressful, that SP were as good as physicians in giving feedback, and that SP were sufficiently trained to judge clinical skills. SP scored students higher than physicians (mean 90.4% +/- 8.9 vs. 82.2% +/- 3.7, d = 1.5, p < 0.001) and there was a weak correlation between the SP and physician scores (coefficient 0.4, p = 0.003). Physician scores were predictive of summative MCQE scores (regression coefficient = 0.88 [0.15, 1.61], P = 0.019) but there was no relationship between SP scores and summative MCQE scores (regression coefficient = -0.23, P = 0.133). CONCLUSION: These results suggest that SP examiners are acceptable to medical students, SP rate students higher than physicians and, unlike physician scores, SP scores are not related to other measures of competence

Longitudinal machine learning modeling of MS patient trajectories improves predictions of disability progression

Background and Objectives: Research in Multiple Sclerosis (MS) has recently focused on extracting knowledge from real-world clinical data sources. This type of data is more abundant than data produced during clinical trials and potentially more informative about real-world clinical practice. However, this comes at the cost of less curated and controlled data sets. In this work we aim to predict disability progression by optimally extracting information from longitudinal patient data in the real-world setting, with a special focus on the sporadic sampling problem. Methods: We use machine learning methods suited for patient trajectories modeling, such as recurrent neural networks and tensor factorization. A subset of 6682 patients from the MSBase registry is used. Results: We can predict disability progression of patients in a two-year horizon with an ROC-AUC of 0.85, which represents a 32% decrease in the ranking pair error (1-AUC) compared to reference methods using static clinical features. Conclusions: Compared to the models available in the literature, this work uses the most complete patient history for MS disease progression prediction and represents a step forward towards AI-assisted precision medicine in MS

Persistence on therapy and propensity matched outcome comparison of two subcutaneous interferon beta 1a dosages for multiple sclerosis

To compare treatment persistence between two dosages of interferon &beta;-1a in a large observational multiple sclerosis registry and assess disease outcomes of first line MS treatment at these dosages using propensity scoring to adjust for baseline imbalance in disease characteristics. Treatment discontinuations were evaluated in all patients within the MSBase registry who commenced interferon &beta;-1a SC thrice weekly (n = 4678). Furthermore, we assessed 2-year clinical outcomes in 1220 patients treated with interferon &beta;-1a in either dosage (22 &micro;g or 44 &micro;g) as their first disease modifying agent, matched on propensity score calculated from pre-treatment demographic and clinical variables. A subgroup analysis was performed on 456 matched patients who also had baseline MRI variables recorded. Overall, 4054 treatment discontinuations were recorded in 3059 patients. The patients receiving the lower interferon dosage were more likely to discontinue treatment than those with the higher dosage (25% vs. 20% annual probability of discontinuation, respectively). This was seen in discontinuations with reasons recorded as &ldquo;lack of efficacy&rdquo; (3.3% vs. 1.7%), &ldquo;scheduled stop&rdquo; (2.2% vs. 1.3%) or without the reason recorded (16.7% vs. 13.3% annual discontinuation rate, 22 &micro;g vs. 44 &micro;g dosage, respectively). Propensity score was determined by treating centre and disability (score without MRI parameters) or centre, sex and number of contrast-enhancing lesions (score including MRI parameters). No differences in clinical outcomes at two years (relapse rate, time relapse-free and disability) were observed between the matched patients treated with either of the interferon dosages. Treatment discontinuations were more common in interferon &beta;-1a 22 &micro;g SC thrice weekly. However, 2-year clinical outcomes did not differ between patients receiving the different dosages, thus replicating in a registry dataset derived from &ldquo;real-world&rdquo; database the results of the pivotal randomised trial. Propensity score matching effectively minimised baseline covariate imbalance between two directly compared sub-populations from a large observational registry

Ocrelizumab versus Interferon Beta-1a in Relapsing Multiple Sclerosis

Supported by F. Hoffmann–La Roche