Can standardized patients replace physicians as OSCE examiners?

BACKGROUND: To reduce inter-rater variability in evaluations and the demand on physician time, standardized patients (SP) are being used as examiners in OSCEs. There is concern that SP have insufficient training to provide valid evaluation of student competence and/or provide feedback on clinical skills. It is also unknown if SP ratings predict student competence in other areas. The objectives of this study were: to examine student attitudes towards SP examiners; to compare SP and physician evaluations of competence; and to compare predictive validity of these scores, using performance on the multiple choice questions examination (MCQE) as the outcome variable. METHODS: This was a cross-sectional study of third-year medical students undergoing an OSCE during the Internal Medicine clerkship rotation. Fifty-two students rotated through 8 stations (6 physician, 2 SP examiners). Statistical tests used were Pearson's correlation coefficient, two-sample t-test, effect size calculation, and multiple linear regression. RESULTS: Most students reported that SP stations were less stressful, that SP were as good as physicians in giving feedback, and that SP were sufficiently trained to judge clinical skills. SP scored students higher than physicians (mean 90.4% +/- 8.9 vs. 82.2% +/- 3.7, d = 1.5, p < 0.001) and there was a weak correlation between the SP and physician scores (coefficient 0.4, p = 0.003). Physician scores were predictive of summative MCQE scores (regression coefficient = 0.88 [0.15, 1.61], P = 0.019) but there was no relationship between SP scores and summative MCQE scores (regression coefficient = -0.23, P = 0.133). CONCLUSION: These results suggest that SP examiners are acceptable to medical students, SP rate students higher than physicians and, unlike physician scores, SP scores are not related to other measures of competence

Longitudinal machine learning modeling of MS patient trajectories improves predictions of disability progression

Background and Objectives: Research in Multiple Sclerosis (MS) has recently focused on extracting knowledge from real-world clinical data sources. This type of data is more abundant than data produced during clinical trials and potentially more informative about real-world clinical practice. However, this comes at the cost of less curated and controlled data sets. In this work we aim to predict disability progression by optimally extracting information from longitudinal patient data in the real-world setting, with a special focus on the sporadic sampling problem. Methods: We use machine learning methods suited for patient trajectories modeling, such as recurrent neural networks and tensor factorization. A subset of 6682 patients from the MSBase registry is used. Results: We can predict disability progression of patients in a two-year horizon with an ROC-AUC of 0.85, which represents a 32% decrease in the ranking pair error (1-AUC) compared to reference methods using static clinical features. Conclusions: Compared to the models available in the literature, this work uses the most complete patient history for MS disease progression prediction and represents a step forward towards AI-assisted precision medicine in MS

Persistence on therapy and propensity matched outcome comparison of two subcutaneous interferon beta 1a dosages for multiple sclerosis

To compare treatment persistence between two dosages of interferon &beta;-1a in a large observational multiple sclerosis registry and assess disease outcomes of first line MS treatment at these dosages using propensity scoring to adjust for baseline imbalance in disease characteristics. Treatment discontinuations were evaluated in all patients within the MSBase registry who commenced interferon &beta;-1a SC thrice weekly (n = 4678). Furthermore, we assessed 2-year clinical outcomes in 1220 patients treated with interferon &beta;-1a in either dosage (22 &micro;g or 44 &micro;g) as their first disease modifying agent, matched on propensity score calculated from pre-treatment demographic and clinical variables. A subgroup analysis was performed on 456 matched patients who also had baseline MRI variables recorded. Overall, 4054 treatment discontinuations were recorded in 3059 patients. The patients receiving the lower interferon dosage were more likely to discontinue treatment than those with the higher dosage (25% vs. 20% annual probability of discontinuation, respectively). This was seen in discontinuations with reasons recorded as &ldquo;lack of efficacy&rdquo; (3.3% vs. 1.7%), &ldquo;scheduled stop&rdquo; (2.2% vs. 1.3%) or without the reason recorded (16.7% vs. 13.3% annual discontinuation rate, 22 &micro;g vs. 44 &micro;g dosage, respectively). Propensity score was determined by treating centre and disability (score without MRI parameters) or centre, sex and number of contrast-enhancing lesions (score including MRI parameters). No differences in clinical outcomes at two years (relapse rate, time relapse-free and disability) were observed between the matched patients treated with either of the interferon dosages. Treatment discontinuations were more common in interferon &beta;-1a 22 &micro;g SC thrice weekly. However, 2-year clinical outcomes did not differ between patients receiving the different dosages, thus replicating in a registry dataset derived from &ldquo;real-world&rdquo; database the results of the pivotal randomised trial. Propensity score matching effectively minimised baseline covariate imbalance between two directly compared sub-populations from a large observational registry

Comparative effectiveness of autologous hematopoietic stem cell transplant vs fingolimod, natalizumab, and ocrelizumab in highly active relapsing-remitting multiple sclerosis

Importance: Autologous hematopoietic stem cell transplant (AHSCT) is available for treatment of highly active multiple sclerosis (MS). Objective: To compare the effectiveness of AHSCT vs fingolimod, natalizumab, and ocrelizumab in relapsing-remitting MS by emulating pairwise trials. Design, Setting, and Participants: This comparative treatment effectiveness study included 6 specialist MS centers with AHSCT programs and international MSBase registry between 2006 and 2021. The study included patients with relapsing-remitting MS treated with AHSCT, fingolimod, natalizumab, or ocrelizumab with 2 or more years study follow-up including 2 or more disability assessments. Patients were matched on a propensity score derived from clinical and demographic characteristics. Exposure: AHSCT vs fingolimod, natalizumab, or ocrelizumab. Main outcomes: Pairwise-censored groups were compared on annualized relapse rates (ARR) and freedom from relapses and 6-month confirmed Expanded Disability Status Scale (EDSS) score worsening and improvement. Results: Of 4915 individuals, 167 were treated with AHSCT; 2558, fingolimod; 1490, natalizumab; and 700, ocrelizumab. The prematch AHSCT cohort was younger and with greater disability than the fingolimod, natalizumab, and ocrelizumab cohorts; the matched groups were closely aligned. The proportion of women ranged from 65% to 70%, and the mean (SD) age ranged from 35.3 (9.4) to 37.1 (10.6) years. The mean (SD) disease duration ranged from 7.9 (5.6) to 8.7 (5.4) years, EDSS score ranged from 3.5 (1.6) to 3.9 (1.9), and frequency of relapses ranged from 0.77 (0.94) to 0.86 (0.89) in the preceding year. Compared with the fingolimod group (769 [30.0%]), AHSCT (144 [86.2%]) was associated with fewer relapses (ARR: mean [SD], 0.09 [0.30] vs 0.20 [0.44]), similar risk of disability worsening (hazard ratio [HR], 1.70; 95% CI, 0.91-3.17), and higher chance of disability improvement (HR, 2.70; 95% CI, 1.71-4.26) over 5 years. Compared with natalizumab (730 [49.0%]), AHSCT (146 [87.4%]) was associated with marginally lower ARR (mean [SD], 0.08 [0.31] vs 0.10 [0.34]), similar risk of disability worsening (HR, 1.06; 95% CI, 0.54-2.09), and higher chance of disability improvement (HR, 2.68; 95% CI, 1.72-4.18) over 5 years. AHSCT (110 [65.9%]) and ocrelizumab (343 [49.0%]) were associated with similar ARR (mean [SD], 0.09 [0.34] vs 0.06 [0.32]), disability worsening (HR, 1.77; 95% CI, 0.61-5.08), and disability improvement (HR, 1.37; 95% CI, 0.66-2.82) over 3 years. AHSCT-related mortality occurred in 1 of 159 patients (0.6%). Conclusion: In this study, the association of AHSCT with preventing relapses and facilitating recovery from disability was considerably superior to fingolimod and marginally superior to natalizumab. This study did not find evidence for difference in the effectiveness of AHSCT and ocrelizumab over a shorter available follow-up time

Ocrelizumab versus Interferon Beta-1a in Relapsing Multiple Sclerosis

Supported by F. Hoffmann–La Roche