Sources of Measurement Error in an ECG Examination: Implications for Performance-Based Assessments

Objective: To assess the sources of measurement error in an electrocardiogram (ECG) interpretation examination given in a third-year internal medicine clerkship. Design: Three successive generalizability studies were conducted. 1) Multiple faculty rated student responses to a previously administered exam. 2) The rating criteria were revised and study 1 was repeated. 3) The examination was converted into an extended matching format including multiple cases with the same underlying cardiac problem. Results: The discrepancies among raters (main effects and interactions) were dwarfed by the error associated with case specificity. The largest source of the differences among raters was in rating student errors of commission rather than student errors of omission. Revisions in the rating criteria may have helped increase inter-rater reliability slightly however, due to case specificity, it had little impact on the overall reliability of the exam. The third study indicated the majority of the variability in student performance across cases was in performance across cases within the same type of cardiac problem rather than between different types of cardiac problems. Conclusions: Case specificity was the overwhelming source of measurement error. The variation among cases came mainly from discrepancies in performance between examples of the same cardiac problem rather than from differences in performance across different types of cardiac problems. This suggests it is necessary to include a large number of cases even if the goal is to assess performance on only a few types of cardiac problems

Preoperative Factors and Three Year Weight Change in the Longitudinal Assessment of Bariatric Surgery (LABS) Consortium

BACKGROUND: Limited data guide the prediction of weight loss success or failure following bariatric surgery according to pre-surgery factors. There is significant variation in weight change following bariatric surgery and much interest in identifying pre-operative factors that may contribute to these differences. OBJECTIVE: This report evaluates the associations of a comprehensive set of baseline factors and three-year weight change. SETTING: Ten hospitals in six geographically diverse clinical centers in the United States. METHODS: PARTICIPANTS AND INTERVENTIONS: Adults undergoing a first bariatric surgical procedure as part of clinical care by participating surgeons were recruited between 2006 and 2009. Participants completed research assessments utilizing standardized and detailed data collection on over 100 preoperative and operative parameters for individuals undergoing Roux-en-Y gastric bypass (RYGB) and laparoscopic adjustable gastric banding (LAGB). Weight was measured 3 years following surgery. METHODS: MAIN OUTCOME MEASURES: Percent weight change for RYGB or LAGB from baseline to 3 years was analyzed as both a continuous and dichotomous outcome with cut points at 25% for RYGB and 10% for LAGB. Multivariable linear and logistic regression models were used to identify independent baseline predictors of the continuous and categorical outcomes, respectively. RESULTS: The median weight loss 3 years following surgery for RYGB (n=1513) participants was 31.5% (IQR: 24.6%–38.4%; range, 59.2% loss to 0.9% gain) of baseline weight and 16.0% (IQR: 8.1%–23.1%; range, 56.1% loss to 12.5% gain) for LAGB (n=509) participants. The median age was 46 years for RYGB and 48 years for LAGB; 80% of RYGB participants and 75% of LAGB participants were female; and the median baseline Body Mass Index (BMI) was 46 kg/m(2) for RYGB and 44 kg/m(2) for LAGB. For RYGB, Black participants lost 2.7% less weight compared to Whites and participants with diabetes at baseline had 3.7% less weight loss at year 3 than those without diabetes at baseline. There were small but statistically significant differences in weight change for RYGB in those with abnormal kidney function and current or recent smoking. For LAGB participants, those with a large band had 75% greater odds of experiencing less than 10% weight loss after adjusting for BMI and sex. CONCLUSIONS: Few baseline variables were associated with three year weight change and the effects were small. These results indicate that baseline variables have limited predictive value for an individual’s chance of a successful weight loss outcome following bariatric surgery. TRIAL REGISTRATION: NCT00465829, ClinicalTrials.go