A multi-source feedback tool for measuring a subset of Pediatrics Milestones

Background: The Pediatrics Milestones Assessment Pilot employed a new multisource feedback (MSF) instrument to assess nine Pediatrics Milestones among interns and subinterns in the inpatient context. Objective: To report validity evidence for the MSF tool for informing milestone classification decisions. Methods: We obtained MSF instruments by different raters per learner per rotation. We present evidence for validity based on the unified validity framework. Results: One hundred and ninety two interns and 41 subinterns at 18 Pediatrics residency programs received a total of 1084 MSF forms from faculty (40%), senior residents (34%), nurses (22%), and other staff (4%). Variance in ratings was associated primarily with rater (32%) and learner (22%). The milestone factor structure fit data better than simpler structures. In domains except professionalism, ratings by nurses were significantly lower than those by faculty and ratings by other staff were significantly higher. Ratings were higher when the rater observed the learner for longer periods and had a positive global opinion of the learner. Ratings of interns and subinterns did not differ, except for ratings by senior residents. MSF-based scales correlated with summative milestone scores. Conclusion: We obtain moderately reliable MSF ratings of interns and subinterns in the inpatient context to inform some milestone assignments