While the selection of qualified applicants often relies, in part, on scores generated from a medical school pre-admission interview (MSPI), the growth of regional medical campuses (RMCs) – many with specialized rural tracks, programs, or missions – has challenged schools to accommodate a wider range of stakeholder input. This study examines the reliabilities of main (urban) and regional (rural) campus interviewers’ assessments of applicants to a Rural Physician Leadership Program (RPLP) located in the southeastern United States.
Data from RPLP applicants completing MSPIs on two campuses from 2009-2017 (n = 232) were examined in a generalizability analysis. In two separate interviews on each campus (4 total), raters independently evaluated applicants’ overall acceptability and likelihood of practicing in a rural area of the state. Results provided campus-specific and combined (composite) estimates of obtained and projected reliabilities.
The person-by-campus interaction accounted for 11% and 5% of the respective variance in interviewers’ ratings of overall applicant acceptability and likelihood of rural in-state practice, and the reliability of mean scores across the four independent interviews (each with a single, unique rater) was 0.73 and 0.82. Error variances were higher among main campus interviewers, but scores correlated highly between the two campuses.
While broadening the universe of generalization often results in decreased reliability, reliability was shown to be enhanced with the addition of regional (rural) campus interviews. As the RPLP matures, an examination of graduates’ actual practice locations should yield insights into the predictive validity of these pre-admissions assessments. More generally, research may wish to explore the conditions under which increasing the diversity of stakeholder input can be accommodated without concomitant reductions in overall reliability
