Operationalizing item difficulty modeling in a medical certification context

This research study modeled item difficulty in general pediatric test items using content, cognitive complexity, linguistic, and text-based variables. The research first presents an introduction which addresses the current shortcomings found in item development and alternative methods such as principled assessment design which aim to address those shortcomings. Next, a review of the literature is presented which addresses traditional item development, item development using cognitive demands, item difficulty modeling, and the Coh-Metrix (Grasser et al., 2004) linguistic tool. The methods section outlines how content, cognitive, linguistic, and text-based variables were defined and coded using both subject matter experts (SMEs) and Coh-Metrix web-based software. The methods section goes on to outline the backward multiple regression analysis which was conducted to determine the proportion of variance in Rasch item difficulty accounted for by the defined variables and a study which can be used to demonstrate the impact of the current findings on examinee ability calibration. The results of the study demonstrate an operationalizable process for determining item difficulty variables. The results also found that Rasch item difficulty was significantly predicted by five item difficulty variables which accounted for .324 variance in Rasch item difficulty. The research concludes with a discussion of the findings, including steps that can be taken in future studies to build upon the current research and results