Pseudo-likelihood estimation of multidimensional polytomous item response theory models

Log-multiplicative association (LMA) models, special cases of log-linear models, can be used as multidimensional item response theory (MIRT) models for polytomous items (Anderson, Verkuilen and Peyton, 2010; Anderson, 2013). LMA models do not require numerical integration for their estimation nor do they require assumptions regarding the marginal distribution of the latent variables. However, maximum likelihood estimation (MLE) of LMA models requires iteratively computing fitted values for all possible response patterns. Standard estimation methods for large numbers of items fail because the number of possible response patterns increases exponentially as the number of items and response options per item increase. In this study, a new algorithm is proposed to solve this estimation problem. Anderson, Li and Vermunt (2007) proposed using pseudo-likelihood estimation (PLE); however, their solution only applies to models in the Rasch family, which exploits the relationship between log-linear and logistic regression models. Their method is extended to more general models by adding an additional step that estimates slope (item discrimination) parameters for the latent variables. The new algorithm has two basic steps and simplifies for special cases. In Step 1, a (multinomial) logistic regression model is fit by MLE to one item using rest-scores as an explanatory variable to get new estimates of item slopes that are used in the rest-score for the next item. This process is repeated for each item until all item slopes have been up-dated. Step 2 involves fitting a single conditional logistic regression model for a data set formed by stacking the conditional logistic regressions for each item. This yields new estimates of location (item difficulty) parameters and the covariance matrix for the latent variables. Steps 1 and 2 are repeated until all parameter estimates converge. The results of simulation and empirical studies with real data show that the proposed algorithm successfully estimates parameters in more general LMA models with both location and slope parameters as MIRT models

Multidimensional item response theory: A SAS MDIRT macro and empirical study of PIAT math test.

Even though unidimensional item response theory (IRT) provides a better framework for practical test settings than classical test theory (CTT), theoretical and empirical evidence shows that most response data violate the assumption of unidimensionality. There are several computer programs dedicated to estimating parameters based on the multidimensional perspective (MIRT). However, their accessibility is still costly, and they are not easy to use. In this paper, we present a SAS macro called MDIRT-FIT, to increase accessibility to the benefits obtained from this recent measurement theory development. The program is developed to estimate parameters based on a compensatory multidimensional item response theory (MIRT) model for dichotomous data. The full information item factor analysis model with an EM algorithm suggested in Bock & Aitken (1988) is implemented in the SAS programs. The estimation program written in SAS/IMLRTM provides both parameters of MIRT and parameters of the factor analysis model with their associated standard errors and overall model fit statistics. The maximum number of latent traits that can be estimated with this program is limited to five latent dimensions because of both computational burden and practical sufficiency. The accuracy and stability of the SAS macro is examined by utilizing simulated data of examinees' responses. The PIAT math test, a subset of the Peabody Individual Achievement Test, was examined to validate the comparability of the SAS macro to TESTFACT which is widely used for estimating parameters of MIRT models

Better assessment of physical function: item improvement is neglected but essential

INTRODUCTION: Physical function is a key component of patient-reported outcome (PRO) assessment in rheumatology. Modern psychometric methods, such as Item Response Theory (IRT) and Computerized Adaptive Testing, can materially improve measurement precision at the item level. We present the qualitative and quantitative item-evaluation process for developing the Patient Reported Outcomes Measurement Information System (PROMIS) Physical Function item bank. METHODS: The process was stepwise: we searched extensively to identify extant Physical Function items and then classified and selectively reduced the item pool. We evaluated retained items for content, clarity, relevance and comprehension, reading level, and translation ease by experts and patient surveys, focus groups, and cognitive interviews. We then assessed items by using classic test theory and IRT, used confirmatory factor analyses to estimate item parameters, and graded response modeling for parameter estimation. We retained the 20 Legacy (original) Health Assessment Questionnaire Disability Index (HAQ-DI) and the 10 SF-36\u27s PF-10 items for comparison. Subjects were from rheumatoid arthritis, osteoarthritis, and healthy aging cohorts (n = 1,100) and a national Internet sample of 21,133 subjects. RESULTS: We identified 1,860 items. After qualitative and quantitative evaluation, 124 newly developed PROMIS items composed the PROMIS item bank, which included revised Legacy items with good fit that met IRT model assumptions. Results showed that the clearest and best-understood items were simple, in the present tense, and straightforward. Basic tasks (like dressing) were more relevant and important versus complex ones (like dancing). Revised HAQ-DI and PF-10 items with five response options had higher item-information content than did comparable original Legacy items with fewer response options. IRT analyses showed that the Physical Function domain satisfied general criteria for unidimensionality with one-, two-, three-, and four-factor models having comparable model fits. Correlations between factors in the test data sets were \u3e 0.90. CONCLUSIONS: Item improvement must underlie attempts to improve outcome assessment. The clear, personally important and relevant, ability-framed items in the PROMIS Physical Function item bank perform well in PRO assessment. They will benefit from further study and application in a wider variety of rheumatic diseases in diverse clinical groups, including those at the extremes of physical functioning, and in different administration modes

Development and psychometric analysis of the PROMIS pain behavior item bank

The measurement of pain behavior is a key component of the assessment of persons with chronic pain; however few self-reported pain behavior instruments have been developed. We developed a pain behavior item bank as part of the Patient Reported Outcome Measurement Information System (PROMIS). For the Wave I testing, because of the large number of PROMIS items, a complex sampling approach was used where participants were randomly assigned to either respond to two full item banks or to multiple 7-item blocks of items. A web-based survey was designed and completed by 15,528 members of the general population and 967 individuals with different types of chronic pain. Item response theory (IRT) analysis models were used to evaluate item characteristics and to scale both items and individuals on the pain behavior domain. The pain behavior item bank demonstrated good fit to a unidimensional model (Comparative Fit Index = 0.94). Several iterations of IRT analyses resulted in a final 39 item pain behavior bank, and different IRT models were fit to the total sample and to those participants who experienced some pain. The results indicated that these items demonstrated good coverage of the pain behavior construct. Pain behavior scores were strongly related to pain intensity and moderately related to self-reported general health status. Mean pain behavior scores varied significantly by groups based on pain severity and general health status. The PROMIS pain behavior item bank can be used to develop static short-form and dynamic measures of pain behavior for clinical studies

Risk Compensation Is Not Associated with Male Circumcision in Kisumu, Kenya: A Multi-Faceted Assessment of Men Enrolled in a Randomized Controlled Trial

Three randomized controlled trials (RCTs) have confirmed that male circumcision (MC) significantly reduces acquisition of HIV-1 infection among men. The objective of this study was to perform a comprehensive, prospective evaluation of risk compensation, comparing circumcised versus uncircumcised controls in a sample of RCT participants.Between March 2004 and September 2005, we systematically recruited men enrolled in a RCT of MC in Kenya. Detailed sexual histories were taken using a modified Timeline Followback approach at baseline, 6, and 12 months. Participants provided permission to obtain circumcision status and laboratory results from the RCT. We evaluated circumcised and uncircumcised men's sexual behavior using an 18-item risk propensity score and acquisition of incident infections of gonorrhea, chlamydia, and trichomoniasis. Of 1780 eligible RCT participants, 1319 enrolled (response rate = 74%). At the baseline RCT visit, men who enrolled in the sub-study reported the same sexual behaviors as men who did not. We found a significant reduction in sexual risk behavior among both circumcised and uncircumcised men from baseline to 6 (p<0.01) and 12 (p = 0.05) months post-enrollment. Longitudinal analyses indicated no statistically significant differences between sexual risk propensity scores or in incident infections of gonorrhea, chlamydia, and trichomoniasis between circumcised and uncircumcised men. These results are based on the most comprehensive analysis of risk compensation yet done.In the context of a RCT, circumcision did not result in increased HIV risk behavior. Continued monitoring and evaluation of risk compensation associated with circumcision is needed as evidence supporting its' efficacy is disseminated and MC is widely promoted for HIV prevention

Evaluating the Impact of Construct Shift on Item Parameter Invariance, Test Equating and Proficiency Estimates

Common Core State Standards in English Language Arts and Mathematics at grades K to 12 were introduced in 2009 and at one time had been accepted by 45 of the states in the U.S. The new standards have created national curricula in these two subject areas. Along with this reform, new assessment systems have been developed too. Many of these new tests are showing signs of being more multidimensional than the tests they were replacing because of the use of new item formats, and the assessment of higher level thinking skills and various performance skills. In the short term at least, the new testing programs will still be using unidimensional IRT models because multidimensional IRT models are not well developed for wide-scale operational use. Test equating and proficiency estimates are likely to be influenced due to the violation of the unidimensionality assumption in the test data, which posed a threat to test fairness and the validity of score interpretations. Looking at the consequences of unidimensional models with multidimensional data is an important problem to investigate. In the first study, the potential effect of multidimensionality on item parameter invariance and IRT model fit at the item level was examined. The finding highlighted the fact that very modest changes in construct shift in the operational items of a test would not be problematic. As an extension of the first study, the second study introduced multidimensionality to a much greater magnitude and evaluated impact on IRT equating and proficiency estimates. It was found when construct shift occurred in the operational items, the impact was minimal in most aspects. Whereas, when construct shift occurred in the anchor and when these equating items were included in the total test, the influence of construct shift was quite substantial. The present research has provided evidence of consequences caused by construct shift as a function of the correlations between pairs of dimensions, the amount of shift, and where the shift occurs (equating or operational items). In general, it was found that construct shift had increasing impact as the amount of shift increased and the correlations between dimensions decreased

Detecting Local Item Dependence in Polytomous Adaptive Data

A rapidly expanding arena for item response theory (IRT) is in attitudinal and health-outcomes survey applications, often with polytomous items. In particular, there is interest in computer adaptive testing (CAT). Meeting model assumptions is necessary to realize the benefits of IRT in this setting, however. Although initial investigations of local item dependence (LID) have been studied both for polytomous items in fixed-form settings and for dichotomous items in CAT settings, there have been no publications applying LID detection methodology to polytomous items in CAT despite its central importance to these applications. The research documented herein investigates the extension of widely used methods of LID detection, Yen's Q3 statistic and Pearson's Statistic X2, in this context, via a simulation study. The simulation design and results are contextualized throughout with a real item bank and data set of this type from the Patient-Reported Outcomes Measurement Information System (PROMIS)