A Mixture Model for Random Responding Behavior in Forced-Choice Noncognitive Assessment:Implication and Application in Organizational Research

For various reasons, respondents to forced-choice assessments (typically used for noncognitive psychological constructs) may respond randomly to individual items due to indecision or globally due to disengagement. Thus, random responding is a complex source of measurement bias and threatens the reliability of forced-choice assessments, which are essential in high-stakes organizational testing scenarios, such as hiring decisions. The traditional measurement models rely heavily on nonrandom, construct-relevant responses to yield accurate parameter estimates. When survey data contain many random responses, fitting traditional models may deliver biased results, which could attenuate measurement reliability. This study presents a new forced-choice measure-based mixture item response theory model (called M-TCIR) for simultaneously modeling normal and random responses (distinguishing completely and incompletely random). The feasibility of the M-TCIR was investigated via two Monte Carlo simulation studies. In addition, one empirical dataset was analyzed to illustrate the applicability of the M-TCIR in practice. The results revealed that most model parameters were adequately recovered, and the M-TCIR was a viable alternative to model both aberrant and normal responses with high efficiency.</p

Psychometric Properties of the SAS, BAI, and S-AI in Chinese University Students

Three widely-used self-report anxiety scales, including the Self-Rating Anxiety Scale (SAS), the Beck Anxiety Inventory (BAI), and the State Anxiety Inventory (S-AI), were used to simultaneously compare the psychometric properties via an item response theory (IRT) model with Chinese university students as the sample. Although these scales were probably to measure the same underlying construct, namely, anxiety, their psychometric properties were different. Results showed that the BAI’s measurement error was fewer than that of the other scales, with their anxiety severity ranging approximately from the 0.8 standard deviations below the mean to 3 standard deviations above the mean, while the S-AI’s measurement error was fewer than that of the other degrees of anxiety. The S-AI provided more information than the other scales when the student’s scale was less than approximately 0.8 standard deviations below the mean of anxiety severity. In general, the BAI showed better, for it provided more information than the other scales at the broadest range of anxiety severity. The SAS provided less information than the other scales at all anxiety severity range. In conclusion, BAI shows good psychometric quality. Finally, the three instruments were combined on a common scale by using IRT model and a conversion table was provided so as to achieve the transformation of each scale score

Theorems and Methods of a Complete Q Matrix With Attribute Hierarchies Under Restricted Q-Matrix Design

The design of test Q matrix can directly influence the classification accuracy of a cognitive diagnostic assessment. In this paper, we focus on Q matrix design when attribute hierarchies are known prior to test development. A complete Q matrix design is proposed and theorems are presented to demonstrate that it is a necessary and sufficient condition to guarantee the identifiability of ideal response patterns. A simulation study is also conducted to detect the effects of the proposed design on a family of conjunctive diagnostic models. The results revealed that the proposed Q matrix design is the key condition for guaranteeing classification accuracy. When only one type of item pattern in R matrix is missing from the associated test Q matrix, the related attribute-wise agreement rate will decrease dramatically. When the entire R matrix is missing, both the pattern-wise and attribute-wise agreement rates will decrease sharply. This indicates that the proposed procedures for complete Q matrix design with attribute hierarchies can serve as guidelines for test blueprint development prior to item writing in a cognitive diagnostic assessment

Structure of Arabic Scale of Death Anxiety With Chinese College Students: A Bifactor Approach

The Arabic Scale of Death Anxiety (ASDA), as one of the most widely used measures of death anxiety (DA), has increasingly been applied in many studies. However, the structures derived from different studies are highly inconsistent. In this study, both traditional and novel (bifactor) modeling approaches were used, to investigate the most optimal structure of the ASDA in a sample of 984 Chinese college students. After a series of comparisons, the results showed that the bifactor model, with a dominant general DA factor and three distinct sub-dimensions, was the most optimal measurement structure, and measurement invariance of this bifactor model between sexes was also confirmed. Based on the implications of this bifactor model, the discussion was focused mainly on whether distinct dimensions should be interpreted or not. Some strengths and limitations of the study were also discussed at the end of the paper

Development of a Computerized Adaptive Testing for Internet Addiction

Internet addiction disorder has become one of the most popular forms of addiction in psychological and behavioral areas, and measuring it is growing increasingly important in practice. This study aimed to develop a computerized adaptive testing to measure and assess internet addiction (CAT-IA) efficiently. Four standardized scales were used to build the original item bank. A total of 59 polytomously scored items were finally chosen after excluding 42 items for failing the psychometric evaluation. For the final 59-item bank of CAT-IA, two simulation studies were conducted to investigate the psychometric properties, efficiency, reliability, concurrent validity, and predictive validity of CAT-IA under different stopping rules. The results showed that (1) the final 59 items met IRT assumptions, had high discrimination, showed good item-model fit, and were without DIF; and (2) the CAT-IA not only had high measurement accuracy in psychometric properties but also sufficient efficiency, reliability, concurrent validity, and predictive validity. The impact and limitations of CAT-IA were discussed, and several suggestions for future research were provided

Development and Validation of an Item Bank for Depression Screening in the Chinese Population Using Computer Adaptive Testing: A Simulation Study

With the increasing prevalence of depression, creating a simple and precise tool for measuring depression is becoming more important. This study developed a computer adaptive testing for depression (CAT-Depression) from a Chinese sample. The depression item bank was constructed from a sample of 1,135 participants with or without depression using the Graded Response Model (GRM; Samejima, 1969). The final depression item bank with strict unidimensionality comprised 68 items, which had local independence, good item-fit, high discrimination, no differential item functioning (DIF), and each item measured at least one symptom of diagnostic criteria for depression in ICD-10. In addition, the mean IRT discrimination of the item bank reached 1.784, which clearly showed that the item bank of CAT-Depression was high-quality. Moreover, a simulation CAT study with real response data was conducted to investigate the characteristics, marginal reliability, criterion-related validity, and predictive utility (sensitivity and specificity) of CAT-Depression. The results revealed that the proposed CAT-Depression had acceptable and reasonable marginal reliability, criterion-related validity, and sensitivity and specificity

A New Measurement of Internet Addiction Using Diagnostic Classification Models

To obtain accurate, valid, and rich information from the questionnaires for internet addiction, a diagnostic classification test for internet addiction (the DCT-IA) was developed using diagnostic classification models (DCMs), a cutting-edge psychometric theory, based on DSM-5. A calibration sample and a validation sample were recruited in this study to calibrate the item parameters of the DCT-IA and to examine the sensitivity and specificity. The DCT-IA had high reliability and validity based on both CTT and DCMs, and had a sensitivity of 0.935 and a specificity of 0.817 with AUC = 0.919. More important, different from traditional questionnaires, the DCT-IA can simultaneously provide general-level diagnostic information and the detailed symptom criteria-level information about the posterior probability of satisfying each symptom criterion in DMS-5 for each patient, which gives insight into tailoring individual-specific treatments for internet addiction

Data_Sheet_1_Theorems and Methods of a Complete Q Matrix With Attribute Hierarchies Under Restricted Q-Matrix Design.DOCX

<p>The design of test Q matrix can directly influence the classification accuracy of a cognitive diagnostic assessment. In this paper, we focus on Q matrix design when attribute hierarchies are known prior to test development. A complete Q matrix design is proposed and theorems are presented to demonstrate that it is a necessary and sufficient condition to guarantee the identifiability of ideal response patterns. A simulation study is also conducted to detect the effects of the proposed design on a family of conjunctive diagnostic models. The results revealed that the proposed Q matrix design is the key condition for guaranteeing classification accuracy. When only one type of item pattern in R matrix is missing from the associated test Q matrix, the related attribute-wise agreement rate will decrease dramatically. When the entire R matrix is missing, both the pattern-wise and attribute-wise agreement rates will decrease sharply. This indicates that the proposed procedures for complete Q matrix design with attribute hierarchies can serve as guidelines for test blueprint development prior to item writing in a cognitive diagnostic assessment.</p

Data_Sheet_2_Theorems and Methods of a Complete Q Matrix With Attribute Hierarchies Under Restricted Q-Matrix Design.docx

<p>The design of test Q matrix can directly influence the classification accuracy of a cognitive diagnostic assessment. In this paper, we focus on Q matrix design when attribute hierarchies are known prior to test development. A complete Q matrix design is proposed and theorems are presented to demonstrate that it is a necessary and sufficient condition to guarantee the identifiability of ideal response patterns. A simulation study is also conducted to detect the effects of the proposed design on a family of conjunctive diagnostic models. The results revealed that the proposed Q matrix design is the key condition for guaranteeing classification accuracy. When only one type of item pattern in R matrix is missing from the associated test Q matrix, the related attribute-wise agreement rate will decrease dramatically. When the entire R matrix is missing, both the pattern-wise and attribute-wise agreement rates will decrease sharply. This indicates that the proposed procedures for complete Q matrix design with attribute hierarchies can serve as guidelines for test blueprint development prior to item writing in a cognitive diagnostic assessment.</p

Supplementary Material, Online_Appendix – Item Selection Methods in Multidimensional Computerized Adaptive Testing With Polytomously Scored Items

<p>Supplementary Material, Online_Appendix for Item Selection Methods in Multidimensional Computerized Adaptive Testing With Polytomously Scored Items by Dongbo Tu, Yuting Han, Yan Cai and Xuliang Gao in Applied Psychological Measurement</p