Robust automated test assembly for testlet-based tests: an illustration with the analytical reasoning section of the LSAT

In many high-stakes tests, subsets of questions (i.e., items) grouped around a common stimulus are often utilized to increase testing efficiency. These subsets of items are commonly called testlets. Since responses to items belonging to the same testlet not only depend on the test taker’s ability, but also on the correct reading, understanding, and interpretation of the stimulus, the assumption that the responses to these items are independent of one another does not always hold.\ud A mathematical model called item response theory is often applied in automated test assembly (ATA) with testlets. Testlet response theory (TRT) models have been developed to deal with dependency among items within a testlet. This report addresses some of the questions that arise in the application of TRT models to ATA. Specifically, a robust ATA method is applied. The results obtained by this method, as well as the advantages it offers, are discussed. Finally, recommendations about the use of the new method are given

Identifying critical testlet features using tree-based regression: An illustration with the Analytical Reasoning section of the LSAT

A Publication of the Law School Admission CouncilThe Law School Admission Council (LSAC) is a nonprofit corporation that provides unique, state-of-theart admission products and services to ease the admission process for law schools and their applicants worldwide. More than 200 law schools in the United States, Canada, and Australia are members of the Council and benefit from LSAC’s services

Evaluating the Construct Validity of the Norwegian Version of the Level of Personality Functioning Scale - Brief Form 2.0 in a Large Clinical Sample

The Level of Personality Functioning - Brief Form 2.0 (LPFS-BF 2.0) is a 12-item self-report questionnaire developed to gain a quick impression of the severity of personality pathology according to the DSM-5 Alternative Model for Personality Disorders (AMPD). The current study evaluated the construct validity and reliability of the Norwegian version of the LPFS-BF 2.0 in a large clinical sample (N = 1673). Dimensionality was examined using confirmatory factor analysis and bifactor analysis followed by an analysis of distinctiveness of the subscales using the proportional reduction in mean squared error (PRMSE), and the concurrent validity was examined using correlations with self-report questionnaires and clinical interviews assessing PDs according to section II of the DSM-5. Taking the findings of the dimensionality and concurrent validity results together, we found moderate to good support for the use of total scores for the Norwegian version of the LPFS-BF 2.0. We would advise against the use of subscale scores, since the subscales provided only a small amount of reliable unique variance.</p