Robust automated test assembly for testlet-based tests: an illustration with the analytical reasoning section of the LSAT

In many high-stakes tests, subsets of questions (i.e., items) grouped around a common stimulus are often utilized to increase testing efficiency. These subsets of items are commonly called testlets. Since responses to items belonging to the same testlet not only depend on the test taker’s ability, but also on the correct reading, understanding, and interpretation of the stimulus, the assumption that the responses to these items are independent of one another does not always hold.\ud A mathematical model called item response theory is often applied in automated test assembly (ATA) with testlets. Testlet response theory (TRT) models have been developed to deal with dependency among items within a testlet. This report addresses some of the questions that arise in the application of TRT models to ATA. Specifically, a robust ATA method is applied. The results obtained by this method, as well as the advantages it offers, are discussed. Finally, recommendations about the use of the new method are given

Identifying critical testlet features using tree-based regression: An illustration with the Analytical Reasoning section of the LSAT

A Publication of the Law School Admission CouncilThe Law School Admission Council (LSAC) is a nonprofit corporation that provides unique, state-of-theart admission products and services to ease the admission process for law schools and their applicants worldwide. More than 200 law schools in the United States, Canada, and Australia are members of the Council and benefit from LSAC’s services

The content validity of the Behavioural Appraisal Scales in people with profound intellectual and multiple disabilities:A Delphi study

The Behavioural Appraisal Scales (BAS) were developed in The Netherlands to assess functional abilities of people with profound intellectual and multiple disabilities (PIMD). It is recommended that further studies examine the psychometric properties of the BAS. The aim of this study was to evaluate and, if necessary, improve the content validity of the BAS. A two-round Delphi study was used. In Round 1, parents, researchers and practice professionals indicated for each item whether they would retain, adapt or remove it, explaining their answers. The BAS were adapted based on this feedback. In Round 2, participants were given the adapted version and asked to indicate for each item whether they wanted to retain or remove it, again providing an explanation. Feedback was analysed by calculating the percentage of participants who wanted to retain, adapt or remove a certain item. Content analysis was used to analyse participants' explanations. In Round 1, more than 20% of the participants wanted to remove 11 from the 122 items, and wanted to retain or adapt the remaining items. Seven categories of adaptations emerged: (1) feasibility, (2) the construct the item measures, (3) applicability, (4) similarity among items, (5) splitting items, (6) reassignment and (7) the professionals who score the items. In Round 2, for all items >80% of the participants indicated they wanted to retain the item. Experts felt that the content validity of the BAS could be improved. After the adaptations, the content validity was evaluated as good. Further studies are needed to evaluate whether the new or adapted items are not too easy or too difficult, whether information from different informant groups results in being scored differently, and to evaluate possible differential item functioning

Using the Three-Step Test Interview to understand how patients perceive the St. George’s Respiratory Questionnaire for COPD patients (SGRQ-C)

Purpose The aim of this study was to assess the experiences of patients with chronic obstructive pulmonary disease (COPD) while they were completing the St. George’s Respiratory Questionnaire for COPD patients (SGRQ-C), using qualitative research methods. Methods Twenty Dutch COPD patients were recruited through pulmonary physicians [13 women; mean age = 63.3 years (SD = 11.4)]. A trained interviewer applied the Three-Step Test Interview which allowed the interviewer to follow the thought process of the patient filling out the SGRQ-C. The official Dutch translation of the SGRQ-C was used. Results Patients missed a recall period for the Symptoms subscale; were uncertain how to interpret specific words and phrases like “good days”, “games”, and “housework”; were confused by long-item stems that included a list of activities; and were frustrated by the dichotomous format used for the majority of SGRQ-C items (true/false). Conclusions Overall, patients were satisfied with the SGRQ-C. Nevertheless, making minor adjustments could further increase its quality. This includes reintroducing a recall period in the first set of items such as used in the previous version and splitting up items consisting of multiple activities. Furthermore, we recommend using the same response format (4 or 5 response categories) for all items

A systematic review of measurement uncertainty visualizations in the context of standardized assessments

This systematic review summarized findings of 29 studies evaluating visual presentation formats appropriate for communicating measurement uncertainty associated with standardized clinical assessment instruments. Studies were identified through systematic searches of multiple databases (Medline, Embase, PsycInfo, ERIC, Scopus, and Web of Science). Strikingly, we found no studies which were conducted using samples of clinicians and included clinical decision-making scenarios. Included studies did however find that providing participants with information about measurement uncertainty may increase awareness of uncertainty and promote more optimal decision making. Formats which visualize the shape of the underlying probability distribution were found to promote more accurate probability estimation and appropriate interpretations of the underlying probability distribution shape. However, participants in the included studies did not seem to benefit from the additional information provided by such plots during decision-making tasks. Further explorations into how presentations of measurement uncertainty impact clinical decision making are needed to examine whether findings of the included studies generalize to clinician populations. This review provides an important overview of pitfalls associated with formats commonly used to communicate measurement uncertainty in clinical assessment instruments, and a potential starting point for further explorations into promising alternatives. Finally, our review offers specific recommendations on how remaining research questions might be addressed.</p

Identifying key domains of health-related quality of life for patients with Chronic Obstructive Pulmonary Disease: the patient perspective

Background Numerous instruments are available to measure health-related quality of life (HRQoL) in patients with Chronic Obstructive Pulmonary Disease (COPD), covering a wide array of domains ranging from symptoms such as dyspnea, cough and wheezing, to social and emotional functioning. Currently no information or guide is available yet to aid the selection of domains for a particular study or disease population. The aim of this paper is to identify which domains of HRQoL are most important with respect to COPD, from the patient perspective. Methods Twenty-one Dutch patients with COPD were asked to describe important domains impacted by COPD freely; second, they were presented with cues (domains from the Patient-Reported Outcomes Measurement Information System (PROMIS) framework) and were asked to select the domains that were most relevant to them. During the interview, the patients were asked to indicate in which way the selected domains impact their lives. Both the answers to the open question, and the patient statements motivating nomination of PROMIS domains were coded into themes. Results The most relevant (sub)domains of HRQoL for patients with COPD were: physical health (fatigue, physical functioning), social health (instrumental support, ability to participate in social roles and activities, companionship, and emotional support), and coping with COPD. Conclusion We identified which domains of HRQoL are most important to patients with COPD. One of these (coping with COPD) is not explicitly covered by PROMIS, or by traditional questionnaires that are used to measure HRQoL in COPD