Construct Validity and Cognitive Diagnostic Assessment

Cognitive diagnostic assessment (CDA) is increasingly a major focus in psychological and educational measurement. Instead of inferring a general response tendency or behavior consistency of an examinee over a target domain of measurement, diagnostic assessment results provide a detailed account of the underlying cognitive basis of the examinee's performance by mining the richer information that is afforded by specific response patterns

Problem Restructuring Processes for Ill-Structured Verbal Analogies

Copyright 1981 Psychonomic Society, Inc.Published under the name Susan E. Whitely.The processes involved in analogy solving have been an important investigative area in cognitive psychology. Although problem restructuring has been a central construct in problem solving theory, no restructuring processes have been proposed for analogical reasoning. Yet, the stimulus terms for analogies, as they appear on ability tests, are often ill-structured. That is, they are ordered in a way that does not permit direct problem comprehension. In the current study, both perceptual and semantic problem restructuring processes were hypothesized for analogy solving. The independence, stage of execution, and susceptibility to strategic control of the two processes were examined. The results from two experiments indicated that (1) ill-structured analogies are restructured during problem solving, (2) perceptual and semantic restructuring processes are independent and executed at different stages of analogy solving, and (3) both processes exhibited automaticity since repetition of analogy solution attenuated but did not eliminate either restructuring process. A model of analogical reasoning that incorporated both restructuring processes and their execution sequences was proposed. The nature and automaticity of perceptual and semantic analogy restructuring processes were discussed

Hyperarousal symptoms after traumatic and nontraumatic births

Background: Measurement is critical in postnatal posttraumatic stress disorder (PTSD) because symptoms may be influenced by normal postnatal phenomena such as physiological changes and fatigue. Objective: This study examined: (1) whether hyperarousal symptoms differ between women who have traumatic or nontraumatic births; (2) whether the construct of hyperarousal is coherent in postnatal women; and (3) whether hyperarousal symptoms are useful for identifying women who have traumatic births or PTSD. Methods: A survey of PTSD symptoms in 1,078 women recruited via the community or Internet who completed an online or paper questionnaire measuring childbirth-related PTSD symptoms between 1 and 36 months after birth. Women who had a traumatic birth as defined by DSM-IV criterion A (n = 458) were compared with women who did not have a traumatic birth (n = 591). Results: A one-factor dimension of hyperarousal was identified that included all five hyperarousal items. Diagnostic criteria of two or more hyperarousal symptoms in the previous week were reported by 75.3% of women with traumatic birth and 50.5% of women with nontraumatic births. The difference in mean hyperarousal symptoms between groups was substantial at 0.76 of a standard deviation (Hedge’s g, CI = 0.64, 0.89). A larger difference was observed between women with and without diagnostic PTSD (g = 1.64, CI 1.46, 1.81). However, receiver operating characteristic analyses showed hyperarousal symptoms have poor specificity and alternative ways of calculating symptoms did not improve this. Comparison with other PTSD symptoms found re-experiencing symptoms were most accurate at identifying women with traumatic births. Conclusions: Results suggest hyperarousal symptoms are associated with traumatic birth and are a coherent construct in postnatal women. However, they have poor specificity and should only be used as part of diagnostic criteria, not as a sole indicator

Cognitive Design Principles and the Successful Performer: A Study on Spatial Ability

An important trend in educational measurement is the use of principles of cognitive psychology to design achievement and ability test items. Many studies show that manipulating the stimulus features of items influences the processes, strategies, and knowledge structures that are involved in solution. However, little is known about how cognitive design influences individual differences. That is, does applying cognitive design principles change the background skills and abilities that are associated with successful performance? This study compared the correlates of two spatial ability tests that used the same item type but different test design principles (cognitive design versus psychometric design). The results indicated differences in factorial complexity in the two tests; specifically, the impact of verbal abilities was substantially reduced by applying the cognitive design principles

Measuring Human Intelligence with Artificial Intelligence

Adaptive item generation may be the next innovation in intelligence testing. In adaptive item generation, the optimally informative item is developed anew for the examinee during the test. Reminiscent of computer versus person chess games, the computer generates the next item based on the previous pattern of the examinee's responses. Adaptive item generation requires the merger of two lines of research, psychometric methods for adaptive testing and a cognitive analysis of items

Measuring and Validating Cognitive Modifiability as an Ability: A Study in the Spatial Domain

Measuring cognitive modifiability from the responsiveness of an individual's performance to intervention has long been viewed (e.g., Dearborne, 1921) as an alternative to traditional (static) ability measurement. Currently, dynamic testing in which cues or instruction are presented with ability test items, is a popular method for assessing cognitive modifiability. Despite the long-standing interest, however, little data exists to support the validity of cognitive modifiability measures in any ability domain. Several special methodological difficulties have limited validity studies, including psychometric problems in measuring modifiability (i.e., as change), lack of appropriate validation criteria, and difficulty in linking modifiability to cognitive theory. In this article, relatively new developments for solving the validation problems are applied to measuring and validating spatial modifiability. Criterion-related validity for predicting learning in an applied knowledge domain, as well as construct validity, is supported

Models, Meanings and Misunderstandings: Some Issues in Applying Rasch's Theory

Published under the name Susan E. Whitely.Wright (1977) shows that a debate is developing between those who strongly advocate use of the Rasch model and those who have certain reservations about the extent to which the model meets some traditional concerns in trait measurement. In an earlier article, Whitely and Dawis (1974) presented the Rasch model in the context of least squares estimation, and noted some features that may limit the utility of the model in test development. Wright (1977) questions several of the specific interpretations and conclusions that were given in the earlier article. The current article is a response to those questions

The Second Century of Ability Testing: Some Predictions and Speculations

The seventh annual William H. Angoff Memorial Lecture was presented at Educational Testing Service, Princeton, New Jersey, on January 11, 2001. Copyright © 2003 by Educational Testing Service. All rights reserved. Educational Testing Service is an Affirmative Action/Equal Opportunity Employer. Educational Testing Service, ETS, and the ETS logos are registered trademarks of Educational Testing Service.Dazzling changes in many areas, such as technology and communications, marked the 20th century—the first century of ability testing. Predictions about the second century of testing seem difficult in such a context. Yet, looking back to the turn of the last century, Kirkpatrick (1900) in his APA presidential address presented fundamental desiderata for ability testing (normative age standards, emphasis on culture-general tasks, simultaneous measurement of many persons, and adult ability measurement) that, in fact, guides major testing research even today. An overview of the last century shows that most fundamental principles in psychometrics and testing were available by 1930. With a few notable exceptions, the remainder of the last century of testing was devoted to applying or refining these principles. I predict that the same pattern will occur in this century of testing. Further developments in model-based measurement and cognitive psychology principles in testing, intermingled with technology, will guide ability testing throughout the next century. These changes, which I will elaborate in detail, include fundamental changes in test development procedures, the nature of the measuring tasks, aspects of ability that are measured, and types of interpretations given to ability

Multicomponent Latent Trait Models for Complex Tasks

Published in the Journal of Applied Measurement, 7(3), 335-550, copyright 2006.Contemporary views on cognitive theory (e.g., Sternberg and Perez, 2005) regard typical measurement tasks, such as ability and achievement test items, multidimensional, rather than unidimensional. Assessing the levels and the sources of multidimensionality in an item domain is important for item selection as well as for item revision and development. In this paper, multicomponent latent trait models (MLTM) and traditional multidimensional item response theory models are described mathematically and compared for the nature of the dimensions that can be estimated. Then, sonic applications are presented to provide examples of MLTM. Last, practical estimation procedures are described, along with syntax, for the estimation of MLTM and a related model