Estimating measurement error on highly speeded tests

Despite many advances in test theory, estimating measurement error which arises from temporary changes in the person or test situation has remained relatively unchanged. Unfortunately, not only are short-term instabilities the most important source of measurement error for many traits, especially those measured from highly speeded tests; but the classical test-retest formula for estimating error is based on untenable assumptions with respect to practice effects. The current paper presents a method which gives maximum likelihood estimates of measurement error within the context of a simplex model for practice effects. The appropriateness of the model is tested for five traits, and error estimates are compared to the classical formula estimates.Whitely, Susan E.. (1979). Estimating measurement error on highly speeded tests. Retrieved from the University Digital Conservancy, https://hdl.handle.net/11299/99574

Individual inconsistency: Implications for test reliability and behavioral predictability

The nature of individual inconsistency in performance on trait measurements is an important topic in psychometrics because of its direct relevance to measurement reliability. Several studies have supported short-term inconsistency as a systematic source of variation among individuals by finding some evidence for generalizability and relationship to behavioral predictability. However, these findings are questionable, since these studies confounded change with short-term fluctuation in their response inconsistency measure. The current research separates these two sources of inconsistency in a reanalysis of the data from one major study on short-term consistency and finds little evidence for generalizability or a relationship to behavioral predictability. These results support the popular assumption that measurement error from short-term fluctuations is not due to systematic individual differences in response consistency, as well as supporting a more limited definition of the individual inconsistency construct.Whitely, Susan E.. (1978). Individual inconsistency: Implications for test reliability and behavioral predictability. Retrieved from the University Digital Conservancy, https://hdl.handle.net/11299/99455

Information structure for geometric analogies: A test theory approach

Although geometric analogies are popular items for measuring intelligence, the information processes that are involved in their solution have not been studied in a test theory context. In the current study, processing is examined by testing alternative models of information structure on geometric analogies. In contrast to the treatment of models in other studies that have appeared in the cognitive literature, the models are tested jointly as mathematical models of processing and as latent trait models of individual differences. The joint modeling was achieved by applying the one-parameter linear logistic latent trait model to predict response accuracy from information structure. The results supported the model that distinguished between spatial distortion and spatial displacement transformations, which have opposite effects on item difficulty. Further, no significant sex difference in overall accuracy or processing were observed. Implications of the results for processing mechanisms and test design are discussed.Whitely, Susan E.; Schneider, Lisa M.. (1981). Information structure for geometric analogies: A test theory approach. Retrieved from the University Digital Conservancy, https://hdl.handle.net/11299/100415

Analyses and validation of central assessment instruments of the research program TEDS-M

The TEDS-Validate project has been carried out within the research program derived from Teacher Education and Development Study: Learning to Teach Mathematics (TEDS-M). In this chapter, we describe the aim of the study, which is related to the question of whether research findings brought forward by measurement instruments to test professional competence of mathematics teachers have predictive validity for the quality of their classroom instruction and the learning gains of their students. Based on this, we question whether situation-specific skills (measured via video-based assessments) contribute to explain instructional quality and learning gains of students – in addition to the effects of professional knowledge of teachers. To answer the research questions, data was collected in Thuringia, Saxony and Hesse from 2016 to 2019 with a survey of 113 in-service teachers. They were tested using web-based instruments to capture their mathematics, mathematics pedagogical and general pedagogical knowledge as well as their noticing competencies. TEDS-Validate points out the broad applicability of the instruments for the comprehensive measurement of mathematics teachers competencies. To evaluate the effects of prospective teachers’ practical activities during their school practicum within the master study of initial teacher education, these instruments will be applied in a follow-up transfer project