From byproduct to design factor. On validating the interpretation of process indicators based on log data

International large-scale assessments such as PISA or PIAAC have started to provide public or scientific use files for log data; that is, events, event-related attributes and timestamps of test-takers\u27 interactions with the assessment system. Log data and the process indicators derived from it can be used for many purposes. However, the intended uses and interpretations of process indicators require validation, which here means a theoretical and/or empirical justification that inferences about (latent) attributes of the test-taker\u27s work process are valid. This article reviews and synthesizes measurement concepts from various areas, including the standard assessment paradigm, the continuous assessment approach, the evidence-centered design (ECD) framework, and test validation. Based on this synthesis, we address the questions of how to ensure the valid interpretation of process indicators by means of an evidence-centered design of the task situation, and how to empirically challenge the intended interpretation of process indicators by developing and implementing correlational and/or experimental validation strategies. For this purpose, we explicate the process of reasoning from log data to low-level features and process indicators as the outcome of evidence identification. In this process, contextualizing information from log data is essential in order to reduce interpretative ambiguities regarding the derived process indicators. Finally, we show that empirical validation strategies can be adapted from classical approaches investigating the nomothetic span and construct representation. Two worked examples illustrate possible validation strategies for the design phase of measurements and their empirical evaluation. (DIPF/Orig.

Controlling speed in component skills of reading improves the explanation of reading comprehension

Efficiency in reading component skills is crucial for reading comprehension, as efficient subprocesses do not extensively consume limited cognitive resources, making them available for comprehension processes. Cognitive efficiency is typically measured with speeded tests of relatively easy items. Observed responses and response times indicate the latent variables of ability and speed. Interpreting only ability or speed as efficiency may be misleading because there is a within-person dependency between both variables (speed-ability tradeoff [SAT]). Therefore, the present study measures efficiency as ability conditional on speed by controlling speed experimentally with item-level time limits. The proposed timed ability measures of reading component skills are expected to have a clearer interpretation in terms of efficiency and to be better predictors for reading comprehension. To support this claim, this study investigates two component skills, visual word recognition and sentence-level semantic integration (sentence reading), to understand how differences in ability in a timed condition are related to differences in ability and speed in a traditional untimed condition. Moreover, untimed and timed reading component skill measures were used to explain reading comprehension. A German subsample from Programme for International Student Assessment (PISA) 2012 completed the reading component skills tasks with and without item-level time limits and PISA reading tasks. The results showed that timed ability is only moderately related to untimed ability. Furthermore, timed ability measures proved to be stronger predictors of sentence-level and text-level reading comprehension than the corresponding untimed ability and speed measures, although using untimed ability and speed jointly as predictors increased the amount of explained variance. (DIPF/Orig.

From byproduct to design factor: on validating the interpretation of process indicators based on log data

International large-scale assessments such as PISA or PIAAC have started to provide public or scientific use files for log data; that is, events, event-related attributes and timestamps of test-takers’ interactions with the assessment system. Log data and the process indicators derived from it can be used for many purposes. However, the intended uses and interpretations of process indicators require validation, which here means a theoretical and/or empirical justification that inferences about (latent) attributes of the test-taker’s work process are valid. This article reviews and synthesizes measurement concepts from various areas, including the standard assessment paradigm, the continuous assessment approach, the evidence-centered design (ECD) framework, and test validation. Based on this synthesis, we address the questions of how to ensure the valid interpretation of process indicators by means of an evidence-centered design of the task situation, and how to empirically challenge the intended interpretation of process indicators by developing and implementing correlational and/or experimental validation strategies. For this purpose, we explicate the process of reasoning from log data to low-level features and process indicators as the outcome of evidence identification. In this process, contextualizing information from log data is essential in order to reduce interpretative ambiguities regarding the derived process indicators. Finally, we show that empirical validation strategies can be adapted from classical approaches investigating the nomothetic span and construct representation. Two worked examples illustrate possible validation strategies for the design phase of measurements and their empirical evaluation

Using Response Times for Joint Modeling of Careless Responding and Attentive Response Styles

Questionnaires are by far the most common tool for measuring noncognitive constructs in psychology and educational sciences. Response bias may pose an additional source of variation between respondents that threatens validity of conclusions drawn from questionnaire data. We present a mixture modeling approach that leverages response time data from computer-administered questionnaires for the joint identification and modeling of two commonly encountered response bias that, so far, have only been modeled separately—careless and insufficient effort responding and response styles (RS) in attentive answering. Using empirical data from the Programme for International Student Assessment 2015 background questionnaire and the case of extreme RS as an example, we illustrate how the proposed approach supports gaining a more nuanced understanding of response behavior as well as how neglecting either type of response bias may impact conclusions on respondents’ content trait levels as well as on their displayed response behavior. We further contrast the proposed approach against a more heuristic two-step procedure that first eliminates presumed careless respondents from the data and subsequently applies model-based approaches accommodating RS. To investigate the trustworthiness of results obtained in the empirical application, we conduct a parameter recovery study

Automated and controlled processes in comprehending multiple documents

The study investigates automated and controlled cognitive processes that occur when university students read multiple documents (MDs). We examined data of 401 students dealing with two MD sets in a digital environment. Performance was assessed through several comprehension questions. Recorded log data gave indications about students\u27 time allocation, corroboration, and sourcing. Independent measures were used for reading speed to tap the effects of automatic processing and for working memory and single-text reading comprehension to tap effects of controlled processing, with working memory considered the mental capacity for performing controlled processing. We found that faster readers completed the MD tasks faster and showed more corroboration behavior. At the same time, students skilled in comprehension allocated more time to processing MD tasks and were more likely to show MD-specific behaviors of corroboration and sourcing. Students\u27 success in MD tasks was predicted by reading speed and working memory, with the effect of working memory being mediated by single-text comprehension. Behavioral indicators contributed independently in predicting students\u27 MD comprehension. Results suggest that reading MDs resembles a problem-solving situation where students need to engage in controlled, non-routine processing to build up a comprehensive representation of MDs and benefit from highly automated, lower-level reading processes. (DIPF/Orig.

Invariance of the Response Processes Between Gender and Modes in an Assessment of Reading

In this paper, we developed a method to extract item-level response times from log data that are available in computer-based assessments (CBA) and paper-based assessments (PBA) with digital pens. Based on response times that were extracted using only time differences between responses, we used the bivariate generalized linear IRT model framework (B-GLIRT, [1]) to investigate response times as indicators for response processes. A parameterization that includes an interaction between the latent speed factor and the latent ability factor in the cross-relation function was found to fit the data best in CBA and PBA. Data were collected with a within-subject design in a national add-on study to PISA 2012 administering two clusters of PISA 2009 reading units. After investigating the invariance of the measurement models for ability and speed between boys and girls, we found the expected gender effect in reading ability to coincide with a gender effect in speed in CBA. Taking this result as indication for the validity of the time measures extracted from time differences between responses, we analyzed the PBA data and found the same gender effects for ability and speed. Analyzing PBA and CBA data together we identified the ability mode effect as the latent difference between reading measured in CBA and PBA. Similar to the gender effect the mode effect in ability was observed together with a difference in the latent speed between modes. However, while the relationship between speed and ability is identical for boys and girls we found hints for mode differences in the estimated parameters of the cross-relation function used in the B-GLIRT model

Entwicklung und Skalierung eines Tests zur Erfassung des Verständnisses multipler Dokumente von Studierenden

Das Verständnis multipler Dokumente (Multiple Document Comprehension, MDC) wird als Fähigkeit verstanden, aus verschiedenen Informationsquellen eine integrierte Repräsentation eines inhaltlichen Gegenstandsbereichs zu konstruieren. Als solche ist sie sowohl für die erfolgreiche Bewältigung eines Studiums als auch für gesellschaftliche Partizipation eine wichtige Kompetenz. Bislang gibt es jedoch kein etabliertes Diagnostikum in diesem Bereich. Um diese Lücke zu schließen, wurde ein Test entwickelt, der vier zentrale kognitive Anforderungen von MDC abdeckt und auf Basis der Daten von 310 Studierenden sozial- und geisteswissenschaftlicher Fächer überprüft wurde. Die im MDC-Test gemessene Kompetenz erwies sich als eindimensional. Der MDC-Testwert wies theoriekonforme Zusammenhänge mit der Abiturnote, dem Studienabschnitt und der Leistung in einer Essay-Aufgabe auf. Insgesamt liefern die Ergebnisse empirische Belege dafür, dass der Testwert aus dem MDC-Test die fächerübergreifende Fähigkeit von Studierenden wiedergibt, multiple Dokumente zu verstehen. (DIPF/Orig.)Multiple document comprehension (MDC) is defined as the ability to construct an integrated representation based on different sources of information on a particular topic. It is an important competence for both the successful accomplishment of university studies and participation in societal discussions. Yet, there is no established assessment instrument for MDC. Therefore, we developed a test covering four theory-based cognitive requirements of MDC. Based on the data of 310 university students of social sciences and humanities, the MDC test proved to be a unidimensional measure. Furthermore, the test score was related to the final school exam grade, the study level (bachelor / master), and the performance in an essay task. The empirical results suggest that the score of the MDC test can be interpreted as the generic competence of university students to understand multiple documents. (DIPF/Orig.

Electrophysiological Properties of Adult Zebrafish Oligodendrocyte Progenitor Cells

Low remyelination efficiency after spinal cord injury (SCI) is a major restraint to successful axonal and functional regeneration in mammals. In contrast, adult zebrafish can: (i) regenerate oligodendrocytes and myelin sheaths within 2 weeks post lesion; (ii) re-grow axonal projections across the lesion site and (iii) recover locomotor function within 6 weeks after spinal cord transection. However, little is known about the intrinsic properties of oligodendrocyte progenitor cells (OPCs), the remyelinating cells of the central nervous system (CNS). Here, we demonstrate that purified OPCs from the adult zebrafish spinal cord are electrically active. They functionally express voltage-gated K+ and Na+ channels, glutamate receptors and exhibit depolarizing, tetrodotoxin (TTX)-sensitive spikes, as previously seen in rodent and human OPCs. Furthermore, we show that the percentage of zebrafish OPCs exhibiting depolarizing spikes and Nav-mediated currents is lower as compared to rodent white matter OPCs, where these membrane characteristics have been shown to underlie OPC injury susceptibility. These findings imply that adult zebrafish OPCs resemble electrical properties found in mammals and represent a relevant cell type towards understanding the biology of the primary cells targeted in remyelination therapies for non-regenerative species. The in vitro platform introduced in this study could be used in the future to: (i) elucidate how membrane characteristics of zebrafish OPCs change upon injury and (ii) identify potential signaling components underlying OPC injury recognition

Environmental changes in oxygen tension reveal ROS-dependent neurogenesis and regeneration in the adult newt brain

Acknowledgements: We thank A Elewa, N Dantuma, C Sjögren for many helpful comments on the manuscript, and H Wang and M Kirkham for advice. This work was supported by grants from the European Research Council, Swedish Research Council, Swedish Cancer Society, AFA Insurances to AS. YC´s laboratory is supported by research grants from the Swedish Research Council, the Swedish Cancer Foundation, the Karolinska Institute Foundation, the Karolinska Institute distinguished professor award, the Torsten Soderbergs foundation, the NOVO Nordisk Foundation, the Advanced grant from the NOVO Nordisk foundation, and the Alice Wallenberg foundation This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.Peer reviewedPublisher PD