Validity of Comparative Judgment Scores: How Assessors Evaluate Aspects of Text Quality When Comparing Argumentative Texts

The advantage of comparative judgment is that it is particularly suited to assess multidimensional and complex constructs as text quality. This is because assessors are asked to compare texts holistically and to make a quality judgment for each text in a pairwise comparison based upon on the most salient and critical differences. Also, the resulted rank order is based on the judgment of all assessors, representing the shared consensus. In order to be able to select the right number of assessors, the question is to what extent the conceptualization of assessors prevails in the aspects they base their judgment on, or whether comparative judgment minimizes the differences between assessors. In other words, can we detect types of assessors who tend to consider certain aspects of text quality more often than others? A total of 64 assessors compared argumentative texts, after which they provided decision statements on what aspects of text quality had informed their judgment. These decision statements were coded on six overarching themes of text quality: argumentation, organization, language use, language conventions, source use, references, and layout. Using a multilevel-latent class analysis, four different types of assessors could be distinguished: narrowly focused, broadly focused, source-focused, and language-focused. However, the analysis also showed that all assessor types mainly focused on argumentation and organization, and that assessor types only partly explained whether the aspect of text quality was mentioned in a decision statement. We conclude that comparative judgment is a strong method for comparing complex constructs like text quality. First, because the rank order combines different views on text quality, but foremost because the method of comparative judgment minimizes differences between assessors

BTL analyses in R

This project contains a git repository with R functions for estimating the Bradley-Terry-Luce mode

Differential effects of a long teacher training internship on students’ learning-to-teach patterns

To become a lifelong learner as a teacher, student teachers already have to learn how to direct their own learning during initial teacher education programmes. Previous empirical research has shown that student teachers differ in their patterns of learning-to-teach, but few is known about the changeability of these learning patterns throughout teacher education and the role of teacher training internships in this. In this study, the changes in student teachers’ patterns of learning-to-teach amongst pre-service teachers were investigated using a longitudinal design. 253 student teachers were asked to complete the ‘Inventory Learning To Teach Process’ questionnaire at two points during the last semester of a three-year teaching programme: immediately prior to and immediately following a long teacher training internship. The results of this study demonstrate that learning-to-teach patterns are subject to a relative degree of change. In particular, survival-oriented student teachers appear to have undergone a greater degree of change after the long teacher training internship