Removing situation descriptions from situational judgment test items: Does the impact differ for video-based versus text-based formats?

Recent research has shown that many text-based situational judgment test (SJT) items can be solved even when the situational descriptions in the item stems are not presented to test takers. This finding challenges the traditional view of SJTs as low-fidelity simulations that rely on 'situational' (context-dependent) judgment. However, media richness theory and construal level theory suggest that situation descriptions presented in a richer and more concrete format (video format) will reduce uncertainty about inherent requirements and facilitate the perception that the situation is taking place in the here and now. Therefore, we hypothesized that situational judgment would be more important in video situation descriptions than in text situation descriptions. We adapted a leadership SJT to realize a 3 (situation description in the item stem: video vs. text vs. none) x 2 (response format: video response options vs. text response options) between-subjects design (N = 279). Participants were randomly assigned to one of the six conditions. The removal of video-based situation descriptions in item stems led to an equivalent decrease in SJT scores as the removal of text-based situation descriptions in item stems (video-based version: Cohen's d = 0.535 vs. text-based version: Cohen's d = 0.531). SJT scores were also contingent on the presentation format of both situation descriptions and response options: The highest scores were observed when situation descriptions and response options were presented in the same format. Implications for SJT theory and research are discussed. Practitioner points The presentation format did not moderate the effect of omitting situation descriptions in SJTs - that is, the context dependency of SJT performance did not increase when the SJT was administered in a video-based rather than a text-based format. The elimination of situation descriptions in item stems had a medium effect on overall test scores: SJT scores were significant lower without situation descriptions in comparison to SJT scores with situation descriptions (video-based version: Cohen's d = 0.535 vs. text-based version: Cohen's d = 0.531). It is important to match the stimulus and response formats in SJTs

Face to face.

160 p. : ill ; 28 cm

Face to face.

224 p. : ill ; 28 cm

Empiricism versus connoisseurship: establishing the appropriacy of texts in tests of academic reading

This paper explores the potential of using information technology for the empirical measurement of contextual parameters identified in the socio-cognitive framework (the framework developed by CRELLA for test validation). Previously the level of difficulty represented by reading texts used in English Language tests has been estimated through expert judgement. The study, in contrast, employs computational tools for automated text analyses, demonstrating how relevant contextual features have been opened to empirical measurement by new technology. The techniques showcased in this paper allow for a more rigorous approach to reading test specification

Sequencing as an item type.

A text’s coherence clearly depends upon the way ideas are related within that text, both in terms of their logical relations, as well as the cohesive devices that show, or create, the links between ideas, across paragraphs and sentences. Thus, it would appear that part of the ability of a competent reader is to recognize the appropriate order of ideas in text, to identify cohesion and coherence in text in order to relate the ideas to each other, and to understand authorial intention with respect to the sequence of ideas. It follows from this that a potentially useful test method that might tap such abilities is to require candidates to inspect text in which the elements are out of order, and to reconstruct the original order. This, it might be supposed, would require candidates to detect the relationship among ideas, to identify cohesive devices and their interrelationships. Such test methods are, indeed, increasingly common in so-called reading tests. However, we know of no reports of research into, or even descriptions of the use of, this promising task type. In this article we report on potential problems in scoring responses to sequencing tests, the development of a computer program to overcome these difficulties, and an exploration of the value of various scoring procedures