Anchoring vignettes are used as a methodological technique for removing differential interpretation of response categories (DIRC) from scores on subjective self-report measures (King, Murray, Slomon, & Tandon, 2004). This technique requires participants to read one or more short scenarios, or vignettes, designed to represent various levels of a construct. Vignette ratings are used as an indication of DIRC, which is a source of differential item functioning (DIF). Prior research primarily used indirect methods for evaluating vignette quality. In response, the present set of studies proposes using invariance testing as a more direct evaluation of how the use of anchoring vignettes impacts the presence of DIRC. The effort subscale from the Student Opinion Scale (SOS) is used to demonstrate this set of procedures. It is also argued that DIRC will manifest as non-uniform DIF given that corrections using anchoring vignettes should impact the rank order of cases.
In these studies, 819 participants were randomly assigned to either a control group (n = 478) or a group that received vignettes (n = 341) prior to responding to the SOS. Invariance testing was completed in two studies. The first study examined the factor structure between the control group and the vignette group before adjusting scores using the vignettes to determine what effect reading the vignettes may have had on the factor structure. The second study examined the invariance between the control group and the vignette group after score adjustment to determine what effect adjusting scores using the vignettes may have had on the factor structure. Results for the first study supported strict factorial invariance (configural, metric, and scalar invariance, and residuals) and equivalent latent means, which suggests that just viewing the vignettes had insubstantial impact on the factor structure of the SOS effort subscale. Results for the second study also supported strict factorial invariance, but there was a substantial difference in the group’s latent means. This result suggests that DIRC was not removed from the sample, however using anchoring vignettes to adjust scores resulted in systematically lower observed scores after adjustment. Implications for measuring effort along with general conclusions about using invariance testing to evaluate anchoring vignettes is also provided
