Thresholds and intransitivities in pairwise judgments: A multilevel analysis

Abstract

The method of paired comparisons belongs to a small class of models that simultaneously allow for consistency checks of the response behavior of a person and the estimation of item parameters. As a result, the paired comparison technique is of much importance when there are reasons to expect that respondents have difficulties appraising their values or preferences about an issue of interest. This article presents a hierarchical framework for the analysis of paired comparison data with three response categories that allow judges to be indifferent or undecided. The approach can be viewed as a stochastic representation of Luce's (1956) semiorder. It facilitates statistical analyses of different types of inconsistent responses and yields new tests of the underlying judgmental process. An extensive analysis of a survey study illustrates the usefulness of a multilevel approach for modeling multiple trinary judgments