Confidence intervals for maximal reliability of probability judgments

Abstract

Subjective probabilities play an important role in marketingresearch, for example where individuals rate the likelihood thatthey will purchase a new to develop product. The tau-equivalentmodel can describe the joint behaviour of multiple test itemsmeasuring the same subjective probability. It improves thereliability of the subjective probability estimate by using aweighted sum as the outcome of the test rather than an unweightedsum. One can choose the weights to obtain maximal reliability.In this paper we stress the use of confidence intervals to assessmaximal reliability, as this allows for a more critical assessmentof the items as measurement instruments. Furthermore, two newconfidence intervals for the maximal reliability are derived andcompared to intervals derived earlier in \\citet{YuanBentler2002,RaykovPenev2006}. The comparison involves coverage curves, amethodology that is new in the field of reliability. The existingYuan-Bentler and Raykov-Penev intervals are shown to overestimatethe maximal reliability, whereas one of our proposed intervals, thestable interval, performs very well. This stable interval hardlyshows any bias, and has a coverage for the true value which isapproximately equal to the confidence level.confidence intervals;subjective probability;coverage curves;maximal reliability;measurement scales;tau-equivalent model