A comparison of reliability coefficients for psychometric tests that consist of two parts

If a test consists of two parts the Spearman-Brown formula and Flanagan's coefficient (Cronbach's alpha) are standard tools for estimating the reliability. However, the coefficients may be inappropriate if their associated measurement models fail to hold. We study the robustness of reliability estimation in the two-part case to coefficient misspecification. We compare five reliability coefficients and study various conditions on the standard deviations and lengths of the parts. Various conditional upper bounds of the differences between the coefficients are derived. It is shown that the difference between the Spearman-Brown formula and Horst's formula is negligible in many cases. We conclude that all five reliability coefficients can be used if there are only small or moderate differences between the standard deviations and the lengths of the parts

Kappa coefficients for dichotomous-nominal classifications

Two types of nominal classifications are distinguished, namely regular nominal classifications and dichotomous-nominal classifications. The first type does not include an 'absence' category (for example, no disorder), whereas the second type does include an 'absence' category. Cohen's unweighted kappa can be used to quantify agreement between two regular nominal classifications with the same categories, but there are no coefficients for assessing agreement between two dichotomous-nominal classifications. Kappa coefficients for dichotomous-nominal classifications with identical categories are defined. All coefficients proposed belong to a one-parameter family. It is studied how the coefficients for dichotomous-nominal classifications are related and if the values of the coefficients depend on the number of categories. It turns out that the values of the new kappa coefficients can be strictly ordered in precisely two ways. The orderings suggest that the new coefficients are measuring the same thing, but to a different extent. If one accepts the use of magnitude guidelines, it is recommended to use stricter criteria for the new coefficients that tend to produce higher values

Additive kappa can be increased by combining adjacent categories

Weighted kappa is a measure that is commonly used for quantifying similarity between two ordinal variables with identical categories. Additive kappa is a special case of weighted kappa that allows the researcher to specify distances between adjacent categories. It is shown that additive kappa is a weighted average of the additive kappas of all collapsed tables of a specific size. It follows that, if the reliability of a categorical rating instrument is assessed with additive kappa, the reliability can be increased by combining categories.<br/

On the negative bias of the Gini coefficient due to grouping

The Gini coefficient is a measure of statistical dispersion that is commonly used as a measure of inequality of income, wealth or opportunity. Empirical research has shown that the coefficient may have a nonnegligible downward bias when data are grouped. It is unknown under which grouping conditions the downward bias occurs. In this note it is shown that the Gini coefficient strictly decreases if the data are partitioned into equal sized groups

On Association Measures for Continuous Variables and Correction for Chance

This paper studies correction for chance for association measures for continuous variables. The set of linear transformations of Pearson's product-moment correlation is used as the domain of the correction for chance function. Examples of measures in this set are Tucker's congruence coefficient, Jobson's coefficient, and Pearson's correlation. An equivalence relation on the set of linear transformations is defined. The fixed points of the correction for chance function are characterized. It is shown that each linear transformation is mapped to the fixed point in its equivalence class

Ordering Properties of the First Eigenvector of Certain Similarity Matrices

It is shown for coefficient matrices of Russell-Rao coefficients and two asymmetric Dice coefficients that ordinal information on a latent variable model can be obtained from the eigenvector corresponding to the largest eigenvalue

Properties of Bangdiwala's B

Cohen's kappa is the most widely used coefficient for assessing interobserver agreement on a nominal scale. An alternative coefficient for quantifying agreement between two observers is Bangdiwala's B. To provide a proper interpretation of an agreement coefficient one must first understand its meaning. Properties of the kappa coefficient have been extensively studied and are well documented. Properties of coefficient B have been studied, but not extensively. In this paper, various new properties of B are presented. Category B-coefficients are defined that are the basic building blocks of B. It is studied how coefficient B, Cohen's kappa, the observed agreement and associated category coefficients may be related. It turns out that the relationships between the coefficients are quite different for 2x2 tables than for agreement tables with three or more categories

Symmetric kappa as a function of unweighted kappas

It is shown that a symmetric kappa corresponding to a c x c table with c>2 categories can be written as a function of the unweighted kappa corresponding to the same table and the c(c - 1)/2 distinct unweighted kappas associated with the (c - 1) x (c - 1) tables that are obtained by combining two categories. The result is a new MGB-type result