Multivariate analysis of psychological data - ou

Warrens, M.J.

Leiden University Scholary Publications

DOI: 10.1007/s00357-008-On the Equivalence of Cohen’s Kappa and theHubert-Arabie Adjusted Rand IndexMatthijs J. WarrensLeiden University, The NetherlandsAbstract: It is shown that one can calculate the Hubert-Arabie adjusted Rand indexby first forming the fourfold contingency table counting the number of pairs of objectsthat were placed in the same cluster in both partitions, in the same cluster in onepartition but in different clusters in the other partition, and in different clusters inboth, and then computing Cohen’s κ on this fourfold table.Keywords: Correction for chance agreement; Partitions; Clustering method; Match-ing table; Simple matching coefficient; Similarity indices; Resemblance measures.1. IntroductionIt is shown in this note that two resemblance measures used in differ-ent realms of research are in fact equivalent. The first index is used in clus-ter analysis for comparing two partitions obtained from different clusteringmethods. There seems to be some agreement in the cluster community thatthe preferred measure for comparing two partitions is the Hubert-Arabie ad-justed Rand index, proposed by Hubert and Arabie (1985) (see, for example,Steinley 2004). The second measure is often used in psychological researchto assess the agreement of judgments made by two observers, and is knownas the kappa statistic proposed by Cohen (1960). Cohen’s κ is appropriateThe author thanks Willem Heiser, Mark de Rooij, Marian Hickendorff and threeanonymous reviewers for their helpful comments and valuable suggestions on earlier ver-sions of this article.Author’s Address: Psychometrics and Research Methodology Group, Leiden Uni-versity Institute for Psychological Research, Leiden University, Wassenaarseweg 52, P.O.Box 9555, 2300 RB Leiden, The Netherlands, e-mail: warrens@fsw.leidenuniv.nl.9023-7Journal of Classification 25:177-183 (2008)Published online 6 December 2008M. J. Warrensfor testing whether agreement exceeds chance levels for binary and nominalratings.One step in comparing two clusterings, is to obtain the 2 × 2 contin-gency matrix summarizing the matching table of two clusterings. A similarfourfold table may be obtained in psychological research as the cross classi-fication of judgments by two observers on the presence or absence of a trait.It turns out that the same coefficient value is obtained when one applies ei-ther Cohen’s κ or the Hubert-Arabie adjusted Rand index to these types offourfold tables.2. Comparing Two PartitionsIn cluster analysis, one may be interested in comparing two clusteringmethods (Rand 1971; Fowlkes and Mallows 1983; Hubert and Arabie 1985;Steinley 2004; Albatineh, Niewiadomska-Bugaj and Mihalko 2006). Sup-pose we have two partitions of m data points. To compare these two cluster-ings, a first step is to obtain a so-called matching table M = {mij}, wheremij indicates the number of data points placed in cluster i (i = 1, 2, ..., I)according to the first clustering method and in cluster j (j = 1, 2, ..., J) ac-cording to the second method. The total number of points being clusteredis given by m =∑Ii=1∑Jj=1 mij . The cluster sizes in the two clusteringsconsidered are the row and column totals of table M, given bymi+ =J∑j=1mij and m+j =I∑i=1mij .Furthermore, we define the quantityT =I∑i=1J∑j=1(mij2)=12⎡⎣ I∑i=1J∑j=1m2ij − m⎤⎦ ,which equals the number of object pairs that were placed in the same clusteraccording to both clustering methods, and the three quantitiesP =I∑i=1(mi+2), Q =J∑j=1(m+j2)and N =(m2).The quantity N equals the total number of pairs of objects given m points.As a second step, one may calculate some sort of resemblance mea-sure that summarizes the information in table M. A well-known measurefor the similarity of two partitions is the Rand index (Rand 1971), given byR =N + 2T − P − QN.178Cohen’s Kappa and the Hubert-Arabie Adjusted Rand IndexThe Rand index may be adjusted for agreement due to chance (Cohen 1960;Hubert and Arabie 1985; Albatineh et al. 2006). In general, a similarityindex S after such correction has a formAS =S − E(S)max(S) − E(S) . (1)The expectation E(S) in (1) is conditional upon fixed sets of marginal num-bers corresponding to the table of which S is the summary index. The quan-tity max(S) in (1) is the maximum value of index S regardless of the mar-ginal numbers (Hubert and Arabie 1985).Fowlkes and Mallows (1983) and Hubert and Arabie (1985, p. 197)note that, under the generalized hypergeometric assumption with respect totable M, the expectation E(T ) is given byE (T ) =PQN. (2)Using (2), the expectation E(R) corresponding to the Rand index is givenbyE(R) = 1 +2PQN2− P + QN(Hubert and Arabie 1985, p. 198).Using R and E(R) in (1), we obtain the Hubert-Arabie adjusted Rand index(Hubert and Arabie 1985, p. 198), which is given byAR =T − PQ/N12(P + Q) − PQ/N=2(NT − PQ)N(P + Q) − 2PQ.3. Reformulation of the Rand IndexAs noted in, for example, Steinley (2004) or Albatineh et al. (2006),the information in a matching table M of two clustering partitions on thesame data points, can be summarized in a fourfold table like Table 1. InTable 1, a is the number of object pairs that were placed in the same clusteraccording to both clustering methods, b (c) is the number of pairs that wereplaced in the same cluster according to one method but not according tothe other, and d is the number of pairs that were not in the same clusteraccording to either of the methods. It then holds thata + b + c + d = N (3)where a = T , b = P − T , c = Q − T and d = N + T − P − Q, andp1 = a+ b = P and q1 = c+d = N −P . The four different types of object179M. J. WarrensTable 1. 2 × 2 Contingency Table Representation of a Matching Table M.Second partitionPair in same Pair in TotalFirst partition cluster different clusterPair in same cluster a b p1Pair in different cluster c d q1Total p2 q2 Npairs are also distinguished in Hubert and Arabie (1985, p. 194). However,these authors expressed their formulas in terms T , P , Q, and N , instead ofthe quantities a, b, c, and d.The information in Table 1 can be summarized by some sort of resem-blance index or similarity coefficient, and a vast amount of formulas can befound in the classification literature (see, for example, one of the followingpapers that have appeared in the Journal of Classification: Gower and Legen-dre 1986; Baulieu 1989; Batagelj and Bren 1995; Albatineh et al. 2006). Awell-known resemblance measure is the simple matching coefficient (Sokaland Michener 1958), given bySM =a + da + b + c + d=a + dN. (4)Expressing the Rand index in terms of the quantities a, b, c, and d we obtainthe formula in (4). Thus, if the Rand index is formulated in terms of thequantities a, b, c, and d, it is equivalent to the simple matching coefficient(see, for example, Steinley 2004, or Albatineh et al. 2006).4. Similarity Between RatingsIn psychological research, one may be interested in the degree of con-sensus between two observers who rate each of a sample of subjects on anominal scale. Table 1 can be obtained as a cross classification of judgmentsby two observers on the presence or absence of a trait: a is the number oftimes a trait was present according to both observers, b (c) is the number oftimes a trait was present according to one observer but not according to theother, and d is the number of subjects for which a trait was absent accordingto both observers.One may want to calculate a resemblance measure that summarizesthe information in Table 1. A possible resemblance measure is the simplematching coefficient in (4). However, a coefficient that is often used forrater data, is the kappa statistic introduced by Cohen (1960). Cohen (1960)180Cohen’s Kappa and the Hubert-Arabie Adjusted Rand Indexconsidered correction for agreement due to chance for the simple matchingcoefficient. Cohen assumed that the data are a product of chance of twodifferent frequency distributions underlying the rows and columns of Table1. The expectation of the quantity a in Table 1 is then the product of themarginals corresponding to a, p1 and p2, divided by N , that is, E(a) =p1p2/N . The complete case of statistical independence is presented in Table2.The expectation E(SM) corresponding to the simple matching coef-ficient given independence, is given byE(SM) = E(a + dN)=E(a + d)N=p1p2N2+q1q2N2.Expectation E(SM) can be obtained by considering all permutations of theobservations of one of the two raters, while preserving the order of the ob-servations of the other rater. For each permutation the value of SM can bedetermined. The arithmetic mean of these values is (p1p2 + q1q2)/N2.Using SM and E(SM) in (1), we obtainCohen’s κ =2(ad − bc)p1q2 + p2q1. (5)5. Equivalence of AR and Cohen’s κSimilar to the Rand index, the adjusted Rand index may be expressedin terms of the quantities a, b, c, and d. Expressing the Hubert-Arabie ad-justed Rand index in these quantities, we obtain, following Steinley (2004,p. 388), the formulaAR =N(a + d) − [(a + b)(a + c) + (b + d)(c + d)]N2 − [(a + b)(a + c) + (b + d)(c + d)] . (6)Using (3), we have the equalities(a + b)(a + c) = a(a + b + c) + bc = a(N − d) + bcand (b + d)(c + d) = d(b + c + d) + bc = d(N − a) + bc.Using these equalities, the numerator of (6) can be written asN(a + d) − [(a + b)(a + c) + (b + d)(c + d)]= Na + Nd − a(N − d) − d(N − a) − 2bc= 2(ad − bc).181M. J. WarrensTable 2. The Expected Values of Quantities a, b, c, and d in Table 1 under the AssumptionThat the Data are a Product of Chance of Two Different Distribution Functions.Second partitionPair in same Pair in TotalFirst partition cluster different clusterPair in same cluster p1p2/N p1q2/N p1Pair in different cluster p2q1/N q1q2/N q1Total p2 q2 NFurthermore, the denominator of (6) equalsN2 − [(a + b)(a + c) + (b + d)(c + d)]= N2 − p1p2 − q1q2= (p1 + q1)(p2 + q2) − p1p2 − q1q2= p1q2 + p2q1.Hence, Equation (6) is equivalent to Equation (5). Thus, if the adjusted Randindex is formulated in terms of the quantities a, b, c, and d, it is equivalentto Cohen’s κ. Moreover, expectation E(T ) in (2) can be written asE(T ) =PQN=(a + b)(a + c)N=p1p2N= E(a).Hence, statistical independence under the generalized hypergeometric dis-tribution function used in Hubert and Arabie (1985) for the matching tableof two partitions, is equivalent to the case of statistical independence underthe binomial distribution function for the fourfold contingency table.6. DiscussionA practical conclusion is that we can calculate the Hubert-Arabie ad-justed Rand index by first forming the fourfold contingency table countingthe number of pairs of objects that were placed in the same cluster in bothpartitions, in the same cluster in one partition but in different clusters in theother partition, and in different clusters in both, and then computing Cohen’sκ on this fourfold table (compare Saltstone and Stange 1996, p. 171).As pointed out by a referee, it is important to note that, although theformulas of the Rand index and the simple matching coefficient, and theformulas of the Hubert-Arabie adjusted Rand index and Cohen’s κ, are thesame, the inputs are quite different. The simple matching coefficient andCohen’s κ operate on singletons and require the labels of the clusters for182Cohen’s Kappa and the Hubert-Arabie Adjusted Rand Indexmatching the partitions, or the labels for rating, to be known. However, theRand index and the adjusted Rand index are based on pairs of observationsthat appear in the same cluster in each of the partitions. Steinley (2004), forexample, considers this one of the biggest advantages of the adjusted Randindex over other measures that require some knowledge about the labeling.ReferencesALBATINEH, A.N., NIEWIADOMSKA-BUGAJ, M., and MIHALKO, D. (2006), “OnSimilarity Indices and Correction for Chance Agreement,” Journal of Classification,23, 301-313.BATAGELJ, V., and BREN, M. (1995), “Comparing Resemblance Measures,” Journal ofClassification, 12, 73-90.BAULIEU, F.B. (1989), “A Classification of Presence/Absence Based Dissimilarity Coeffi-cients,” Journal of Classification, 6, 233-246.COHEN, J. (1960), “A Coefficient of Agreement for Nominal Scales,” Educational andPsychological Measurement, 20, 37-46.FOWLKES, E.B., and MALLOWS, C. L. (1983), “A Method for Comparing Two Hierar-chical Clusterings,” Journal of the American Statistical Association, 78, 553-569.GOWER, J.C., and LEGENDRE, P. (1986), “Metric and Euclidean Properties of Dissimi-larity Coefficients,” Journal of Classification, 3, 5-48.HUBERT, L.J., and ARABIE, P. (1985), “Comparing Partitions,” Journal of Classification,2, 193-218.RAND, W.M. (1971), “Objective Criteria for the Evaluation of Clustering Methods,” Jour-nal of the American Statistical Association, 66, 846-850.SALTSTONE, R., and STANGE, K. (1996), “A Computer Program to Calculate Hubert andArabie’s Adjusted Rand Index,” Journal of Classification, 13, 169-172.SOKAL, R.R., and MICHENER, C.D. (1958), “A Statistical Method for Evaluating Sys-tematic Relationships,” University of Kansas Science Bulletin, 38, 1409-1438.STEINLEY, D. (2004), “Properties of the Hubert-Arabie Adjusted Rand Index,” Psycho-logical Methods, 9, 386-396.183

On the equivalence of Cohen's kappa and the Hubert-Arabie adjusted rand index.

https://scholarlypublications.universiteitleiden.nl/access/item%3A2872286/download

On the equivalence of Cohen's kappa and the Hubert-Arabie adjusted rand index.

Abstract

Similar works

Full text

Available Versions

Leiden University Scholary Publications