Validating a novel web-based method to capture disease progression outcomes in multiple sclerosis

The Expanded Disability Status Scale (EDSS) is the current ‘gold standard’ for monitoring disease severity in multiple sclerosis (MS). The EDSS is a physician-based assessment. A patient-related surrogate for the EDSS may be useful in remotely capturing information. Eighty-one patients (EDSS range 0–8) having EDSS as part of clinical trials were recruited. All patients carried out the web-based survey with minimal assistance. Full EDSS scores were available for 78 patients. The EDSS scores were compared to those generated by the online survey using analysis of variance, matched pair test, Pearson’s coefficient, weighted kappa coefficient, and the intra-class correlation coefficient. The internet-based EDSS scores showed good correlation with the physician-measured assessment (Pearson’s coefficient = 0.85). Weighted kappa for full agreement was 0.647. Full agreement was observed in 20 patient

Properties of Bangdiwala's B

Cohen's kappa is the most widely used coefficient for assessing interobserver agreement on a nominal scale. An alternative coefficient for quantifying agreement between two observers is Bangdiwala's B. To provide a proper interpretation of an agreement coefficient one must first understand its meaning. Properties of the kappa coefficient have been extensively studied and are well documented. Properties of coefficient B have been studied, but not extensively. In this paper, various new properties of B are presented. Category B-coefficients are defined that are the basic building blocks of B. It is studied how coefficient B, Cohen's kappa, the observed agreement and associated category coefficients may be related. It turns out that the relationships between the coefficients are quite different for 2x2 tables than for agreement tables with three or more categories

Inter-Coder Agreement for Computational Linguistics

This article is a survey of methods for measuring agreement among corpus annotators. It exposes the mathematics and underlying assumptions of agreement coefficients, covering Krippendorff's alpha as well as Scott's pi and Cohen's kappa; discusses the use of coefficients in several annotation tasks; and argues that weighted, alpha-like coefficients, traditionally less used than kappa-like measures in computational linguistics, may be more appropriate for many corpus annotation tasks—but that their use makes the interpretation of the value of the coefficient even harder. </jats:p

Measuring Consensus in Binary Forecasts: NFL Game Predictions

Previous research on defining and measuring consensus (agreement) among forecasters has been concerned with evaluation of forecasts of continuous variables. This previous work is not relevant when the forecasts involve binary decisions: up-down or win-lose. In this paper we use Cohen¡¯s kappa coefficient, a measure of inter-rater agreement involving binary choices, to evaluate forecasts of National Football League games. This statistic is applied to the forecasts of 74 experts and 31 statistical systems that predicted the outcomes of games during two NFL seasons. We conclude that the forecasters, particularly the systems, displayed significant levels of agreement and that levels of agreement in picking game winners were higher than in picking against the betting line. There is greater agreement among statistical systems in picking game winners or picking winners against the line as the season progresses, but no change in levels of agreement among experts. High levels of consensus among forecasters are associated with greater accuracy in picking game winners, but not in picking against the line.binary forecasts, NFL, agreement, consensus, kappa coefficient

A comparison of antibiotic disks from different sources on Quicolor and Mueller-Hinton agar media in evaluation of antibacterial susceptibility testing

Background and Objectives: Antibacterial susceptibility testing of clinical bacterial isolates through disk diffusion method plays a major role in antibacterial treatment. One of the main factors affecting the result of these tests is the type, structure and quality of the disks. The main objective of this study was to compare the agreement of antibiotic disks originated from three companies on Quicolor and Mueller-Hinton agar. Materials and Methods: Quicolor and Mueller-Hinton agar media were used in disk diffusion method. Seventy clinical isolates from Enterobacteriaceae family (21 Klebsiella spp., 36 Escherichia coli, 1 Enterobacter spp. and 12 Shigella spp.) were investigated in the study. After obtaining data, the results were interpreted as resistant, sensitive or intermediate. Kappa coefficient measured the agreement of two media. Coefficient of variation (CV) was also calculated for antibiotic disks. Results: The kappa agreement values for three types of antibiotic disks on Quicolor and Mueller-Hinton agar plates were good or excellent for all the examined antibiotics. CV values were also very satisfactory in the majority of cases. Conclusion: Antibiotic disks from three manufacturers can successfully be used on both Quicolor and Mueller-Hinton agar plates

Generalization of the kappa coeficient for ordinal categorical data, multiple observers and incomplete designs

This paper presents a generalization of the kappa coefficient for multiple observers and incomplete designs. This generalization involves ordinal categorical data and includes weights which permit pondering the severity of disagreement. A generalization for incomplete designs of the kappa coefficient based on explicit definitions of agreement is also proposed. Both generalizations are illustrated with data from a medical diagnosis pilot study

Generalization of the kappa coeficient for ordinal categorical data, multiple observers and incomplete designs

This paper presents a generalization of the kappa coefficient for multiple observers and incomplete designs. This generalization involves ordinal categorical data and includes weights which permit pondering the severity of disagreement. A generalization for incomplete designs of the kappa coefficient based on explicit definitions of agreement is also proposed. Both generalizations are illustrated with data from a medical diagnosis pilot study

Agreement between an isolated rater and a group of raters

The agreement between two raters judging items on a categorical scale is traditionally assessed by Cohen’s kappa coefficient. We introduce a new coefficient for quantifying the degree of agreement between an isolated rater and a group of raters on a nominal or ordinal scale. The group of raters is regarded as a whole, a reference or gold-standard group with its own heterogeneity. The coefficient, defined on a population-based model, requires a specific definition of the concept of perfect agreement. It has the same properties as Cohen’s kappa coefficient and reduces to the latter when there is only one rater in the group. The new approach overcomes the problem of consensus within the group of raters and generalizes Schouten’s index. The method is illustrated on published syphilis data and on data collected from a study assessing the ability of medical students in diagnostic reasoning when compared with expert knowledge