Determining Reliability of Subjective and Multi-label Emotion Annotation through Novel Fuzzy Agreement Measure

The paper presents a new fuzzy agreement measure γf for determining the agreement in multi-label and subjective annotation task. In this annotation framework, one data item may belong to a category or a class with a belief value denoting the degree of confidence of an annotator in assigning the data item to that category. We have provided a notion of disagreement based on the belief values provided by the annotators with respect to a category. The fuzzy agreement measure γf has been proposed by defining different fuzzy agreement sets based on the distribution of difference of belief values provided by the annotators. The fuzzy agreement has been computed by studying the average agreement over all the data items and annotators. Finally, we elaborate on the computation γf measure with a case study on emotion text data where a data item (sentence) may belong to more than one emotion category with varying belief values. 1