Inter-Coder Agreement for Computational Linguistics

This article is a survey of methods for measuring agreement among corpus annotators. It exposes the mathematics and underlying assumptions of agreement coefficients, covering Krippendorff's alpha as well as Scott's pi and Cohen's kappa; discusses the use of coefficients in several annotation tasks; and argues that weighted, alpha-like coefficients, traditionally less used than kappa-like measures in computational linguistics, may be more appropriate for many corpus annotation tasks—but that their use makes the interpretation of the value of the coefficient even harder. </jats:p

The agreement process: an empirical investigation of human-human computer-mediated collaborative dialogs

this paper, we investigate the empirical correlates of the agreement process. Informally, the agreement process is the dialogue process by which collaborators achieve joint commitment on a joint action. We propose a specific instantiation of the agreement process, derived from our theoretical model, that integrates the IRMA framework for rational problem solving (Bratman, Israel, and Pollack 1988) with Clark&apos;s work (1992; 1996) on language as a collaborative activity; and from the characteristics of our task, a simple design problem (furnishing a two room apartment) in which knowledge is equally distributed among agents, and needs to be shared. The main contribution of our paper is an empirical study of some of the components of the agreement process. We first discuss why we believe the findings from our corpus of computer-mediated dialogues are applicable to human-human collaborative dialogues in general. We then present our theoretical model, and apply it to make predictions about the components of the agreement process. We focus on how information is exchanged in order to arrive at a proposal, and on what constitutes a proposal and its acceptance / rejection. Our corpus study makes use of features of both the dialogue and the domain reasoning situation, and led us to discover that the notion of commitment is more useful to model the agreement process than that of acceptance / rejection, as it more closely relates to the unfolding of negotiation

An Empirical Investigation of Proposals in Collaborative Dialogues

We describe a corpus-based investigation of proposals in dialogue. First, we describe our DRI compliant coding scheme and report our inter-coder reliability results. Next, we test several hypotheses about what constitutes a well-formed proposal. 1 Introduction Our project&apos;s long-range goal (see http://www.isp. pitt.edu/~intgen/) is to create a unified architecture for collaborative discourse, accommodating both interpretation and generation. Our computational approach (Thomason and Hobbs, 1997) uses a form of weighted abduction as the reasoning mechanism (Hobbs et al., 1993) and modal operators to model context. In this paper, we describe the corpus study portion of our project, which is an integral part of our investigation into recognizing how conversational participants coordinate agreement. From our first annotation trials, we found that the recognition of &quot;classical&quot; speech acts (Austin, 1962; Searle, 1975) by coders is fairly reliable, while recognizing contextual relationships..

An Empirical Investigation of Proposals in Collaborative Dialogues

We describe a corpus-based investigation of propos-als in dialogue. First, we describe our DRI compliant coding scheme and report our inter-coder reliability results. Next, we test several hypotheses about what constitutes a well-formed proposal.