Automatic Concept Identification In Goal-Oriented Conversations

Abstract

We address the problem of identifying key domain conceptsautomatically from an unannotated corpus of goal-orientedhuman-human conversations. We examine two clusteringalgorithms, one based on mutual information and another onebased on Kullback-Liebler distance. In order to compare theresults from both techniques quantitatively, we evaluate theoutcome clusters against reference concept labels usingprecision and recall metrics adopted from the evaluation oftopic identification task. However, since our system allowsmore than one cluster to associate with each concept anadditional metric, a singularity score, is added to better capturecluster quality. Based on the proposed quality metrics, theresults show that Kullback-Liebler-based clusteringoutperforms mutual information-based clustering for both theoptimal quality and the quality achieved using an automaticstopping criterion</p