Survey on Evaluation Methods for Dialogue Systems

In this paper we survey the methods and concepts developed for the evaluation of dialogue systems. Evaluation is a crucial part during the development process. Often, dialogue systems are evaluated by means of human evaluations and questionnaires. However, this tends to be very cost and time intensive. Thus, much work has been put into finding methods, which allow to reduce the involvement of human labour. In this survey, we present the main concepts and methods. For this, we differentiate between the various classes of dialogue systems (task-oriented dialogue systems, conversational dialogue systems, and question-answering dialogue systems). We cover each class by introducing the main technologies developed for the dialogue systems and then by presenting the evaluation methods regarding this class

Challenging Neural Dialogue Models with Natural Data: Memory Networks Fail on Incremental Phenomena

Natural, spontaneous dialogue proceeds incrementally on a word-by-word basis; and it contains many sorts of disfluency such as mid-utterance/sentence hesitations, interruptions, and self-corrections. But training data for machine learning approaches to dialogue processing is often either cleaned-up or wholly synthetic in order to avoid such phenomena. The question then arises of how well systems trained on such clean data generalise to real spontaneous dialogue, or indeed whether they are trainable at all on naturally occurring dialogue data. To answer this question, we created a new corpus called bAbI+ by systematically adding natural spontaneous incremental dialogue phenomena such as restarts and self-corrections to the Facebook AI Research's bAbI dialogues dataset. We then explore the performance of a state-of-the-art retrieval model, MemN2N, on this more natural dataset. Results show that the semantic accuracy of the MemN2N model drops drastically; and that although it is in principle able to learn to process the constructions in bAbI+, it needs an impractical amount of training data to do so. Finally, we go on to show that an incremental, semantic parser -- DyLan -- shows 100% semantic accuracy on both bAbI and bAbI+, highlighting the generalisation properties of linguistically informed dialogue models.Comment: 9 pages, 3 figures, 2 tables. Accepted as a full paper for SemDial 201

The Pragmatics of Arabic Religious Posts on Facebook: A Relevance-Theoretic Account

Despite growing interest in the impact of computer-mediated communication on our lives, linguistic studies on such communication conducted in the Arabic language are scarce. Grounded in Relevance Theory, this paper seeks to fill this void by analysing the linguistic structure of Arabic religious posts on Facebook. First, I discuss communication on Facebook, treating it as a relevance-seeking process of writing or sharing posts, with the functions of ‘Like’ and ‘Share’ seen as cues for communicating propositional attitude. Second, I analyse a corpus of around 80 posts, revealing an interesting use of imperatives, interrogatives and conditionals which manipulate the interpretation of such posts between descriptive and interpretive readings. I also argue that a rigorous system of incentives is employed in such posts in order to boost their relevance. Positive, negative and challenging incentives link the textual to the visual message in an attempt to raise more cognitive effects for the readers

"How May I Help You?": Modeling Twitter Customer Service Conversations Using Fine-Grained Dialogue Acts

Given the increasing popularity of customer service dialogue on Twitter, analysis of conversation data is essential to understand trends in customer and agent behavior for the purpose of automating customer service interactions. In this work, we develop a novel taxonomy of fine-grained "dialogue acts" frequently observed in customer service, showcasing acts that are more suited to the domain than the more generic existing taxonomies. Using a sequential SVM-HMM model, we model conversation flow, predicting the dialogue act of a given turn in real-time. We characterize differences between customer and agent behavior in Twitter customer service conversations, and investigate the effect of testing our system on different customer service industries. Finally, we use a data-driven approach to predict important conversation outcomes: customer satisfaction, customer frustration, and overall problem resolution. We show that the type and location of certain dialogue acts in a conversation have a significant effect on the probability of desirable and undesirable outcomes, and present actionable rules based on our findings. The patterns and rules we derive can be used as guidelines for outcome-driven automated customer service platforms.Comment: 13 pages, 6 figures, IUI 201

The Role of Pragmatics in Cross-cultural

We here try to find out the role of pragmatics in the cross-cultural contexts. Pragmatics is the way we convey meaning through communication (Deda, 2013). Other factors beyond competence are the adjustments between contexts and situations that can change the ordinary meaning of elements/sentences according to the language situation. The culture of an organization decides the way employees behave amongst themselves as well as the people outside the organization. Pragmatic culture more emphasis is placed on the clients and the external parties. Customer satisfaction is the main motive of the employees in a pragmatic culture. In linguistics, pragmatic competence is the ability to use language effectively in a contextually appropriate fashion. Pragmatic competence is a fundamental aspect of a more general communicative competence