Survey on Evaluation Methods for Dialogue Systems

In this paper we survey the methods and concepts developed for the evaluation of dialogue systems. Evaluation is a crucial part during the development process. Often, dialogue systems are evaluated by means of human evaluations and questionnaires. However, this tends to be very cost and time intensive. Thus, much work has been put into finding methods, which allow to reduce the involvement of human labour. In this survey, we present the main concepts and methods. For this, we differentiate between the various classes of dialogue systems (task-oriented dialogue systems, conversational dialogue systems, and question-answering dialogue systems). We cover each class by introducing the main technologies developed for the dialogue systems and then by presenting the evaluation methods regarding this class

Augmenting Ad-Hoc IR Dataset for Interactive Conversational Search

A peculiarity of conversational search systems is that they involve mixed-initiatives such as system-generated query clarifying questions. Evaluating those systems at a large scale on the end task of IR is very challenging, requiring adequate datasets containing such interactions. However, current datasets only focus on either traditional ad-hoc IR tasks or query clarification tasks, the latter being usually seen as a reformulation task from the initial query. The only two datasets known to us that contain both document relevance judgments and the associated clarification interactions are Qulac and ClariQ. Both are based on the TREC Web Track 2009-12 collection, but cover a very limited number of topics (237 topics), far from being enough for training and testing conversational IR models. To fill the gap, we propose a methodology to automatically build large-scale conversational IR datasets from ad-hoc IR datasets in order to facilitate explorations on conversational IR. Our methodology is based on two processes: 1) generating query clarification interactions through query clarification and answer generators, and 2) augmenting ad-hoc IR datasets with simulated interactions. In this paper, we focus on MsMarco and augment it with query clarification and answer simulations. We perform a thorough evaluation showing the quality and the relevance of the generated interactions for each initial query. This paper shows the feasibility and utility of augmenting ad-hoc IR datasets for conversational IR

Feasibility report: Delivering case-study based learning using artificial intelligence and gaming technologies

This document describes an investigation into the technical feasibility of a game to support learning based on case studies. Information systems students using the game will conduct fact-finding interviews with virtual characters. We survey relevant technologies in computational linguistics and games. We assess the applicability of the various approaches and propose an architecture for the game based on existing techniques. We propose a phased development plan for the development of the game

Social talk capabilities for dialogue systems

Small talk capabilities are an important but very challenging extension to dialogue systems. Small talk (or “social talk”) refers to a kind of conversation, which does not focus on the exchange of information, but on the negotiation of social roles and situations. The goal of this thesis is to provide knowledge, processes and structures that can be used by dialogue systems to satisfactorily participate in social conversations. For this purpose the thesis presents research in the areas of natural-language understanding, dialogue management and error handling. Nine new models of social talk based on a data analysis of small talk conversations are described. The functionally-motivated and content-abstract models can be used for small talk conversations on various topics. The basic elements of the models consist of dialogue acts for social talk newly developed on basis of social science theory. The thesis also presents some conversation strategies for the treatment of so-called “out-of-domain” (OoD) utterances that can be used to avoid errors in the input understanding of dialogue systems. Additionally, the thesis describes a new extension to dialogue management that flexibly manages interwoven dialogue threads. The small talk models as well as the strategies for handling OoD utterances are encoded as computational dialogue threads

Social talk capabilities for dialogue systems

Small talk capabilities are an important but very challenging extension to dialogue systems. Small talk (or social talk) refers to a kind of conversation, which does not focus on the exchange of information, but on the negotiation of social roles and situations. The goal of this thesis is to provide knowledge, processes and structures that can be used by dialogue systems to satisfactorily participate in social conversations. For this purpose the thesis presents research in the areas of natural-language understanding, dialogue management and error handling. Nine new models of social talk based on a data analysis of small talk conversations are described. The functionally-motivated and content-abstract models can be used for small talk conversations on various topics. The basic elements of the models consist of dialogue acts for social talk newly developed on basis of social science theory. The thesis also presents some conversation strategies for the treatment of so-called out-of-domain (OoD) utterances that can be used to avoid errors in the input understanding of dialogue systems. Additionally, the thesis describes a new extension to dialogue management that flexibly manages interwoven dialogue threads. The small talk models as well as the strategies for handling OoD utterances are encoded as computational dialogue threads

Interactional Competence in L2 English Upper Secondary Oral Assessment: Exploring Teachers’ Understandings and Practices

I denne masteroppgaven har jeg undersøkt hvordan interaksjonell kompetanse (interactional competence, IC) er inkludert i læreres vurdering av muntlige ferdigheter i engelskfaget på videregående skole. Læreplanen (LK20) legger vekt på at muntlige ferdigheter i engelsk handler om å skape mening gjennom å lytte, tale og samtale. Det er derfor relevant å undersøke hvordan interaksjonell kompetanse blir inkludert og operasjonalisert i muntlig vurdering ettersom samhandling er en sentral del av elevenes muntlige ferdigheter. Læreres oppfatninger og praksis har blitt undersøkt gjennom en mixed methods tilnærming ved bruk av spørreundersøkelse, intervjuer og analyse av læreres oppgaver og vurderingskriterier for muntlig vurdering. Et teoretisk rammeverk for interaksjonell kompetanse tilpasset muntlig vurdering i engelsk på videregående skole blir presentert i studien og funnene har blitt analysert ut fra dette og annen relevant teori. Funnene fra studien viser at begrepet IC er ukjent for de fleste, men at lærere likevel oppfatter at denne kompetansen omhandler ferdigheter knyttet til samhandling med andre. Når lærerne blir bedt om å foreslå hvordan IC kan bli operasjonalisert i kriterier for muntlig vurdering, foreslår de fleste kriterier som reflekterer kjennetegnene på IC som er definert i oppgavens rammeverk. Det indikerer at det teoretiske rammeverket som er definert og foreslått i denne oppgaven, kan være passende å inkludere i muntlig vurdering i engelskfaget på videregående skole. Videre viser analysen at vurderingskriterier relatert til elevers kommunikative ferdigheter er i mindre grad språkliggjort og operasjonalisert enn kriterier som omhandler språk. Likevel rapporterer majoriteten av lærere å bruke gruppevurderinger som muntlig vurdering, en oppgaveform som inkluderer samhandling. Dette indikerer at det er behov for et rammeverk som kan konkretisere og operasjonalisere også deler av elevers muntlige ferdigheter som ikke er knyttet til språklige elementer. Studien finner også at det i varierende grad er samsvar mellom læreres oppfatninger og praksis rundt muntlig vurdering, noe som indikerer at det er potensiale for å konkretisere flere aspekter knyttet til elevers muntlige ferdigheter. Jeg håper at denne oppgaven kan bidra til å utvikle læreres begrepsapparat knyttet til samhandling og inspirere til å inkludere konkrete kriterier knyttet til de implisitte strategiene som er en viktig del av elevers muntlige og kommunikative kompetanse. IC er lite undersøkt i sammenheng med muntlig vurdering i engelskfaget på videregående skole og videre forskning er nødvendig for å kunne vurdere effekten av å inkludere IC i vurdering.Engelsk mastergradsoppgaveENG350MAHF-ENGMAHF-LÆF