34,169 research outputs found

    Survey on Evaluation Methods for Dialogue Systems

    Get PDF
    In this paper we survey the methods and concepts developed for the evaluation of dialogue systems. Evaluation is a crucial part during the development process. Often, dialogue systems are evaluated by means of human evaluations and questionnaires. However, this tends to be very cost and time intensive. Thus, much work has been put into finding methods, which allow to reduce the involvement of human labour. In this survey, we present the main concepts and methods. For this, we differentiate between the various classes of dialogue systems (task-oriented dialogue systems, conversational dialogue systems, and question-answering dialogue systems). We cover each class by introducing the main technologies developed for the dialogue systems and then by presenting the evaluation methods regarding this class

    Augmenting Ad-Hoc IR Dataset for Interactive Conversational Search

    Full text link
    A peculiarity of conversational search systems is that they involve mixed-initiatives such as system-generated query clarifying questions. Evaluating those systems at a large scale on the end task of IR is very challenging, requiring adequate datasets containing such interactions. However, current datasets only focus on either traditional ad-hoc IR tasks or query clarification tasks, the latter being usually seen as a reformulation task from the initial query. The only two datasets known to us that contain both document relevance judgments and the associated clarification interactions are Qulac and ClariQ. Both are based on the TREC Web Track 2009-12 collection, but cover a very limited number of topics (237 topics), far from being enough for training and testing conversational IR models. To fill the gap, we propose a methodology to automatically build large-scale conversational IR datasets from ad-hoc IR datasets in order to facilitate explorations on conversational IR. Our methodology is based on two processes: 1) generating query clarification interactions through query clarification and answer generators, and 2) augmenting ad-hoc IR datasets with simulated interactions. In this paper, we focus on MsMarco and augment it with query clarification and answer simulations. We perform a thorough evaluation showing the quality and the relevance of the generated interactions for each initial query. This paper shows the feasibility and utility of augmenting ad-hoc IR datasets for conversational IR

    Feasibility report: Delivering case-study based learning using artificial intelligence and gaming technologies

    Get PDF
    This document describes an investigation into the technical feasibility of a game to support learning based on case studies. Information systems students using the game will conduct fact-finding interviews with virtual characters. We survey relevant technologies in computational linguistics and games. We assess the applicability of the various approaches and propose an architecture for the game based on existing techniques. We propose a phased development plan for the development of the game

    Social talk capabilities for dialogue systems

    Get PDF
    Small talk capabilities are an important but very challenging extension to dialogue systems. Small talk (or “social talk”) refers to a kind of conversation, which does not focus on the exchange of information, but on the negotiation of social roles and situations. The goal of this thesis is to provide knowledge, processes and structures that can be used by dialogue systems to satisfactorily participate in social conversations. For this purpose the thesis presents research in the areas of natural-language understanding, dialogue management and error handling. Nine new models of social talk based on a data analysis of small talk conversations are described. The functionally-motivated and content-abstract models can be used for small talk conversations on various topics. The basic elements of the models consist of dialogue acts for social talk newly developed on basis of social science theory. The thesis also presents some conversation strategies for the treatment of so-called “out-of-domain” (OoD) utterances that can be used to avoid errors in the input understanding of dialogue systems. Additionally, the thesis describes a new extension to dialogue management that flexibly manages interwoven dialogue threads. The small talk models as well as the strategies for handling OoD utterances are encoded as computational dialogue threads

    Social talk capabilities for dialogue systems

    Get PDF
    Small talk capabilities are an important but very challenging extension to dialogue systems. Small talk (or social talk) refers to a kind of conversation, which does not focus on the exchange of information, but on the negotiation of social roles and situations. The goal of this thesis is to provide knowledge, processes and structures that can be used by dialogue systems to satisfactorily participate in social conversations. For this purpose the thesis presents research in the areas of natural-language understanding, dialogue management and error handling. Nine new models of social talk based on a data analysis of small talk conversations are described. The functionally-motivated and content-abstract models can be used for small talk conversations on various topics. The basic elements of the models consist of dialogue acts for social talk newly developed on basis of social science theory. The thesis also presents some conversation strategies for the treatment of so-called out-of-domain (OoD) utterances that can be used to avoid errors in the input understanding of dialogue systems. Additionally, the thesis describes a new extension to dialogue management that flexibly manages interwoven dialogue threads. The small talk models as well as the strategies for handling OoD utterances are encoded as computational dialogue threads

    Interactional Competence in L2 English Upper Secondary Oral Assessment: Exploring Teachers’ Understandings and Practices

    Get PDF
    I denne masteroppgaven har jeg undersĂžkt hvordan interaksjonell kompetanse (interactional competence, IC) er inkludert i lĂŠreres vurdering av muntlige ferdigheter i engelskfaget pĂ„ videregĂ„ende skole. LĂŠreplanen (LK20) legger vekt pĂ„ at muntlige ferdigheter i engelsk handler om Ă„ skape mening gjennom Ă„ lytte, tale og samtale. Det er derfor relevant Ă„ undersĂžke hvordan interaksjonell kompetanse blir inkludert og operasjonalisert i muntlig vurdering ettersom samhandling er en sentral del av elevenes muntlige ferdigheter. LĂŠreres oppfatninger og praksis har blitt undersĂžkt gjennom en mixed methods tilnĂŠrming ved bruk av spĂžrreundersĂžkelse, intervjuer og analyse av lĂŠreres oppgaver og vurderingskriterier for muntlig vurdering. Et teoretisk rammeverk for interaksjonell kompetanse tilpasset muntlig vurdering i engelsk pĂ„ videregĂ„ende skole blir presentert i studien og funnene har blitt analysert ut fra dette og annen relevant teori. Funnene fra studien viser at begrepet IC er ukjent for de fleste, men at lĂŠrere likevel oppfatter at denne kompetansen omhandler ferdigheter knyttet til samhandling med andre. NĂ„r lĂŠrerne blir bedt om Ă„ foreslĂ„ hvordan IC kan bli operasjonalisert i kriterier for muntlig vurdering, foreslĂ„r de fleste kriterier som reflekterer kjennetegnene pĂ„ IC som er definert i oppgavens rammeverk. Det indikerer at det teoretiske rammeverket som er definert og foreslĂ„tt i denne oppgaven, kan vĂŠre passende Ă„ inkludere i muntlig vurdering i engelskfaget pĂ„ videregĂ„ende skole. Videre viser analysen at vurderingskriterier relatert til elevers kommunikative ferdigheter er i mindre grad sprĂ„kliggjort og operasjonalisert enn kriterier som omhandler sprĂ„k. Likevel rapporterer majoriteten av lĂŠrere Ă„ bruke gruppevurderinger som muntlig vurdering, en oppgaveform som inkluderer samhandling. Dette indikerer at det er behov for et rammeverk som kan konkretisere og operasjonalisere ogsĂ„ deler av elevers muntlige ferdigheter som ikke er knyttet til sprĂ„klige elementer. Studien finner ogsĂ„ at det i varierende grad er samsvar mellom lĂŠreres oppfatninger og praksis rundt muntlig vurdering, noe som indikerer at det er potensiale for Ă„ konkretisere flere aspekter knyttet til elevers muntlige ferdigheter. Jeg hĂ„per at denne oppgaven kan bidra til Ă„ utvikle lĂŠreres begrepsapparat knyttet til samhandling og inspirere til Ă„ inkludere konkrete kriterier knyttet til de implisitte strategiene som er en viktig del av elevers muntlige og kommunikative kompetanse. IC er lite undersĂžkt i sammenheng med muntlig vurdering i engelskfaget pĂ„ videregĂ„ende skole og videre forskning er nĂždvendig for Ă„ kunne vurdere effekten av Ă„ inkludere IC i vurdering.Engelsk mastergradsoppgaveENG350MAHF-ENGMAHF-LÆF
    • 

    corecore