1,323 research outputs found
Augmenting Ad-Hoc IR Dataset for Interactive Conversational Search
A peculiarity of conversational search systems is that they involve
mixed-initiatives such as system-generated query clarifying questions.
Evaluating those systems at a large scale on the end task of IR is very
challenging, requiring adequate datasets containing such interactions. However,
current datasets only focus on either traditional ad-hoc IR tasks or query
clarification tasks, the latter being usually seen as a reformulation task from
the initial query. The only two datasets known to us that contain both document
relevance judgments and the associated clarification interactions are Qulac and
ClariQ. Both are based on the TREC Web Track 2009-12 collection, but cover a
very limited number of topics (237 topics), far from being enough for training
and testing conversational IR models. To fill the gap, we propose a methodology
to automatically build large-scale conversational IR datasets from ad-hoc IR
datasets in order to facilitate explorations on conversational IR. Our
methodology is based on two processes: 1) generating query clarification
interactions through query clarification and answer generators, and 2)
augmenting ad-hoc IR datasets with simulated interactions. In this paper, we
focus on MsMarco and augment it with query clarification and answer
simulations. We perform a thorough evaluation showing the quality and the
relevance of the generated interactions for each initial query. This paper
shows the feasibility and utility of augmenting ad-hoc IR datasets for
conversational IR
Mining Mixed-Initiative Dialogs
Human-computer dialogs are an important vehicle through which to produce a rich and compelling form of human-computer interaction. We view the specification of a human-computer dialog as a set of sequences of progressive interactions between a user and a computer system, and mine partially ordered sets, which correspond to mixing dialog initiative, embedded in these sets of sequencesâa process we refer to as dialog miningâbecause partially ordered sets can be advantageously exploited to reduce the control complexity of a dialog implementation. Our mining losslessly compresses the specification of a dialog. We describe our mining algorithm and report the results of a simulation-oriented evaluation. Our algorithm is sound, and our results indicate that it can compress nearly all dialog specifications, and some to a high degree. This work is part of broader research on the specification and implementation of mixed-initiative dialogs
Survey on Evaluation Methods for Dialogue Systems
In this paper we survey the methods and concepts developed for the evaluation
of dialogue systems. Evaluation is a crucial part during the development
process. Often, dialogue systems are evaluated by means of human evaluations
and questionnaires. However, this tends to be very cost and time intensive.
Thus, much work has been put into finding methods, which allow to reduce the
involvement of human labour. In this survey, we present the main concepts and
methods. For this, we differentiate between the various classes of dialogue
systems (task-oriented dialogue systems, conversational dialogue systems, and
question-answering dialogue systems). We cover each class by introducing the
main technologies developed for the dialogue systems and then by presenting the
evaluation methods regarding this class
Robust Dialog Management Through A Context-centric Architecture
This dissertation presents and evaluates a method of managing spoken dialog interactions with a robust attention to fulfilling the human userâs goals in the presence of speech recognition limitations. Assistive speech-based embodied conversation agents are computer-based entities that interact with humans to help accomplish a certain task or communicate information via spoken input and output. A challenging aspect of this task involves open dialog, where the user is free to converse in an unstructured manner. With this style of input, the machineâs ability to communicate may be hindered by poor reception of utterances, caused by a userâs inadequate command of a language and/or faults in the speech recognition facilities. Since a speech-based input is emphasized, this endeavor involves the fundamental issues associated with natural language processing, automatic speech recognition and dialog system design. Driven by ContextBased Reasoning, the presented dialog manager features a discourse model that implements mixed-initiative conversation with a focus on the userâs assistive needs. The discourse behavior must maintain a sense of generality, where the assistive nature of the system remains constant regardless of its knowledge corpus. The dialog manager was encapsulated into a speech-based embodied conversation agent platform for prototyping and testing purposes. A battery of user trials was performed on this agent to evaluate its performance as a robust, domain-independent, speech-based interaction entity capable of satisfying the needs of its users
Chain-based recommendations
Recommender systems are discovery tools. Typically, they infer a user's preferences from her behaviour and make personalized suggestions. They are one response to the overwhelming choices that the Web affords its users. Recent studies have shown that a user of a recommender system is more likely to be satisfied by the recommendations if the system provides explanations that allow the user to understand their rationale, and if the system allows the user to provide feedback on the recommendations to improve the next round of recommendations so that they take account of the user's ephemeral needs. The goal of this dissertation is to introduce a new recommendation framework that offers a better user experience, while giving quality recommendations. It works on content-based principles and addresses both the issues identified in the previous paragraph, i.e.\ explanations and recommendation feedback. We instantiate our framework to produce two recommendation engines, each focusing on one of the themes: (i) the role of explanations in producing recommendations, and (ii) helping users to articulate their ephemeral needs. For the first theme, we show how to unify recommendation and explanation to a greater degree than has been achieved hitherto. This results in an approach that enables the system to find relevant recommendations with explanations that have a high degree of both fidelity and interpretability. For the second theme, we show how to allow users to steer the recommendation process using a conversational recommender system. Our approach allows the user to reveal her short-term preferences and have them taken into account by the system and thus assists her in making a good decision efficiently. Early work on conversational recommender systems considers the case where the candidate items have structured descriptions (e.g.\ sets of attribute-value pairs). Our new approach works in the case where items have unstructured descriptions (e.g.\ sets of genres or tags). For each of the two themes, we describe the problem settings, the state-of-the-art, our system design and our experiment design. We evaluate each system using both offline analyses as well as user trials in a movie recommendation domain. We find that the proposed systems provide relevant recommendations that also have a high degree of serendipity, low popularity-bias and high diversity
A Language-Based Model for Specifying and Staging Mixed-Initiative Dialogs
Specifying and implementing flexible human-computer dialogs, such as those used in kiosks, is complex because of the numerous and varied directions in which each user might steer a dialog. The objective of this research is to improve dialog specification and implementation. To do so we developed a model for specifying and staging mixed-initiative dialogs. The model involves a dialog authoring notation, based on concepts from programming languages, for specifying a variety of unsolicited reporting, mixed-initiative dialogs in a concise representation that serves as a design for dialog implementation. Guided by this foundation, we built a dialog staging engine which operationalizes dialogs specified in this notation. The model, notation, and engine help automate the engineering of mixed-initiative dialog systems. These results also provide a proof-of-concept for dialog specification and implementation from the perspective of theoretical programming languages. The ubiquity of dialogs in domains such as travel, education, and health care with the increased use of interactive voice-response systems and virtual environments provide a fertile landscape for further investigation of these results
Survey on evaluation methods for dialogue
In this paper we survey the methods and concepts developed for the evaluation of dialogue systems. Evaluation is a crucial part during the development process. Often, dialogue systems are evaluated by means of human evaluations and questionnaires. However, this tends to be very cost and time intensive. Thus, much work has been put into finding methods, which allow to reduce the involvement of human labour. In this survey, we present the main concepts and methods. For this, we differentiate between the various classes of dialogue systems (task-oriented dialogue systems, conversational dialogue systems, and question-answering dialogue systems). We cover each class by introducing the main technologies developed for the dialogue systems and then by presenting the evaluation methods regarding this class
Cross-language Information Retrieval
Two key assumptions shape the usual view of ranked retrieval: (1) that the
searcher can choose words for their query that might appear in the documents
that they wish to see, and (2) that ranking retrieved documents will suffice
because the searcher will be able to recognize those which they wished to find.
When the documents to be searched are in a language not known by the searcher,
neither assumption is true. In such cases, Cross-Language Information Retrieval
(CLIR) is needed. This chapter reviews the state of the art for CLIR and
outlines some open research questions.Comment: 49 pages, 0 figure
- âŠ