4,641 research outputs found
Survey on Evaluation Methods for Dialogue Systems
In this paper we survey the methods and concepts developed for the evaluation
of dialogue systems. Evaluation is a crucial part during the development
process. Often, dialogue systems are evaluated by means of human evaluations
and questionnaires. However, this tends to be very cost and time intensive.
Thus, much work has been put into finding methods, which allow to reduce the
involvement of human labour. In this survey, we present the main concepts and
methods. For this, we differentiate between the various classes of dialogue
systems (task-oriented dialogue systems, conversational dialogue systems, and
question-answering dialogue systems). We cover each class by introducing the
main technologies developed for the dialogue systems and then by presenting the
evaluation methods regarding this class
A Personalized System for Conversational Recommendations
Searching for and making decisions about information is becoming increasingly
difficult as the amount of information and number of choices increases.
Recommendation systems help users find items of interest of a particular type,
such as movies or restaurants, but are still somewhat awkward to use. Our
solution is to take advantage of the complementary strengths of personalized
recommendation systems and dialogue systems, creating personalized aides. We
present a system -- the Adaptive Place Advisor -- that treats item selection as
an interactive, conversational process, with the program inquiring about item
attributes and the user responding. Individual, long-term user preferences are
unobtrusively obtained in the course of normal recommendation dialogues and
used to direct future conversations with the same user. We present a novel user
model that influences both item search and the questions asked during a
conversation. We demonstrate the effectiveness of our system in significantly
reducing the time and number of interactions required to find a satisfactory
item, as compared to a control group of users interacting with a non-adaptive
version of the system
Predicting Causes of Reformulation in Intelligent Assistants
Intelligent assistants (IAs) such as Siri and Cortana conversationally
interact with users and execute a wide range of actions (e.g., searching the
Web, setting alarms, and chatting). IAs can support these actions through the
combination of various components such as automatic speech recognition, natural
language understanding, and language generation. However, the complexity of
these components hinders developers from determining which component causes an
error. To remove this hindrance, we focus on reformulation, which is a useful
signal of user dissatisfaction, and propose a method to predict the
reformulation causes. We evaluate the method using the user logs of a
commercial IA. The experimental results have demonstrated that features
designed to detect the error of a specific component improve the performance of
reformulation cause detection.Comment: 11 pages, 2 figures, accepted as a long paper for SIGDIAL 201
Frames: A Corpus for Adding Memory to Goal-Oriented Dialogue Systems
This paper presents the Frames dataset (Frames is available at
http://datasets.maluuba.com/Frames), a corpus of 1369 human-human dialogues
with an average of 15 turns per dialogue. We developed this dataset to study
the role of memory in goal-oriented dialogue systems. Based on Frames, we
introduce a task called frame tracking, which extends state tracking to a
setting where several states are tracked simultaneously. We propose a baseline
model for this task. We show that Frames can also be used to study memory in
dialogue management and information presentation through natural language
generation
- …