4,358 research outputs found
PARADISE: A Framework for Evaluating Spoken Dialogue Agents
This paper presents PARADISE (PARAdigm for DIalogue System Evaluation), a
general framework for evaluating spoken dialogue agents. The framework
decouples task requirements from an agent's dialogue behaviors, supports
comparisons among dialogue strategies, enables the calculation of performance
over subdialogues and whole dialogues, specifies the relative contribution of
various factors to performance, and makes it possible to compare agents
performing different tasks by normalizing for task complexity.Comment: 10 pages, uses aclap, psfig, lingmacros, time
Active Learning for Dialogue Act Classification
Active learning techniques were employed for classification of dialogue acts over two dialogue corpora, the English human-human Switchboard corpus and the Spanish human-machine Dihana corpus. It is shown clearly that active learning improves on a baseline obtained through a passive learning approach to tagging the same data sets. An error reduction of 7% was obtained on Switchboard, while a factor 5 reduction in the amount of labeled data needed for classification was achieved on Dihana. The passive Support Vector Machine learner used as baseline in itself significantly improves the state of the art in dialogue act classification on both corpora. On Switchboard it gives a 31% error reduction compared to the previously best reported result
Semantic Processing of Out-Of-Vocabulary Words in a Spoken Dialogue System
One of the most important causes of failure in spoken dialogue systems is
usually neglected: the problem of words that are not covered by the system's
vocabulary (out-of-vocabulary or OOV words). In this paper a methodology is
described for the detection, classification and processing of OOV words in an
automatic train timetable information system. The various extensions that had
to be effected on the different modules of the system are reported, resulting
in the design of appropriate dialogue strategies, as are encouraging evaluation
results on the new versions of the word recogniser and the linguistic
processor.Comment: 4 pages, 2 eps figures, requires LaTeX2e, uses eurospeech.sty and
epsfi
Recommended from our members
Dialogue with computers: dialogue games in action
With the advent of digital personal assistants for mobile devices, systems that are marketed as engaging in (spoken) dialogue have reached a wider public than ever before. For a student of dialogue, this raises the question to what extent such systems are genuine dialogue partners. In order to address this question, this study proposes to use the concept of a dialogue game as an analytical tool. Thus, we reframe the question as asking for the dialogue games that such systems play. Our analysis, as applied to a number of landmark systems and illustrated with dialogue extracts, leads to a fine-grained classification of such systems. Drawing on this analysis, we propose that the uptake of future generations of more powerful dialogue systems will depend on whether they are self-validating. A self-validating dialogue system can not only talk and do things, but also discuss the why of what it says and does, and learn from such discussions
Towards Understanding Spontaneous Speech: Word Accuracy vs. Concept Accuracy
In this paper we describe an approach to automatic evaluation of both the
speech recognition and understanding capabilities of a spoken dialogue system
for train time table information. We use word accuracy for recognition and
concept accuracy for understanding performance judgement. Both measures are
calculated by comparing these modules' output with a correct reference answer.
We report evaluation results for a spontaneous speech corpus with about 10000
utterances. We observed a nearly linear relationship between word accuracy and
concept accuracy.Comment: 4 pages PS, Latex2e source importing 2 eps figures, uses icslp.cls,
caption.sty, psfig.sty; to appear in the Proceedings of the Fourth
International Conference on Spoken Language Processing (ICSLP 96
- …