Wizundry: A Cooperative Wizard of Oz Platform for Simulating Future Speech-based Interfaces with Multiple Wizards

Wizard of Oz (WoZ) as a prototyping method has been used to simulate intelligent user interfaces, particularly for speech-based systems. However, as our societies' expectations on artificial intelligence (AI) grows, the question remains whether a single Wizard is sufficient for it to simulate smarter systems and more complex interactions. Optimistic visions of 'what artificial intelligence (AI) can do' places demands on WoZ platforms to simulate smarter systems and more complex interactions. This raises the question of whether the typical approach of employing a single Wizard is sufficient. Moreover, while existing work has employed multiple Wizards in WoZ studies, a multi-Wizard approach has not been systematically studied in terms of feasibility, effectiveness, and challenges. We offer Wizundry, a real-time, web-based WoZ platform that allows multiple Wizards to collaboratively operate a speech-to-text based system remotely. We outline the design and technical specifications of our open-source platform, which we iterated over two design phases. We report on two studies in which participant-Wizards were tasked with negotiating how to cooperatively simulate an interface that can handle natural speech for dictation and text editing as well as other intelligent text processing tasks. We offer qualitative findings on the Multi-Wizard experience for Dyads and Triads of Wizards. Our findings reveal the promises and challenges of the multi-Wizard approach and open up new research questions.Comment: 34 page

A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version

During the past decade, several areas of speech and language understanding have witnessed substantial breakthroughs from the use of data-driven models. In the area of dialogue systems, the trend is less obvious, and most practical systems are still built through significant engineering and expert knowledge. Nevertheless, several recent results suggest that data-driven approaches are feasible and quite promising. To facilitate research in this area, we have carried out a wide survey of publicly available datasets suitable for data-driven learning of dialogue systems. We discuss important characteristics of these datasets, how they can be used to learn diverse dialogue strategies, and their other potential uses. We also examine methods for transfer learning between datasets and the use of external knowledge. Finally, we discuss appropriate choice of evaluation metrics for the learning objective

VALUTARE I SISTEMI FLESSIBILI: UN APPROCCIO GLOBALE ALLA HCI

1995/1996VIII Ciclo1966Versione digitalizzata della tesi di dottorato cartacea

A flexible and reusable framework for dialogue and action management in multi-party discourse

This thesis describes a model for goal-directed dialogue and activity control in real-time for multiple conversation participants that can be human users or virtual characters in multimodal dialogue systems and a framework implementing the model. It is concerned with two genres: task-oriented systems and interactive narratives. The model is based on a representation of participant behavior on three hierarchical levels: dialogue acts, dialogue games, and activities. Dialogue games allow to take advantage of social conventions and obligations to model the basic structure of dialogues. The interactions can be specified and implemented using reoccurring elementary building blocks. Expectations about future behavior of other participants are derived from the state of active dialogue games; this can be useful for, e. g., input disambiguation. The knowledge base of the system is defined in an ontological format and allows individual knowledge and personal traits for the characters. The Conversational Behavior Generation Framework implements the model. It coordinates a set of conversational dialogue engines (CDEs), where each participant is represented by one CDE. The virtual characters can act autonomously, or semi-autonomously follow goals assigned by an external story module (Narrative Mode). The framework allows combining alternative specification methods for the virtual characters\u27; activities (implementation in a general-purpose programming language, by plan operators, or in the specification language Lisa that was developed for the model). The practical viability of the framework was tested and demonstrated via the realization of three systems with different purposes and scope.Diese Arbeit beschreibt ein Modell für zielgesteuerte Dialog- und Ablaufsteuerung in Echtzeit für beliebig viele menschliche Konversationsteilnehmer und virtuelle Charaktere in multimodalen Dialogsystemen, sowie eine Softwareumgebung, die das Modell implementiert. Dabei werden zwei Genres betrachtet: Task-orientierte Systeme und interaktive Erzählungen. Das Modell basiert auf einer Repräsentation des Teilnehmerverhaltens auf drei hierarchischen Ebenen: Dialogakte, Dialogspiele und Aktivitäten. Dialogspiele erlauben es, soziale Konventionen und Obligationen auszunutzen, um die Dialoge grundlegend zu strukturieren. Die Interaktionen können unter Verwendung wiederkehrender elementarer Bausteine spezifiziert und programmtechnisch implementiert werden. Aus dem Zustand aktiver Dialogspiele werden Erwartungen an das zukünftige Verhalten der Dialogpartner abgeleitet, die beispielsweise für die Desambiguierung von Eingaben von Nutzen sein können. Die Wissensbasis des Systems ist in einem ontologischen Format definiert und ermöglicht individuelles Wissen und persönliche Merkmale für die Charaktere. Das Conversational Behavior Generation Framework implementiert das Modell. Es koordiniert eine Menge von Dialog-Engines (CDEs), wobei jedem Teilnehmer eine CDE zugeordet wird, die ihn repräsentiert. Die virtuellen Charaktere können autonom oder semi-autonom nach den Zielvorgaben eines externen Storymoduls agieren (Narrative Mode). Das Framework erlaubt die Kombination alternativer Spezifikationsarten für die Aktivitäten der virtuellen Charaktere (Implementierung in einer allgemeinen Programmiersprache, durch Planoperatoren oder in der für das Modell entwickelten Spezifikationssprache Lisa). Die Praxistauglichkeit des Frameworks wurde anhand der Realisierung dreier Systeme mit unterschiedlichen Zielsetzungen und Umfang erprobt und erwiesen

From dialogue to dialogue system: the other-initiated repairs

Doktoriväitekiri käsitleb suhtlusprobleeme inimese ja arvuti vahelistes ning inimeste-vahelistes dialoogides. Analüüsitakse lauseid (ehk lausungeid) nagu nt ma ei saa aru (mittemõistmine), kas sa ütlesid nii? (üleküsimine), kas sa mõtlesid seda? (ümbersõnastamine). Uurimuse eesmärgiks on selgitada välja, milline korrapära ehk millised mustrid esinevad dialoogide lausungite ülesehituses. Mõistmaks, kuidas suhtlusprobleemidest keeleliselt teada antakse, analüüsitakse erinevat liiki eestikeelseid dialooge Eesti dialoogikorpusest. Sealhulgas on dialoogide kogumiseks viidud läbi nn Võlur OZ-i eksperimendid, kus katseisikud arvavad end suhtlevat arvutiga, kuid tegelikult suhtlevad interneti vahendusel teise inimesega. Ulatusliku eestikeelse dialoogimaterjali alusel uuritakse töös, mis tekitab suhtlusprobleeme, kuidas tekkinud probleemist suhtluspartnerile märku antakse ja kuidas suhtlusprobleeme lahendatakse. Tehakse ka muid tähelepanekuid inimese ja arvuti vahelise loomuliku suhtluse tõhustamise kohta. Uurimuse tulemuseks on mustrid ja kui-siis reeglid, mille kasutamine inimesega eesti keeles suhtlevas dialoogsüsteemis aitaks kaasa ladusamale suhtlemisele arvutiga. Suhtlusprobleemidega toimetulek on üks dialoogsüsteemide loomise ja eduka töötamise põhiküsimustest. Doktoriväitekirja rakenduslik panus on pakkuda dialoogsüsteemide loojatele konkreetseid soovitusi selle küsimuse lahendamiseks.This dissertation analyzes human-computer interaction problems and communication problems in human-human dialogues. Under examination are utterances such as I do not understand (i.e. non-understanding), did you say that? (clarification), and did you mean that? (reformulation). The research aims to find answers as to which regularities (i.e. patterns) appear in the structure of utterances. To understand how interaction/communication problems are linguistically intimated, different kinds of dialogues are analyzed from the Estonian Dialogue Corpus. The data also includes dialogues from the Wizard of OZ (WOZ) experiments, where test participants (users) thought they were interacting with the dialogue system, but they were actually interacting with an experimenter (WOZ) via the Internet. The study examines the following questions: what causes communication problems; how are communication problems indicated; and how are communication problems resolved? The dissertation also contains other observations about human-computer interaction and communication between people. The results of the thesis are patterns and if-then rules for dialogue systems. The management of communication problems is one of the main issues when attempting to create and successfully run a dialogue system. One applied contribution of the dissertation is to offer creators of dialogue systems concrete advice in solving this issue