Design and evaluation of acceleration strategies for speeding up the development of dialog applications

In this paper, we describe a complete development platform that features different innovative acceleration strategies, not included in any other current platform, that simplify and speed up the definition of the different elements required to design a spoken dialog service. The proposed accelerations are mainly based on using the information from the backend database schema and contents, as well as cumulative information produced throughout the different steps in the design. Thanks to these accelerations, the interaction between the designer and the platform is improved, and in most cases the design is reduced to simple confirmations of the “proposals” that the platform dynamically provides at each step. In addition, the platform provides several other accelerations such as configurable templates that can be used to define the different tasks in the service or the dialogs to obtain or show information to the user, automatic proposals for the best way to request slot contents from the user (i.e. using mixed-initiative forms or directed forms), an assistant that offers the set of more probable actions required to complete the definition of the different tasks in the application, or another assistant for solving specific modality details such as confirmations of user answers or how to present them the lists of retrieved results after querying the backend database. Additionally, the platform also allows the creation of speech grammars and prompts, database access functions, and the possibility of using mixed initiative and over-answering dialogs. In the paper we also describe in detail each assistant in the platform, emphasizing the different kind of methodologies followed to facilitate the design process at each one. Finally, we describe the results obtained in both a subjective and an objective evaluation with different designers that confirm the viability, usefulness, and functionality of the proposed accelerations. Thanks to the accelerations, the design time is reduced in more than 56% and the number of keystrokes by 84%

Mining Mixed-Initiative Dialogs

Human-computer dialogs are an important vehicle through which to produce a rich and compelling form of human-computer interaction. We view the specification of a human-computer dialog as a set of sequences of progressive interactions between a user and a computer system, and mine partially ordered sets, which correspond to mixing dialog initiative, embedded in these sets of sequences—a process we refer to as dialog mining—because partially ordered sets can be advantageously exploited to reduce the control complexity of a dialog implementation. Our mining losslessly compresses the specification of a dialog. We describe our mining algorithm and report the results of a simulation-oriented evaluation. Our algorithm is sound, and our results indicate that it can compress nearly all dialog specifications, and some to a high degree. This work is part of broader research on the specification and implementation of mixed-initiative dialogs

A Language-Based Model for Specifying and Staging Mixed-Initiative Dialogs

Specifying and implementing flexible human-computer dialogs, such as those used in kiosks, is complex because of the numerous and varied directions in which each user might steer a dialog. The objective of this research is to improve dialog specification and implementation. To do so we developed a model for specifying and staging mixed-initiative dialogs. The model involves a dialog authoring notation, based on concepts from programming languages, for specifying a variety of unsolicited reporting, mixed-initiative dialogs in a concise representation that serves as a design for dialog implementation. Guided by this foundation, we built a dialog staging engine which operationalizes dialogs specified in this notation. The model, notation, and engine help automate the engineering of mixed-initiative dialog systems. These results also provide a proof-of-concept for dialog specification and implementation from the perspective of theoretical programming languages. The ubiquity of dialogs in domains such as travel, education, and health care with the increased use of interactive voice-response systems and virtual environments provide a fertile landscape for further investigation of these results

Bringing together commercial and academic perspectives for the development of intelligent AmI interfaces

The users of Ambient Intelligence systems expect an intelligent behavior from their environment, receiving adapted and easily accessible services and functionality. This can only be possible if the communication between the user and the system is carried out through an interface that is simple (i.e. which does not have a steep learning curve), fluid (i.e. the communication takes place rapidly and effectively), and robust (i.e. the system understands the user correctly). Natural language interfaces such as dialog systems combine the previous three requisites, as they are based on a spoken conversation between the user and the system that resembles human communication. The current industrial development of commercial dialog systems deploys robust interfaces in strictly defined application domains. However, commercial systems have not yet adopted the new perspective proposed in the academic settings, which would allow straightforward adaptation of these interfaces to various application domains. This would be highly beneficial for their use in AmI settings as the same interface could be used in varying environments. In this paper, we propose a new approach to bridge the gap between the academic and industrial perspectives in order to develop dialog systems using an academic paradigm while employing the industrial standards, which makes it possible to obtain new generation interfaces without the need for changing the already existing commercial infrastructures. Our proposal has been evaluated with the successful development of a real dialog system that follows our proposed approach to manage dialog and generates code compliant with the industry-wide standard VoiceXML.Research funded by projects CICYT TIN2011-28620-C02-01, CICYT TEC2011-28626-C02-02, CAM CONTEXTS (S2009/TIC-1485), and DPS2008- 07029-C02-02.Publicad

An evaluation paradigm for spoken dialog systems based on crowdsourcing and collaborative filtering.

Yang, Zhaojun.Thesis (M.Phil.)--Chinese University of Hong Kong, 2011.Includes bibliographical references (p. 92-99).Abstracts in English and Chinese.Chapter 1 --- Introduction --- p.1Chapter 1.1 --- SDS Architecture --- p.1Chapter 1.2 --- Dialog Model --- p.3Chapter 1.3 --- SDS Evaluation --- p.4Chapter 1.4 --- Thesis Outline --- p.7Chapter 2 --- Previous Work --- p.9Chapter 2.1 --- Approaches to Dialog Modeling --- p.9Chapter 2.1.1 --- Handcrafted Dialog Modeling --- p.9Chapter 2.1.2 --- Statistical Dialog Modeling --- p.12Chapter 2.2 --- Evaluation Metrics --- p.16Chapter 2.2.1 --- Subjective User Judgments --- p.17Chapter 2.2.2 --- Interaction Metrics --- p.18Chapter 2.3 --- The PARADISE Framework --- p.19Chapter 2.4 --- Chapter Summary --- p.22Chapter 3 --- Implementation of a Dialog System based on POMDP --- p.23Chapter 3.1 --- Partially Observable Markov Decision Processes (POMDPs) --- p.24Chapter 3.1.1 --- Formal Definition --- p.24Chapter 3.1.2 --- Value Iteration --- p.26Chapter 3.1.3 --- Point-based Value Iteration --- p.27Chapter 3.1.4 --- A Toy Example of POMDP: The NaiveBusInfo System --- p.27Chapter 3.2 --- The SDS-POMDP Model --- p.31Chapter 3.3 --- Composite Summary Point-based Value Iteration (CSPBVI) --- p.33Chapter 3.4 --- Application of SDS-POMDP Model: The Buslnfo System --- p.35Chapter 3.4.1 --- System Description --- p.35Chapter 3.4.2 --- Demonstration Description --- p.39Chapter 3.5 --- Chapter Summary --- p.42Chapter 4 --- Collecting User Judgments on Spoken Dialogs with Crowdsourcing --- p.46Chapter 4.1 --- Dialog Corpus and Automatic Dialog Classification --- p.47Chapter 4.2 --- User Judgments Collection with Crowdsourcing --- p.50Chapter 4.2.1 --- HITs on Dialog Evaluation --- p.51Chapter 4.2.2 --- HITs on Inter-rater Agreement --- p.53Chapter 4.2.3 --- Approval of Ratings --- p.54Chapter 4.3 --- Collected Results and Analysis --- p.55Chapter 4.3.1 --- Approval Rates and Comments from Mturk Workers --- p.55Chapter 4.3.2 --- Consistency between Automatic Dialog Classification and Manual Ratings --- p.57Chapter 4.3.3 --- Inter-rater Agreement Among Workers --- p.60Chapter 4.4 --- Comparing Experts to Non-experts --- p.64Chapter 4.4.1 --- Inter-rater Agreement on the Let's Go! System --- p.65Chapter 4.4.2 --- Consistency Between Expert and Non-expert Annotations on SDC Systems --- p.66Chapter 4.5 --- Chapter Summary --- p.68Chapter 5 --- Collaborative Filtering for Performance Prediction --- p.70Chapter 5.1 --- Item-Based Collaborative Filtering --- p.71Chapter 5.2 --- CF Model for User Satisfaction Prediction --- p.72Chapter 5.2.1 --- ICFM for User Satisfaction Prediction --- p.72Chapter 5.2.2 --- Extended ICFM for User Satisfaction Prediction --- p.73Chapter 5.3 --- Extraction of Interaction Features --- p.74Chapter 5.4 --- Experimental Results and Analysis --- p.76Chapter 5.4.1 --- Prediction of User Satisfaction --- p.76Chapter 5.4.2 --- Analysis of Prediction Results --- p.79Chapter 5.5 --- Verifying the Generalibility of CF Model --- p.81Chapter 5.6 --- Evaluation of The Buslnfo System --- p.86Chapter 5.7 --- Chapter Summary --- p.87Chapter 6 --- Conclusions and Future Work --- p.89Chapter 6.1 --- Thesis Summary --- p.89Chapter 6.2 --- Future Work --- p.90Bibliography --- p.9