Continuous Interaction with a Virtual Human

Attentive Speaking and Active Listening require that a Virtual Human be capable of simultaneous perception/interpretation and production of communicative behavior. A Virtual Human should be able to signal its attitude and attention while it is listening to its interaction partner, and be able to attend to its interaction partner while it is speaking – and modify its communicative behavior on-the-fly based on what it perceives from its partner. This report presents the results of a four week summer project that was part of eNTERFACE’10. The project resulted in progress on several aspects of continuous interaction such as scheduling and interrupting multimodal behavior, automatic classification of listener responses, generation of response eliciting behavior, and models for appropriate reactions to listener responses. A pilot user study was conducted with ten participants. In addition, the project yielded a number of deliverables that are released for public access

Modelling Users, Intentions, and Structure in Spoken Dialog

We outline how utterances in dialogs can be interpreted using a partial first order logic. We exploit the capability of this logic to talk about the truth status of formulae to define a notion of coherence between utterances and explain how this coherence relation can serve for the construction of AND/OR trees that represent the segmentation of the dialog. In a BDI model we formalize basic assumptions about dialog and cooperative behaviour of participants. These assumptions provide a basis for inferring speech acts from coherence relations between utterances and attitudes of dialog participants. Speech acts prove to be useful for determining dialog segments defined on the notion of completing expectations of dialog participants. Finally, we sketch how explicit segmentation signalled by cue phrases and performatives is covered by our dialog model.Comment: 17 page

Natural language response generation in mixed-initiative dialogs.

Yip Wing Lin Winnie.Thesis (M.Phil.)--Chinese University of Hong Kong, 2004.Includes bibliographical references (leaves 102-105).Abstracts in English and Chinese.Chapter 1 --- Introduction --- p.1Chapter 1.1 --- Overview --- p.1Chapter 1.2 --- Thesis Goals --- p.3Chapter 1.3 --- Thesis Outline --- p.5Chapter 2 --- Background --- p.6Chapter 2.1 --- Natural Language Generation --- p.6Chapter 2.1.1 --- Template-based Approach --- p.7Chapter 2.1.2 --- Rule-based Approach --- p.8Chapter 2.1.3 --- Statistical Approach --- p.9Chapter 2.1.4 --- Hybrid Approach --- p.10Chapter 2.1.5 --- Machine Learning Approach --- p.11Chapter 2.2 --- Evaluation Method --- p.12Chapter 2.2.1 --- Cooperative Principles --- p.13Chapter 2.3 --- Chapter Summary --- p.13Chapter 3 --- Natural Language Understanding --- p.14Chapter 3.1 --- The CUHK Restaurant Domain --- p.15Chapter 3.2 --- "Task Goals, Dialog Acts, Concept Categories and Annotation" --- p.17Chapter 3.2.1 --- Task Goals (TGs) and Dialog Acts (DAs) --- p.17Chapter 3.2.2 --- Concept Categories (CTG/CDA) --- p.20Chapter 3.2.3 --- Utterance Segmentation and Annotation --- p.21Chapter 3.3 --- Task Goal and Dialog Act Identification --- p.22Chapter 3.3.1 --- Belief Networks Development --- p.22Chapter 3.3.2 --- Task Goal and Dialog Act Inference --- p.24Chapter 3.3.3 --- Network Dimensions --- p.25Chapter 3.4 --- Chapter Summary --- p.29Chapter 4 --- Automatic Utterance Segmentation --- p.30Chapter 4.1 --- Utterance Definition --- p.31Chapter 4.2 --- Segmentation Procedure --- p.33Chapter 4.2.1 --- Tokenization --- p.35Chapter 4.2.2 --- POS Tagging --- p.36Chapter 4.2.3 --- Multi-Parser Architecture (MPA) Language Parsing --- p.38Chapter 4.2.4 --- Top-down Generalized Representation --- p.40Chapter 4.3 --- Evaluation --- p.47Chapter 4.3.1 --- Results --- p.47Chapter 4.3.2 --- Analysis --- p.48Chapter 4.4 --- Chapter Summary --- p.50Chapter 5 --- Natural Language Response Generation --- p.52Chapter 5.1 --- System Overview --- p.52Chapter 5.2 --- Corpus-derived Dialog State Transition Rules --- p.55Chapter 5.3 --- Hand-designed Text Generation Templates --- p.56Chapter 5.4 --- Performance Evaluation --- p.59Chapter 5.4.1 --- Task Completion Rate --- p.61Chapter 5.4.2 --- Grice's Maxims and Perceived User Satisfaction --- p.62Chapter 5.4.3 --- Error Analysis --- p.64Chapter 5.5 --- Chapter Summary --- p.65Chapter 6 --- Bilingual Response Generation using Semi-Automatically- Induced Response Templates --- p.67Chapter 6.1 --- Response Data --- p.68Chapter 6.2 --- Semi-Automatic Grammar Induction --- p.69Chapter 6.2.1 --- Agglomerative Clustering --- p.69Chapter 6.2.2 --- Parameters Selection --- p.70Chapter 6.3 --- Application to Response Grammar Induction --- p.71Chapter 6.3.1 --- Parameters Selection --- p.73Chapter 6.3.2 --- Unsupervised Grammar Induction --- p.76Chapter 6.3.3 --- Post-processing --- p.80Chapter 6.3.4 --- Prior Knowledge Injection --- p.82Chapter 6.4 --- Response Templates Generation --- p.84Chapter 6.4.1 --- Induced Response Grammar --- p.84Chapter 6.4.2 --- Template Formation --- p.84Chapter 6.4.3 --- Bilingual Response Templates --- p.89Chapter 6.5 --- Evaluation --- p.89Chapter 6.5.1 --- "Task Completion Rate, Grice's Maxims and User Sat- isfaction" --- p.91Chapter 6.6 --- Chapter Summary --- p.94Chapter 7 --- Conclusion --- p.96Chapter 7.1 --- Summary --- p.96Chapter 7.2 --- Contributions --- p.98Chapter 7.3 --- Future Work --- p.100Bibliography --- p.102Chapter A --- Domain-Specific Task Goals in the CUHK Restaurants Do- main --- p.107Chapter B --- Full List of VERBMOBIL-2 Dialog Acts --- p.109Chapter C --- Dialog Acts for Customer Requests and Waiter Responsesin the CUHK Restaurants Domain --- p.111Chapter D --- Grammar for Task Goal and Dialog Act Identification --- p.116Chapter E --- Utterance Definition --- p.119Chapter F --- Dialog State Transition Rules --- p.121Chapter G --- Full List of Templates Selection Conditions --- p.125Chapter H --- Hand-designed Text Generation Templates --- p.130Chapter I --- Evaluation Test Questionnaire for Dialog System in the CUHK Restaurant Domain --- p.135Chapter J --- POS Tags --- p.137Chapter K --- Full List of Lexicon and contextual rule modifications --- p.139Chapter L --- Top-down Generalized Representations --- p.141Chapter M --- Sample Outputs for Automatic Utterance Segmentation --- p.144Chapter N --- Induced Grammar --- p.145Chapter O --- Seeded Categories --- p.148Chapter P --- Semi-Automatically-Induced Response Templates --- p.150Chapter Q --- Details of the Statistical Testing Regarding Grice's Maxims and User Satisfaction --- p.15

Learning Symmetric Collaborative Dialogue Agents with Dynamic Knowledge Graph Embeddings

We study a symmetric collaborative dialogue setting in which two agents, each with private knowledge, must strategically communicate to achieve a common goal. The open-ended dialogue state in this setting poses new challenges for existing dialogue systems. We collected a dataset of 11K human-human dialogues, which exhibits interesting lexical, semantic, and strategic elements. To model both structured knowledge and unstructured language, we propose a neural model with dynamic knowledge graph embeddings that evolve as the dialogue progresses. Automatic and human evaluations show that our model is both more effective at achieving the goal and more human-like than baseline neural and rule-based models.Comment: ACL 201