An evaluation paradigm for spoken dialog systems based on crowdsourcing and collaborative filtering.

Yang, Zhaojun.Thesis (M.Phil.)--Chinese University of Hong Kong, 2011.Includes bibliographical references (p. 92-99).Abstracts in English and Chinese.Chapter 1 --- Introduction --- p.1Chapter 1.1 --- SDS Architecture --- p.1Chapter 1.2 --- Dialog Model --- p.3Chapter 1.3 --- SDS Evaluation --- p.4Chapter 1.4 --- Thesis Outline --- p.7Chapter 2 --- Previous Work --- p.9Chapter 2.1 --- Approaches to Dialog Modeling --- p.9Chapter 2.1.1 --- Handcrafted Dialog Modeling --- p.9Chapter 2.1.2 --- Statistical Dialog Modeling --- p.12Chapter 2.2 --- Evaluation Metrics --- p.16Chapter 2.2.1 --- Subjective User Judgments --- p.17Chapter 2.2.2 --- Interaction Metrics --- p.18Chapter 2.3 --- The PARADISE Framework --- p.19Chapter 2.4 --- Chapter Summary --- p.22Chapter 3 --- Implementation of a Dialog System based on POMDP --- p.23Chapter 3.1 --- Partially Observable Markov Decision Processes (POMDPs) --- p.24Chapter 3.1.1 --- Formal Definition --- p.24Chapter 3.1.2 --- Value Iteration --- p.26Chapter 3.1.3 --- Point-based Value Iteration --- p.27Chapter 3.1.4 --- A Toy Example of POMDP: The NaiveBusInfo System --- p.27Chapter 3.2 --- The SDS-POMDP Model --- p.31Chapter 3.3 --- Composite Summary Point-based Value Iteration (CSPBVI) --- p.33Chapter 3.4 --- Application of SDS-POMDP Model: The Buslnfo System --- p.35Chapter 3.4.1 --- System Description --- p.35Chapter 3.4.2 --- Demonstration Description --- p.39Chapter 3.5 --- Chapter Summary --- p.42Chapter 4 --- Collecting User Judgments on Spoken Dialogs with Crowdsourcing --- p.46Chapter 4.1 --- Dialog Corpus and Automatic Dialog Classification --- p.47Chapter 4.2 --- User Judgments Collection with Crowdsourcing --- p.50Chapter 4.2.1 --- HITs on Dialog Evaluation --- p.51Chapter 4.2.2 --- HITs on Inter-rater Agreement --- p.53Chapter 4.2.3 --- Approval of Ratings --- p.54Chapter 4.3 --- Collected Results and Analysis --- p.55Chapter 4.3.1 --- Approval Rates and Comments from Mturk Workers --- p.55Chapter 4.3.2 --- Consistency between Automatic Dialog Classification and Manual Ratings --- p.57Chapter 4.3.3 --- Inter-rater Agreement Among Workers --- p.60Chapter 4.4 --- Comparing Experts to Non-experts --- p.64Chapter 4.4.1 --- Inter-rater Agreement on the Let's Go! System --- p.65Chapter 4.4.2 --- Consistency Between Expert and Non-expert Annotations on SDC Systems --- p.66Chapter 4.5 --- Chapter Summary --- p.68Chapter 5 --- Collaborative Filtering for Performance Prediction --- p.70Chapter 5.1 --- Item-Based Collaborative Filtering --- p.71Chapter 5.2 --- CF Model for User Satisfaction Prediction --- p.72Chapter 5.2.1 --- ICFM for User Satisfaction Prediction --- p.72Chapter 5.2.2 --- Extended ICFM for User Satisfaction Prediction --- p.73Chapter 5.3 --- Extraction of Interaction Features --- p.74Chapter 5.4 --- Experimental Results and Analysis --- p.76Chapter 5.4.1 --- Prediction of User Satisfaction --- p.76Chapter 5.4.2 --- Analysis of Prediction Results --- p.79Chapter 5.5 --- Verifying the Generalibility of CF Model --- p.81Chapter 5.6 --- Evaluation of The Buslnfo System --- p.86Chapter 5.7 --- Chapter Summary --- p.87Chapter 6 --- Conclusions and Future Work --- p.89Chapter 6.1 --- Thesis Summary --- p.89Chapter 6.2 --- Future Work --- p.90Bibliography --- p.9

Strategic Argumentation Dialogues for Persuasion: Framework and Experiments Based on Modelling the Beliefs and Concerns of the Persuadee

Persuasion is an important and yet complex aspect of human intelligence. When undertaken through dialogue, the deployment of good arguments, and therefore counterarguments, clearly has a significant effect on the ability to be successful in persuasion. Two key dimensions for determining whether an argument is good in a particular dialogue are the degree to which the intended audience believes the argument and counterarguments, and the impact that the argument has on the concerns of the intended audience. In this paper, we present a framework for modelling persuadees in terms of their beliefs and concerns, and for harnessing these models in optimizing the choice of move in persuasion dialogues. Our approach is based on the Monte Carlo Tree Search which allows optimization in real-time. We provide empirical results of a study with human participants showing that our automated persuasion system based on this technology is superior to a baseline system that does not take the beliefs and concerns into account in its strategy.Comment: The Data Appendix containing the arguments, argument graphs, assignment of concerns to arguments, preferences over concerns, and assignment of beliefs to arguments, is available at the link http://www0.cs.ucl.ac.uk/staff/a.hunter/papers/unistudydata.zip The code is available at https://github.com/ComputationalPersuasion/MCC

Strategic argumentation dialogues for persuasion: Framework and experiments based on modelling the beliefs and concerns of the persuadee

Persuasion is an important and yet complex aspect of human intelligence. When undertaken through dialogue, the deployment of good arguments, and therefore counterarguments, clearly has a significant effect on the ability to be successful in persuasion. Two key dimensions for determining whether an argument is 'good' in a particular dialogue are the degree to which the intended audience believes the argument and counterarguments, and the impact that the argument has on the concerns of the intended audience. In this paper, we present a framework for modelling persuadees in terms of their beliefs and concerns, and for harnessing these models in optimizing the choice of move in persuasion dialogues. Our approach is based on the Monte Carlo Tree Search which allows optimization in real-time. We provide empirical results of a study with human participants that compares an automated persuasion system based on this technology with a baseline system that does not take the beliefs and concerns into account in its strategy

Artificial Collective Intelligence Engineering: a Survey of Concepts and Perspectives

Collectiveness is an important property of many systems--both natural and artificial. By exploiting a large number of individuals, it is often possible to produce effects that go far beyond the capabilities of the smartest individuals, or even to produce intelligent collective behaviour out of not-so-intelligent individuals. Indeed, collective intelligence, namely the capability of a group to act collectively in a seemingly intelligent way, is increasingly often a design goal of engineered computational systems--motivated by recent techno-scientific trends like the Internet of Things, swarm robotics, and crowd computing, just to name a few. For several years, the collective intelligence observed in natural and artificial systems has served as a source of inspiration for engineering ideas, models, and mechanisms. Today, artificial and computational collective intelligence are recognised research topics, spanning various techniques, kinds of target systems, and application domains. However, there is still a lot of fragmentation in the research panorama of the topic within computer science, and the verticality of most communities and contributions makes it difficult to extract the core underlying ideas and frames of reference. The challenge is to identify, place in a common structure, and ultimately connect the different areas and methods addressing intelligent collectives. To address this gap, this paper considers a set of broad scoping questions providing a map of collective intelligence research, mostly by the point of view of computer scientists and engineers. Accordingly, it covers preliminary notions, fundamental concepts, and the main research perspectives, identifying opportunities and challenges for researchers on artificial and computational collective intelligence engineering.Comment: This is the author's final version of the article, accepted for publication in the Artificial Life journal. Data: 34 pages, 2 figure