An evaluation paradigm for spoken dialog systems based on crowdsourcing and collaborative filtering.

Yang, Zhaojun.Thesis (M.Phil.)--Chinese University of Hong Kong, 2011.Includes bibliographical references (p. 92-99).Abstracts in English and Chinese.Chapter 1 --- Introduction --- p.1Chapter 1.1 --- SDS Architecture --- p.1Chapter 1.2 --- Dialog Model --- p.3Chapter 1.3 --- SDS Evaluation --- p.4Chapter 1.4 --- Thesis Outline --- p.7Chapter 2 --- Previous Work --- p.9Chapter 2.1 --- Approaches to Dialog Modeling --- p.9Chapter 2.1.1 --- Handcrafted Dialog Modeling --- p.9Chapter 2.1.2 --- Statistical Dialog Modeling --- p.12Chapter 2.2 --- Evaluation Metrics --- p.16Chapter 2.2.1 --- Subjective User Judgments --- p.17Chapter 2.2.2 --- Interaction Metrics --- p.18Chapter 2.3 --- The PARADISE Framework --- p.19Chapter 2.4 --- Chapter Summary --- p.22Chapter 3 --- Implementation of a Dialog System based on POMDP --- p.23Chapter 3.1 --- Partially Observable Markov Decision Processes (POMDPs) --- p.24Chapter 3.1.1 --- Formal Definition --- p.24Chapter 3.1.2 --- Value Iteration --- p.26Chapter 3.1.3 --- Point-based Value Iteration --- p.27Chapter 3.1.4 --- A Toy Example of POMDP: The NaiveBusInfo System --- p.27Chapter 3.2 --- The SDS-POMDP Model --- p.31Chapter 3.3 --- Composite Summary Point-based Value Iteration (CSPBVI) --- p.33Chapter 3.4 --- Application of SDS-POMDP Model: The Buslnfo System --- p.35Chapter 3.4.1 --- System Description --- p.35Chapter 3.4.2 --- Demonstration Description --- p.39Chapter 3.5 --- Chapter Summary --- p.42Chapter 4 --- Collecting User Judgments on Spoken Dialogs with Crowdsourcing --- p.46Chapter 4.1 --- Dialog Corpus and Automatic Dialog Classification --- p.47Chapter 4.2 --- User Judgments Collection with Crowdsourcing --- p.50Chapter 4.2.1 --- HITs on Dialog Evaluation --- p.51Chapter 4.2.2 --- HITs on Inter-rater Agreement --- p.53Chapter 4.2.3 --- Approval of Ratings --- p.54Chapter 4.3 --- Collected Results and Analysis --- p.55Chapter 4.3.1 --- Approval Rates and Comments from Mturk Workers --- p.55Chapter 4.3.2 --- Consistency between Automatic Dialog Classification and Manual Ratings --- p.57Chapter 4.3.3 --- Inter-rater Agreement Among Workers --- p.60Chapter 4.4 --- Comparing Experts to Non-experts --- p.64Chapter 4.4.1 --- Inter-rater Agreement on the Let's Go! System --- p.65Chapter 4.4.2 --- Consistency Between Expert and Non-expert Annotations on SDC Systems --- p.66Chapter 4.5 --- Chapter Summary --- p.68Chapter 5 --- Collaborative Filtering for Performance Prediction --- p.70Chapter 5.1 --- Item-Based Collaborative Filtering --- p.71Chapter 5.2 --- CF Model for User Satisfaction Prediction --- p.72Chapter 5.2.1 --- ICFM for User Satisfaction Prediction --- p.72Chapter 5.2.2 --- Extended ICFM for User Satisfaction Prediction --- p.73Chapter 5.3 --- Extraction of Interaction Features --- p.74Chapter 5.4 --- Experimental Results and Analysis --- p.76Chapter 5.4.1 --- Prediction of User Satisfaction --- p.76Chapter 5.4.2 --- Analysis of Prediction Results --- p.79Chapter 5.5 --- Verifying the Generalibility of CF Model --- p.81Chapter 5.6 --- Evaluation of The Buslnfo System --- p.86Chapter 5.7 --- Chapter Summary --- p.87Chapter 6 --- Conclusions and Future Work --- p.89Chapter 6.1 --- Thesis Summary --- p.89Chapter 6.2 --- Future Work --- p.90Bibliography --- p.9