Exploring User Satisfaction in a Tutorial Dialogue System

Abstract User satisfaction is a common evaluation metric in task-oriented dialogue systems, whereas tutorial dialogue systems are often evaluated in terms of student learning gain. However, user satisfaction is also important for such systems, since it may predict technology acceptance. We present a detailed satisfaction questionnaire used in evaluating the BEETLE II system (REVU-NL), and explore the underlying components of user satisfaction using factor analysis. We demonstrate interesting patterns of interaction between interpretation quality, satisfaction and the dialogue policy, highlighting the importance of more finegrained evaluation of user satisfaction

When Does Disengagement Correlate with Performance in Spoken Dialog Computer Tutoring?

In this paper we investigate how student disengagement relates to two performance metrics in a spoken dialog computer tutoring corpus, both when disengagement is measured through manual annotation by a trained human judge, and also when disengagement is measured through automatic annotation by the system based on a machine learning model. First, we investigate whether manually labeled overall disengagement and six different disengagement types are predictive of learning and user satisfaction in the corpus. Our results show that although students’ percentage of overall disengaged turns negatively correlates both with the amount they learn and their user satisfaction, the individual types of disengagement correlate differently: some negatively correlate with learning and user satisfaction, while others don’t correlate with eithermetric at all. Moreover, these relationships change somewhat depending on student prerequisite knowledge level. Furthermore, using multiple disengagement types to predict learning improves predictive power. Overall, these manual label-based results suggest that although adapting to disengagement should improve both student learning and user satisfaction in computer tutoring, maximizing performance requires the system to detect and respond differently based on disengagement type. Next, we present an approach to automatically detecting and responding to user disengagement types based on their differing correlations with correctness. Investigation of ourmachine learningmodel of user disengagement shows that its automatic labels negatively correlate with both performance metrics in the same way as the manual labels. The similarity of the correlations across the manual and automatic labels suggests that the automatic labels are a reasonable substitute for the manual labels. Moreover, the significant negative correlations themselves suggest that redesigning ITSPOKE to automatically detect and respond to disengagement has the potential to remediate disengagement and thereby improve performance, even in the presence of noise introduced by the automatic detection process

Adapting the scheduling of illustrations and graphs to learners in conceptual physics tutoring

This research investigates how to schedule multiple graphical representations in a dialogue-based conceptual physics tutor. Research on multiple graphical representations in tutoring suggests either frequently switching representations or fading from concrete to abstract representations. However, other research communities suggest that the best representation or scheduling can be dependent on various student and tutoring context factors. This thesis investigates whether these factors are important when considering a schedule of representations. Three major hypotheses are investigated. H1: that the best representational format for physics concepts is related to properties of the student and the tutoring context. H2: that it is possible to build models that predict the best representational format using student and tutoring context information. H3: that picking the representational format based upon student and tutoring context information will produce better learning gains than not considering student and tutoring context information. Additionally, this work addresses the question of whether multiple representations produce greater learning gains than a single representation (H4). A first experiment was performed to both investigate H1 and to collect data for H2. ANOVAs showed significant interaction effects in learning between low and high pretesters and between high and low spatial reasoning ability subjects, supporting the first hypothesis. Using the data collected and features describing student and tutoring context information, models were learned to predict when to show illustrations or graphs. That these models could be learned, produce meaningful rules, and outperformed a baseline supports H2. A new modeling algorithm was developed to learn these models by augmenting multiple linear regression to consider certain syntactic constraints. A third study was run to test H3 and H4 and to extrinsically evaluate the adaptive policy learned. One third of subjects had an adaptive scheduling of representations, one third a fixed alternating scheduling, and one third saw only one representation. In support of H3, subjects with high incoming knowledge sometimes perform better when receiving adaptive scheduling over an alternating scheduling, but there are also counter examples. For H4, it is not supported in general: showing only illustrations is best overall, but in some cases some subjects benefit from multiple representations

Cognitive architecture of multimodal multidimensional dialogue management

Numerous studies show that participants of real-life dialogues happen to get involved in rather dynamic non-sequential interactions. This challenges the dialogue system designs based on a reactive interlocutor paradigm and calls for dialog systems that can be characterised as a proactive learner, accomplished multitasking planner and adaptive decision maker. Addressing this call, the thesis brings innovative integration of cognitive models into the human-computer dialogue systems. This work utilises recent advances in Instance-Based Learning of Theory of Mind skills and the established Cognitive Task Analysis and ACT-R models. Cognitive Task Agents, producing detailed simulation of human learning, prediction, adaption and decision making, are integrated in the multi-agent Dialogue Man-ager. The manager operates on the multidimensional information state enriched with representations based on domain- and modality-specific semantics and performs context-driven dialogue acts interpretation and generation. The flexible technical framework for modular distributed dialogue system integration is designed and tested. The implemented multitasking Interactive Cognitive Tutor is evaluated as showing human-like proactive and adaptive behaviour in setting goals, choosing appropriate strategies and monitoring processes across contexts, and encouraging the user exhibit similar metacognitive competences