30 research outputs found
Exploring User Satisfaction in a Tutorial Dialogue System
Abstract User satisfaction is a common evaluation metric in task-oriented dialogue systems, whereas tutorial dialogue systems are often evaluated in terms of student learning gain. However, user satisfaction is also important for such systems, since it may predict technology acceptance. We present a detailed satisfaction questionnaire used in evaluating the BEETLE II system (REVU-NL), and explore the underlying components of user satisfaction using factor analysis. We demonstrate interesting patterns of interaction between interpretation quality, satisfaction and the dialogue policy, highlighting the importance of more finegrained evaluation of user satisfaction
The Impact of Interpretation Problems on Tutorial Dialogue
Supporting natural language input may improve learning in intelligent tutoring systems. However, interpretation errors are unavoidable and require an effective recovery policy. We describe an evaluation of an error recovery policy in the BEE-TLE II tutorial dialogue system and discuss how different types of interpretation problems affect learning gain and user satisfaction. In particular, the problems arising from student use of non-standard terminology appear to have negative consequences. We argue that existing strategies for dealing with terminology problems are insufficient and that improving such strategies is important in future ITS research.
Beetle II: an adaptable tutorial dialogue system
We present BEETLE II, a tutorial dialogue system which accepts unrestricted language input and supports experimentation with different dialogue strategies. Our first system evaluation compared two dialogue policies. The resulting corpus was used to study the impact of different tutoring and error recovery strategies on user satisfaction and student interaction style. It can also be used in the future to study a wide range of research issues in dialogue systems.
The "DeMAND" coding scheme: A "common language" for representing and analyzing student discourse
Content, Social, and Metacognitive Statements: An Empirical Study Comparing Human-Human and Human-Computer Tutorial Dialogue
We present a study which compares human-human computer-mediated tutoring with two computer tutoring systems based on the same materials but differing in the type of feedback they provide. Our results show that there are significant differences in interaction style between human-human and human-computer tutoring, as well as between the two computer tutors, and that different dialogue characteristics predict learning gain in different conditions. We show that there are significant differences in the non-content statements that students make to human and computer tutors, but also to different types of computer tutors. These differences also affect which factors are correlated with learning gain and user satisfaction. We argue that ITS designers should pay particular attention to strategies for dealing with negative social and metacognitive statements, and also conduct further research on how interaction style affects human-computer tutoring. © 2010 Springer-Verlag Berlin Heidelberg
Dealing with Interpretation Errors in Tutorial Dialogue.
We describe an approach to dealing with
interpretation errors in a tutorial dialogue
system. Allowing students to provide explanations
and generate contentful talk can
be helpful for learning, but the language
that can be understood by a computer system
is limited by the current technology.
Techniques for dealing with understanding
problems have been developed primarily for
spoken dialogue systems in informationseeking
domains, and are not always appropriate
for tutorial dialogue. We present a
classification of interpretation errors and our
approach for dealing with them within an
implemented tutorial dialogue system
Diagnosing natural language answers to support adaptive tutoring
Understanding answers to open-ended explanation
questions is important in intelligent tutoring systems.
Existing systems use natural language techniques in
essay analysis, but revert to scripted interaction with
short-answer questions during remediation, making
adapting dialogue to individual students difficult. We
describe a corpus study that shows that there is a relationship
between the types of faulty answers and the
remediation strategies that tutors use; that human tutors
respond differently to different kinds of correct answers;
and that re-stating correct answers is associated
with improved learning. We describe a design for a diagnoser
based on this study that supports remediation in
open-ended questions and provides an analysis of natural
language answers that enables adaptive generation
of tutorial feedback for both correct and faulty answers
Talk Like an Electrician: Student Dialogue Mimicking Behavior in an Intelligent Tutoring System
Abstract. Students entering a new field must learn to speak the specialized language of that field. Previous research using automated measures of word overlap has found that students who modify their language to align more closely to a tutor's language show larger overall learning gains. We present an alternative approach that assesses syntactic as well as lexical alignment in a corpus of human-computer tutorial dialogue. We found distinctive patterns differentiating high and low achieving students. Our high achievers were most likely to mimic their own earlier statements and rarely made mistakes when mimicking the tutor. Low achievers were less likely to reuse their own successful sentence structures, and were more likely to make mistakes when trying to mimic the tutor. We argue that certain types of mimicking should be encouraged in tutorial dialogue systems, an important future research direction