Bental, Diana

Callaway, Charles B.

Campbell, Gwendolyn E.

Dzikovska, Myroslava O.

Farrow, Elaine

Moore, Johanna D.

Steinhauser, Natalie B.

English

We present Beetle II, a tutorial dialogue system designed to accept unrestricted language input and support experimentation with different tutorial planning and dialogue strategies. Our first system evaluation used two different tutoring policies and demonstrated that Beetle II can be successfully used as a platform to study the impact of different approaches to tutoring. In the future, the system can also be used to experiment with a variety of parameters that may affect learning in intelligent tutoring systems. © 2010 Springer-Verlag Berlin Heidelberg.</p

Heriot Watt Pure

Intelligent tutoring with natural language support in the BEETLE II system

We present BEETLE II, a tutorial dialogue system designed to accept unrestricted language input and support experimentation with different tutorial planning and dialogue strategies. Our first system evaluation used two different tutoring policies and demonstrated that BEETLE II can be successfully used as a platform to study the impact of different approaches to tutoring. In the future, the system can also be used to experiment with a variety of parameters that may affect learning in intelligent tutoring systems.</p

Edinburgh Research Explorer

     Edinburgh Research Explorer                                      Intelligent Tutoring with Natural Language Support in theBEETLE II SystemCitation for published version:Dzikovska, MO, Bental, D, Moore, JD, Steinhauser, NB, Campbell, GE, Farrow, E & Callaway, CB 2010,'Intelligent Tutoring with Natural Language Support in the BEETLE II System'. in M Wolpers, PA Kirschner,M Scheffel, S Lindstaedt & V Dimitrova (eds), SUSTAINING TEL: FROM INNOVATION TO LEARNINGAND PRACTICE. Springer-Verlag GmbH, BERLIN, pp. 620-625, 5th European Conference on TechnologyEnhanced Learning, Barcelona, 28-1 October.Link:Link to publication record in Edinburgh Research ExplorerDocument Version:Author final version (often known as postprint)Published In:SUSTAINING TEL: FROM INNOVATION TO LEARNING AND PRACTICEGeneral rightsCopyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)and / or other copyright owners and it is a condition of accessing these publications that users recognise andabide by the legal requirements associated with these rights.Take down policyThe University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorercontent complies with UK legislation. If you believe that the public display of this file breaches copyright pleasecontact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately andinvestigate your claim.Download date: 20. Feb. 2015Intelligent tutoring with natural languagesupport in the Beetle II systemMyroslava O. Dzikovska1, Diana Bental1, Johanna D. Moore1, Natalie B.Steinhauser2, Gwendolyn E. Campbell2, Elaine Farrow3, and Charles B.Callaway4?1 School of Informatics, University of Edinburgh, Edinburgh, United Kingdom{m.dzikovska,d.bental,j.moore}@ed.ac.uk2 Naval Air Warfare Center Training Systems Division, Orlando, FL, USA{gwendolyn.campbell,natalie.steinhauser}@navy.mil3 Heriot-Watt University, Edinburgh, United Kingdome.farrow@hw.ac.uk4 University of Haifa, Mount Carmel, Haifa, Israelccallawa@gmail.comAbstract. We present Beetle II, a tutorial dialogue system designedto accept unrestricted language input and support experimentation withdifferent tutorial planning and dialogue strategies. Our first system evalu-ation used two different tutoring policies and demonstrated that BeetleII can be successfully used as a platform to study the impact of differ-ent approaches to tutoring. In the future, the system can also be usedto experiment with a variety of parameters that may affect learning inintelligent tutoring systems.1 IntroductionIntelligent tutoring systems (ITS) have been shown to help learning as part ofa technology-enhanced learning curriculum [1], and there has been a lot of in-terest in developing tutoring systems that understand student explanations (e.g.[2]). However, existing ITS with natural language support typically support onlyshort answer questions; at most, they can analyze student essays but then revertto pre-authored short answer dialogues if remediation is necessary. Acceptinglonger utterances may be beneficial to learning, because high percentages ofself-explanation and contentful talk by students are known to be correlated withbetter learning in human-human tutorial dialogue [3]. But students also need de-tailed feedback on their explanations [4], which is very labor-intensive to providewith pre-authored system responses.We present Beetle II, a tutorial dialogue system designed to accept un-restricted language input and to support experimentation with different ap-proaches to tutoring. The Beetle II architecture supports the understanding? This work has been supported in part by US Office of Naval Research grantsN000140810043 and N0001410WX20278. We thank Katherine Harrison and LeanneTaylor for their help running the evaluation.2 Dzikovska et al.Fig. 1. Screenshot of the Beetle II systemof student explanations by using techniques from computational linguistics toanalyze complex student utterances and generate feedback automatically [5].Our research goal is to investigate whether contentful talk and self-explanationcan be understood and encouraged in a computer system, given state-of-the-artin language technology. The Beetle II system is designed to ask students toexplain their answers, to give detailed feedback and to support students in ar-riving at the correct explanations without the system resorting to short-answerquestions, and without having to hand-author every tutorial response.We demonstrate a version of the system that has undergone successful userevaluation in 2009. The rest of this paper discusses the Beetle II design (Sec-tion 2) the system architecture (Section 3), the user evaluation (Section 4), andfurther research questions that can be investigated using Beetle II (Section 5).2 Instructional DesignThe Beetle II system delivers basic electricity and electronics tutoring to stu-dents with no prior knowledge of the subject. A distinguishing feature of thesystem is that it is based on a course developed by experienced instructional de-signers for use in a human-human tutoring study, without taking into account thepossible limitations of a computer-based tutor. Two research team members, anexperienced electrician and a psychologist, jointly developed a curriculum cover-ing topics including the components of a circuit, how to build a circuit, and howto find faults. The exercises were then transferred into a computer system withIntelligent tutoring with natural language support in the Beetle II system 3Tutor: Why was bulb A on when switch Y was open and switch Z was closed?Student: because it had a closed pathTutor: Right. There is a closed path. Keep going.Student: it has a closed path with the batteryTutor: You’re on the right track. Here’s the answer. Bulb A is still contained ina closed path with the battery.Fig. 2. Example interaction with the system from our corpusonly minor adjustments (e.g., breaking down compound questions into individualones).The instructional design interleaves short presentations of information withinteractive exercises, activities and discussion. The activities use a circuit simu-lator which is realistic enough so that bulbs light up when connected correctly,bulbs may be bright or dim, and measurements may be taken by connectinga simulated meter. Many of the activities follow the “predict-verify-evaluate”(PVE) cycle, in which students are asked to predict the outcome of an activ-ity before conducting an experiment using the simulator, and then discuss theactual outcome and its implications for the underlying principles [6].A screenshot of the system is shown in Figure 1. The student interface in-cludes an area to display reading material, a circuit simulator, and a dialoguehistory window. At present, students interact with the system via a typed chatinterface to avoid the problems associated with automated speech recognition.An example dialogue shown in Figure 2 shows how an answer is built jointlyby the student and tutor over more than one turn. In response to the firststudent input, the system rephrases its understanding of the correct part ofthe answer, and prompts the student to supply the missing information. In thenext turn the system combines the information from the tutor’s hint and thestudent’s answers and restates the completed answer. We will use this dialogueas a running example to explain system capabilities.3 System ArchitectureThe system architecture is modular and combines domain-independent com-ponents for parsing and generation with domain-specific reasoners for decisionmaking.We use a natural language dialogue parser [7] to parse the student input.The parser extracts relevant semantic content from each utterance, recognizingparaphrases that can mean the same thing. For our example problem, the parserwould recognize that “Bulb A and the battery are in the same closed path”,“Bulb A is in a closed path with the battery” and “there is a closed path con-taining both the bulb and the battery” mean the same thing and constitute thecorrect answer to the tutor’s question. The parser can handle fragmentary input,for example it can determine that in response to the question in Figure 1 “Which4 Dzikovska et al.bulbs will be on and which bulbs will be off?”, the answer “off” can be taken tomean “all bulbs in the diagram will be off”.The output from the parser is passed on to the domain reasoning and diag-nosis components [8, 9] to check the validity of the student’s explanation. Thediagnoser outputs lists of correct, contradictory and non-mentioned objects andrelations from the student’s explanation. For the first student utterance in ourexample, the diagnoser will determine that the student correctly mentioned aclosed path as part of their explanation, but they forgot to mention that boththe bulb and the battery must be in the same closed path.The tutorial planner implements a set of generic tutoring strategies and apolicy to choose an appropriate strategy at each point of the interaction. Thecurrently implemented strategies are: acknowledge any correct parts of the an-swer; suggest a slide to read with background material; prompt for missing partsof the answer; provide a hint (at different levels of specificity); re-state an accept-able answer using better terminology; and give away the answer. The tutorialpolicy makes a high-level decision as to which strategies to use. In our examplethe first decision is to restate the correct part of the answer (student correctlymentioned a closed path), and give a contentless prompt for missing explanationparts. The tutorial planner also incorporates an error recovery policy to managesituations when the system cannot interpret the student’s input [10].The tutorial planner’s decisions are realized by automatic text generationcomponents, using a combination of domain-specific content planning and anda domain-independent text generation system [11] to produce the appropriatetext. In our example, the text generation decides that the the chosen tutorialstrategy (restate and contentless prompt) should be realized as “Right. There isa closed path. Keep going”Beetle II provides extensive logging facilities. All of the students’ interac-tions with the system are logged, including both their text utterances and theexperimental circuits they build with the simulator. The students’ interactionswith the system and the system’s responses can be replayed. The system’s utter-ances and its internal decisions are also logged. This provides a detailed corpusfor later analysis.4 Experimental work using Beetle IIThe first experimental evaluation involving 81 participants was completed in2009. Participants were undergraduates recruited from a South-East US Univer-sity, with little or no prior knowledge of the domain. Each participant took apre-test, worked through a lesson with the system, took a post-test, and com-pleted a user satisfaction survey. Each session lasted approximately 4 hours.We implemented two different tutoring policies in the system for this eval-uation, which was made possible by our flexible system architecture. In thebaseline policy the students were given the correct answer straight away. Thesystem made no attempt at remediation, and never indicated whether the stu-Intelligent tutoring with natural language support in the Beetle II system 5dent was understood. In comparison, the full adaptive policy selected a strategybased on student answer analysis and dialogue context, as described above.Out of 81 students, 76 successfully completed the evaluation (data from 5participants had to be discarded due to system crashes). All students completedpre- and post- test questionnaires to assess their knowledge. The mean pre-testscore was 34.56(SD = 12.38), and the mean post-test score was 73.91(SD =15.64) for both conditions combined. The difference was statistically significantwith p < 0.0001, indicating that the students successfully learned the material.More detailed data analysis, comparison between conditions and comparisonto human tutoring are given in [10, 12, 13].5 Conclusions and Future WorkOur ultimate goal is to develop a system flexible enough to conduct a systematicinvestigation into the nature of effective tutoring in technology-enhanced learn-ing environments. The experimental evaluation described in section 4 demon-strates that the Beetle II system can be successfully deployed in experimentswith naive users to test the effect of different tutorial policies. We now havea corpus of interactions between students and the computer tutor that can beanalyzed in more detail and used to devise future experiments.Our initial analyses indicate that additional improvements to remediationstrategies, and especially strategies dealing with interpretation problems, arenecessary to make the interaction less frustrating to the users [10]. We are plan-ning to do this as part of future work. However, the success of our large-scaleevaluation shows that the system can already be used to formulate hypothesesand conduct experimental studies.Three factors whose effects we intend to investigate in the future are: linguis-tic alignment between system and user; the choice of modalities for input andoutput; and the impact of different tutorial strategies.Techniques from computational linguistics allow us to vary and control thechoice of terminology and phrasing that the tutor uses. Current research indi-cates that better alignment between students and tutors (computer or human)with respect to the terminology they use is correlated with higher learning gainand user satisfaction [14] and that student satisfaction is negatively affected ifthe system is using different terminology than the student [10]. However, it maybe important that students learn specific terminology rather than the systemadapt to the students. This can be investigated further through controlled ex-periments possible with Beetle II since automatically generated feedback canbe adjusted as necessary.The timing and amount of feedback has been a topic of interest in the e-learning community (e.g.,[15]). The modular nature of Beetle II will allow usto investigate the relevant issues. For example, it is easy to change the tutorialpolicy to give feedback immediately, and compare the outcome with exerciseswhich make the student follow a PVE cycle which results in delayed feedback.6 Dzikovska et al.Other tutoring strategies, such as giving hints at different levels of specificity,can be investigated as well.References1. Anderson, J.R., Corbett, A.T., Koedinger, K.R., Pelletier, R.: Cognitive tutors:Lessons learned. The Journal of the Learning Sciences 4(2) (1995) 167–2072. Jordan, P., Makatchev, M., Pappuswamy, U., VanLehn, K., Albacete, P.: A nat-ural language tutorial dialogue system for physics. In: Proceedings of the 19thInternational FLAIRS conference. (2006)3. Chi, M.T.H., de Leeuw, N., Chiu, M.H., LaVancher, C.: Eliciting self-explanationsimproves understanding. Cognitive Science 18(3) (1994) 439–4774. Nielsen, R.D., Ward, W., Martin, J.H.: Learning to assess low-level conceptualunderstanding. In: Proceedings 21st International FLAIRS Conference. (2008)5. Callaway, C.B., Dzikovska, M., Farrow, E., Marques-Pita, M., Matheson, C.,Moore, J.D.: The Beetle and BeeDiff tutoring systems. In: Proceedings ofSLaTE’07 (Speech and Language Technology in Education). (2007)6. Guzzetti, B.J., Snyder, K.E., Glass, G.V., Gamas, W.S.: Promoting conceptualchange in science: A comparative meta-analysis of instructional interventions fromreading education and science education. Reading Research Quarterly 28(2) (1993)117–1557. Allen, J., Dzikovska, M., Manshadi, M., Swift, M.: Deep linguistic processingfor spoken dialogue systems. In: Proceedings of the ACL-07 Workshop on DeepLinguistic Processing. (2007)8. Dzikovska, M.O., Callaway, C.B., Farrow, E.: Interpretation and generation ina knowledge-based tutorial system. In: Proceedings of EACL-06 workshop onknowledge and reasoning for language processing, Trento, Italy (April 2006)9. Dzikovska, M.O., Campbell, G.E., Callaway, C.B., Steinhauser, N.B., Farrow, E.,Moore, J.D., Butler, L.A., Matheson, C.: Diagnosing natural language answers tosupport adaptive tutoring. In: Proceedings 21st International FLAIRS Conference,Coconut Grove, Florida (May 2008)10. Dzikovska, M.O., Moore, J.D., Steinhauser, N., Campbell, G.: The impact ofinterpretation problems on tutorial dialogue. In: Proceedings of the 48th AnnualMeeting of the Association for Computational Linguistics(ACL-2010). (2010)11. Elhadad, M., Robin, J.: Controlling content realization with functional unifica-tion grammars. In: Proceedings of the Sixth International Workshop on NaturalLanguage Generation. (1992)12. Steinhauser, N.B., Campbell, G.E., Harrison, K.M., Taylor, L.S., Dzikovska, M.O.,Moore, J.D.: Comparing human-human and human-computer tutorial dialogue. In:Proceedings of the the 32nd Annual Conference of the Cognitive Science Societyposter session. (2010)13. Dzikovska, M.O., Steinhauser, N.B., Moore, J.D., Campbell, G.E., Harrison, K.M.,Taylor, L.S.: Content, social and metacognitive statements: an empirical studycomparing human-human and human-computer tutorial dialogue. In: Proceedingsof ECTEL-2010. (2010)14. Ward, A., Litman, D.: Cohesion and learning in a tutorial spoken dialog system.In: Proceedings of 19th International FLAIRS Conference. (2006)15. Smits, M.H.S.B., Boon, J., Sluijsmans, D.M.A., van Gog, T.: Content and timing offeedback in a web-based learning environment: effects on learning as a function ofprior knowledge. Interactive Learning Environments 16(2) (August 2008) 183–193

A natural language tutorial dialogue system for physics. In:

Cognitive tutors: Lessons learned.

Cohesion and learning in a tutorial spoken dialog system. In:

Comparing human-human and human-computer tutorial dialogue. In:

Content and timing of feedback in a web-based learning environment: eﬀects on learning as a function of prior knowledge.

Controlling content realization with functional uniﬁcation grammars. In:

Deep linguistic processing for spoken dialogue systems. In:

Diagnosing natural language answers to support adaptive tutoring. In:

Eliciting self-explanations improves understanding.

Interpretation and generation in a knowledge-based tutorial system. In:

L.S.: Content, social and metacognitive statements: an empirical study comparing human-human and human-computer tutorial dialogue. In:

Learning to assess low-level conceptual understanding. In:

The Beetle and BeeDiﬀ tutoring systems. In:

The impact of interpretation problems on tutorial dialogue. In:

W.S.: Promoting conceptual change in science: A comparative meta-analysis of instructional interventions from reading education and science education.

Intelligent Tutoring with Natural Language Support in the BEETLE II System

Abstract

Similar works

Full text

Available Versions

Heriot Watt Pure

Edinburgh Research Explorer

Heriot Watt Pure

Crossref