Bell, Peter

Dzikovska, Myroslava

Isard, Amy

We present our work in building a spoken language interface for a tutorial dialogue system. Our goal is to allow natural, unrestricted student interaction with the computer tutor, which has been shown to improve the student's learning gain, but presents challenges for speech recognition and spoken language understanding. Here we describe the system design, focusing on the components used for speech recognition

Edinburgh Research Explorer

     Edinburgh Research Explorer                                      A tutorial dialogue system with unrestricted spoken inputCitation for published version:Bell, P, Dzikovska, M & Isard, A 2012, A tutorial dialogue system with unrestricted spoken input. inINTERSPEECH 2012, 13th Annual Conference of the International Speech Communication Association,Portland, Oregon, USA, September 9-13, 2012. pp. 2113-2114.Link:Link to publication record in Edinburgh Research ExplorerDocument Version:Peer reviewed versionPublished In:INTERSPEECH 2012, 13th Annual Conference of the International Speech Communication Association,Portland, Oregon, USA, September 9-13, 2012General rightsCopyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)and / or other copyright owners and it is a condition of accessing these publications that users recognise andabide by the legal requirements associated with these rights.Take down policyThe University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorercontent complies with UK legislation. If you believe that the public display of this file breaches copyright pleasecontact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately andinvestigate your claim.Download date: 05. Apr. 2019A tutorial dialogue system with unrestricted spoken inputPeter Bell, Myroslava Dzikovska, Amy IsardSchool of Informatics, University of Edinburgh, UK{ skb m k a sit ravzd o. e , dlr .,e y il .mpe a }@ed.ac.ukAbstractWe present our work in building a spoken language interface fora tutorial dialogue system. Our goal is to allow natural, unre-stricted student interaction with the computer tutor, which hasbeen shown to improve the student’s learning gain, but presentschallenges for speech recognition and spoken language under-standing. Here we describe the system design, focusing on thecomponents used for speech recognition.Index Terms: spoken dialogue system, speech recognition,computer tutoring1. IntroductionMost research in spoken dialogue systems has focussed on sys-tems which are task-oriented, designed to help the user achievesome fixed goal in a minimum number of dialogue turns, of-ten using a slot-filling paradigm. We believe that spoken dia-logue systems could be deployed more widely in the domainof computer tutoring, where, in contrast, the primary aim is tomaximise the student’s learning gain from using the system.A substantial body of research eg. [1] has shown that aneffective tutoring technique is to encourage students to producetheir own explanations and generally to talk more about the do-main during problem-solving. This motivated the developmentof dialogue-based intelligent tutoring systems (ITS) which askstudents open-response questions (rather than multiple-choicequestions), and in particular explanation questions. However,to date such systems have largely been limited to using typedinteractions; existing speech-enabled tutorial dialogue systemssuch as [2] have been constrained to small-vocabulary scenar-ios which restrict the student to a limited range of answers, andtherefore restrict opportunities for self-explanation.Figure 1: Screenshot of the BEETLE II tutor showing text-basedinteractionRecently, typed systems attempted to encourage long, open-ended student answers through asking explanation questionsand giving targeted feedback. While this approach was effec-tive in some cases [3], there is evidence that in human-humantutoring spoken dialogue is more effective than typing [4].This motivates our work in adding the capability for nat-ural, unrestricted spoken interaction to BEETLE II, our exist-ing typed tutorial dialogue system [5]. The BEETLE II systemteaches students basic electricity and electronics by introducingthem to the fundamental concepts, then challenging their under-standing by asking them to predict the outcome of exercises in acircuit simulator and to explain their reasoning. This approachencourages deep thinking, requiring the students to articulatethe causes and effects of the events they are observing. Theexample dialogue shown in Figure 2 illustrates the natural lan-guage capabilities of the system; a screenshot of the GraphicalUser Interface (GUI) is shown in Figure 1.Tutor: Why was bulb A on when switch Y was open andswitch Z was closed?Student: because it had a closed pathTutor: Right. There is a closed path. Keep going.Student: it has a closed path with the batteryTutor: You’re on the right track. Here’s the answer. Bulb Ais still contained in a closed path with the battery.Figure 2: Example interaction with the system from the BEETLEII corpusTo our knowledge, the addition of speech modality to com-plement the NLP capabilities of BEETLE II will make it the firstITS capable of processing long spoken explanation answers.Moving from typed to spoken interactions in this type of sys-tem presents a number of challenges, which we discuss in thefollowing sections.2. ArchitectureThe system is highly modular in design, illustrated in figure 3.On the input side, the system employs a deep parser, TRIPS[6] which provides a domain-independent semantic representa-tion, followed by higher-level domain reasoning and diagnosticscomponents which determine the correctness of student expla-nations. Based on this input, the tutorial planner module selectswhich tutorial strategy to use, which is implemented via a deepgeneration module which constructs tutorial feedback using adomain-specific content planner together with relevant contentfrom the student’s own answer.The new ASR module uses ATK1 to perform one-linespeech parametrisation, voice activity detection and speechrecognition in real-time using a multi-threaded design (though1http://htk.eng.cam.ac.ukDialogue managerKnowledge baseTutorial plannerGeneratorTTSSemantic interpreterParserASRFigure 3: The modules of the BEETLE II systemdialogue-management functions are delegated to the existingBEETLE II dialogue manager). We replaced ATK’s nativeViterbi decoder with our own online version of HTK’s HDe-code, to allow efficient large-vocabulary recognition. The lan-guage models and acoustic models are described in the follow-ing sections. Spoken output is provided using the Festival text-to-speech engine. In addition to the natural language compo-nents, the GUI includes an area to display reading material andan interactive circuit simulator.3. Language modellingIn many spoken dialogue systems, ASR is performed usinghand-crafted finite-state networks selected according to the di-alogue state. This is not appropriate for our system, where itis important to allow unrestricted speech, at least in principle,because students often struggle with unfamiliar terminology:effective tutoring requires knowing the words that the studentsaid, even if they are out of domain. Therefore recognition isperformed using an n-gram language model (LM).We have a corpus available of domain-specific data com-prising 90,000 words of typed interactions with the earlier BEE-TLE II system, collected during 2009. However, we wouldexpect the lexical content of the spoken input to differ con-siderably from to the typed inputs: the switch to the spokenmodality is likely to result in more verbose responses, and fur-thermore, the speech may contain disfluencies characteristic ofspontaneous speech. As an illustration of this, Figure 4 showsan example of two different spoken student responses from ourdevelopment data, illustrating the contrast with typed answers.Student one: Row one. If bulb A is out bulb B and C willremain on. So number one is correct. Row two. Bulb B isout therefore bulb C will be out so that is incorrect and viceversa for row number three. If C is out B will also be out.Student two: X is it open? Row two is incorrect. Um. Rowthree is incorrect. Rows two and three are incorrect.Figure 4: Two example responses to the question “Which rowsdo you think are incorrect?” from our development collection ofspoken interaction. Punctuation has been added for readability.To solve this problem, we created an interpolated LM usingtwo further corpora: the Fisher corpus of transcribed telephoneconversations, and a small development corpus of spoken in-teractions with the system. We restricted the recogniser’s vo-cabulary to the complete set of words from the corpus of typedinteractions, plus filled pauses and common contractions suchas “it’s”, “you’ve” etc.4. Acoustic modellingDue to the limited quantities of development audio data avail-able, we did not attempt to train acoustic models on in-domaindata, but instead used models available to us from the AMIDAcorpus [7], which were trained on approximately 130 hours ofspeech from multiparty meetings. They are a reasonable matchfor our domain in terms of the recording conditions, speak-ing style and speaker demographic. The models were standardHMM-GMMs, trained on PLP features using MPE training. Aglobal HLDA transform was used, and online CMN was per-formed using ATK’s standard method. We implemented onlinespeaker adaption using a smoothed version of CMLLR [8]5. Future workConsidering that the output from ASR will always contain er-rors, a number of other problems must be solved to create aneffective spoken language system. Clearly a major challenge isensuring robust spoken language understanding when the WERis relatively high, given that the student utterances often have acomplex semantic representation. The TRIPS parser is designedto provide robust parses over lattices; however, since the higher-level modules are deterministic in nature, we are not yet able touse the deep domain knowledge available to them to re-scoreASR lattices. Furthermore, the parser is tuned to maximise thechance of finding a complete spanning parse, rather than to dis-criminate between alternative hypotheses. We plan to addressthis in future work.Additionally, the system does not yet use statistical dia-logue management. We propose to employ reinforcement learn-ing in a future version of the system. Major unsolved issues toconsider will be determining a suitable low-dimensional state-space for the dialogue, and selecting which measures of systemor student performance should be optimised.6. References[1] M. T. H. Chi, N. de Leeuw, M.-H. Chiu, and C. LaVancher, “Elicit-ing self-explanations improves understanding.” Cognitive Science,vol. 18, no. 3, pp. 439–477, 1994.[2] D. J. Litman and S. Silliman, “ITSPOKE: an intelligent tutoringspoken dialogue system,” in Demonstration Papers at HLT-NAACL2004, 2004, pp. 5–8.[3] N. Person, A. C. Graesser, L. Bautista, E. C. Mathews, and TRG,“Evaluating student learning gains in two versions of AutoTutor,”in Proceedings of AIED-2001, 2001.[4] D. Litman, C. P. Rose´, K. Forbes-Riley, K. VanLehn, D. Bhembe,and S. Silliman, “Spoken versus typed human and computer dia-logue tutoring,” International Journal of Artificial Intelligence inEducation, vol. 16, pp. 145–170, 2006.[5] M. Dzikovska, D. Bental, J. D. Moore, N. B. Steinhauser, G. E.Campbell, E. Farrow, and C. B. Callaway, “Intelligent tutoring withnatural language support in the Beetle II system,” in Proceedingsof ECTEL-2010. Springer, October 2010, pp. 620–625.[6] J. Allen, M. Dzikovska, M. Manshadi, and M. Swift, “Deep lin-guistic processing for spoken dialogue systems,” in Proceedings ofthe ACL-07 Workshop on Deep Linguistic Processing, 2007.[7] T. Hain, L. Burget, J. Dines, P. Garner, A. el Hannani, M. Hui-jbregts, M. Karafiat, M. Lincoln, and V. Wan, “The AMIDA 2009meeting transcription system,” in Proc. Interspeech, 2010.[8] C. Breslin, K. Chin, M. Gales, K. Knill, and H. Xu, “Prior infor-mation for rapid speaker adaptation,” in Proc. Interspeech, 2010.

A tutorial dialogue system with unrestricted spoken input

https://www.pure.ed.ac.uk/ws/files/23796971/2012_Bell_Dzikovska_ET_AL_A_Tutorial_Dialogue_system_with_unrestricted_spoken_input.pdf

A tutorial dialogue system with unrestricted spoken input

Abstract

Similar works

Full text

Available Versions

Edinburgh Research Explorer