We have developed MMDAgent (a fully open-source toolkit for voice interaction systems), which runs on a variety of platforms such as personal computers and smartphones. From this, the editing environment of the dialog scenario also needs to be operated on various platforms. So, we develop a scenario editor that is implemented on a Web browser. The purpose of this paper also includes making it easy to edit the scenario. Experiments were conducted for subjects using the proposed scenario editor. It was found that our proposed system provides better readability of a scenario and allows easier editing

Nishimura, Ryota

Takumi, Ichi

Uchiya, Takahiro

Yamamoto, Daisuke

English

Tokushima University Institutional Repository

Available online at www.sciencedirect.comScienceDirectICT Express 5 (2019) 47–51www.elsevier.com/locate/icteMMDAE: Dialog scenario editor for MMDAgent on the web browserRyota Nishimuraa,∗, Daisuke Yamamotob, Takahiro Uchiyab, Ichi Takumiba Department of Technology, Industrial and Social Science, Tokushima University, Tokushima, Japanb Department of Computer Science, Nagoya Institute of Technology, Nagoya, JapanReceived 7 February 2018; accepted 21 March 2018Available online 12 April 2018AbstractWe have developed MMDAgent (a fully open-source toolkit for voice interaction systems), which runs on a variety of platforms such aspersonal computers and smartphones. From this, the editing environment of the dialog scenario also needs to be operated on various platforms.So, we develop a scenario editor that is implemented on a Web browser. The purpose of this paper also includes making it easy to edit the scenario.Experiments were conducted for subjects using the proposed scenario editor. It was found that our proposed system provides better readability ofa scenario and allows easier editing.c⃝ 2018 The Korean Institute of Communications and Information Sciences (KICS). Publishing Services by Elsevier B.V. This is an open accessarticle under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).Keywords: Spoken dialog system; Scenario editor; Web browser; MMDAgent1. IntroductionVarious techniques of speech processing have recently beendeveloped. Among these, techniques of speech recognitionand speech synthesis are widely used. Spoken dialog systems(SDSs) integrating these technologies have also been devel-oped. Pioneer of the spoken dialog system is a VOYAGER ofMIT. It was developed in the early 1990s. ATIS project [1] hasbeen carried out in the early 1990s led by DARPA in the UnitedStates. In recent years, ‘GalateaToolkit’ a toolkit of spokendialog systems [2] and ‘SCHEMA’ robot that can interactmultiplayer [3] has been developed. Commercial systems, suchas Siri (Apple Inc.) [4], have appeared and gained popularity.However, SDSs are not yet widely used.Therefore, for anyone to be able to use an SDS easily, weconstructed a fully open-source toolkit for voice interactionsystems (MMDAgent [5]) using speech processing technology.As a practical example, digital signage has been set up infront of the main gate of a university (Nagoya Institute of∗ Corresponding author.E-mail address: ryota@nishimura.name (R. Nishimura).Peer review under responsibility of The Korean Institute of Communica-tions and Information Sciences (KICS).Technology) [6], allowing anyone to interact with a life-sizethree-dimensional (3D) character named ‘Mei-chan’ (Fig. 1).SDS software has been developed for the personal computer(PC; running Windows, Mac OS, or Linux), and it has alsobeen ported to Android so as to work on any smartphone [7].Additionally, an SDS using the video communication functionof Skype (Voice over Internet Protocol) has been developed[8].The MMDAgent toolkit includes software for speech recog-nition, speech synthesis, character drawing as part of 3D com-puter graphics and dialog management to meet the requirementsof an SDS. An environment to build an SDS can easily becreated using this toolkit. However, even to create the envi-ronment, expert knowledge of the spoken dialog is necessaryin constructing an SDS. Novice users find it difficult to builda dialog system without such knowledge. In the constructionof the SDS, it is necessary to edit the dialog scenario of thecontents of the conversation. However, in the current editingenvironment, it is difficult to read in a complex dialog scenario,and editing is thus difficult even for the expert user. Therefore,in this paper, we develop a dialog scenario editor to improve theediting environment.https://doi.org/10.1016/j.icte.2018.03.0022405-9595/ c⃝ 2018 The Korean Institute of Communications and Information Sciences (KICS). Publishing Services by Elsevier B.V. This is an open access articleunder the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).48 R. Nishimura, D. Yamamoto, T. Uchiya et al. / ICT Express 5 (2019) 47–51Fig. 1. Life-size 3D character “Mei-chan” at the main gate of a university.2. Dialog scenarioAs a method of describing a spoken dialog, VoiceXML(VXML) is well known [9]. Interaction between human-computer has been described in the XML format. VoiceXMLis a kind of standard XML format of the W3C. XISL have alsobeen developed [10], it is possible to describe the interactionusing any modality.The dialog scenario in MMDAgent is managed according tothe finite-state transducer (FST) format. When the user createsa scenario, it is necessary to describe the state transition ofthe FST in a text file (Fig. 3). The FST format is a list offour values, namely the state number, transition state number,acceptance conditions and command, separated by spaces. Inthe example of Fig. 3, when an event (RECOG STOP|Hello)is received in state 1, the system makes a transition to state 10without any output (⟨eps⟩). It then outputs a command message(MOTION ADD|mei|greet|greet.vmd), and the system makesa transition to state 11. Thus, the system controls the dialog byrepeating the exchange of internal messages and state transi-tions. ⟨eps⟩ denotes the epsilon transition, which is a transitionwithout any input or output.In the scenario file, such as that in Fig. 3, the indentingof each item is performed manually using a space or tab. InFig. 3. Example of a dialog scenario FST (Some notation is simplified).some cases, indentations are not aligned and readability is thuspoor. Additionally, the user must remember the commandsto describe the scenario. Scenario editing in a text editor istherefore difficult even for the expert user.3. Dialog scenario editor (MMDAE)To improve the created environment of spoken dialog sce-narios and thus solve the problems described above, we devel-oped a scenario editor (Fig. 2). The scenario editor is namedMMDAE (MMDAgent scenario Editor). Three features ofMMDAE are discussed in the following.3.1. 1: Completion of the inputTo create a scenario, the input of four items is required, asshown in Fig. 3. It is difficult to type all the dialog scenarios(conditions and commands). An input complement functionwas thus implemented in the system. Furthermore, after thecommand is entered, the text area in which to enter argumentsis displayed.3.2. 2: Execution on various platformsThe MMDAgent can be run on a variety of platforms suchas PCs (running Windows, Mac OS, or Linux) and smartphones(running Android). From this, the editing environment of thedialog scenario also needs to be operated on various platforms.So, we develop a scenario editor that is implemented on a Webbrowser. Furthermore, MMDAgent has the ability to share thedialog scenario on the Internet, and in this respect, the use of aWeb browser is effective.Fig. 2. Dialog scenario editor (MMDAE).R. Nishimura, D. Yamamoto, T. Uchiya et al. / ICT Express 5 (2019) 47–51 49Fig. 4. Simple edit mode.3.3. 3: Changing the edit modeTo change the ease of editing the dialog scenario accordingto the user’s knowledge and experience of using an SDS, itis possible to change the edit mode. Only a few items aredisplayed to the novice user. In this way, the user can edit thedialog scenario without knowledge of the scenario descriptionmethod.Expert users are presented many items and are providedan environment in which to create a scenario using the fullfunctionality of MMDAgent. Furthermore, according to thehardware used to edit a scenario (e.g., a PC or smartphone),it is possible to select an appropriate display method. Forexample, a compact display is presented for the small screenof a smartphone, as shown in Fig. 4.4. EvaluationAn experiment was performed to evaluate the performanceof MMDAE. Subjects were 13 male bachelor’s and master’sstudents in their twenties.The experiments were performed according to the followingprocedure.1. Explanation of the experiment2. Explanation of the interaction scenario (FST)3. Pre-confirmation of questionnaire items4. Experiment (scenario editing)• Editing using a text editor (Notepad)• Explanation of how to use MMDAE• Editing using MMDAE5. questionnaire.Notepad (the standard text editor in Windows) and MMDAEwere compared in the evaluation of the editing environment ofthe dialog scenario. In the experiment, subjects appended thecontents of the dialog in the scenario file. The order of use ofediting tools may affect the experimental results, because thesubject may become used to editing. Therefore, subjects weredivided into two groups, and the order of the use of editing toolswas different for each group. The duration of editing by eachFig. 5. Example of an editing task.subject was recorded. Experiments were performed using a PCthat is usually used by the subjects in the laboratory.The task was presented to each subject in the form of Fig. 5.There were four types of tasks, and subjects used both MMDAEand Notepad for each task.A questionnaire was conducted after the experiment. For thefollowing items, the subjects gave a score on a five-point Likertscale. In the case of Q1, for example, a response that the subjectfound it difficult to edit is awarded 1 point, and a response thatit was easy to edit is awarded 5 points.• Easy to edit (Q1: Notepad, Q2: MMDAE)• Readability (Q3: Notepad, Q4: MMDAE)• Easy to understand the usage(Q5: Notepad, Q6: MMDAE)• Convenience of function– Q7: input complement function– Q8: focus switched by TAB key– Q9: text area division in accordance with the num-ber of arguments• About MMDAgent– Q10: Were you aware of MMDAgent previously?– Q11: Have you used MMDAgent previously?• For the following items, subjects wrote freely– Q12: What functions does the system require?– Q13: Other comments (good and bad points aboutthe system).5. ResultsThe results of subjective evaluation based on the five-pointLikert scale and objective evaluation based on the editingduration are presented below.5.1. Subjective evaluationExperimental results are shown in Fig. 6. The questionsQ1–11 are as listed in the previous section. Fig. 6 shows that, incomparing the two editing systems (i.e., Q1 vs. Q2, Q3 vs. Q4,and Q5 vs. Q6), MMDAE obtained higher marks for all items,and the difference between scores was more than 1 point. In50 R. Nishimura, D. Yamamoto, T. Uchiya et al. / ICT Express 5 (2019) 47–51Fig. 6. Results of subjective evaluation (five-point scales).Table 1Edit time.Notepad MMDAEAverage 4:50 3:58particular, in terms of easy editing and easy understanding ofuse, MMDAE scored 4 points or more on average. Additionally,the functionality of MMDAE (Q7, Q8, and Q9) scored highly.According to these results, MMDAE is more suitable thanNotepad for dialog editing.5.2. Objective evaluationThe editing duration and efficiency were compared betweenediting systems. The results are given in Table 1. Whereas theediting duration using Notepad was 4 min 50 s on average, thatusing MMDAE was 3 min 58 s on average; i.e., the editing timewas reduced on average by 52 s (or about 18% of the averageduration using Notepad) when using MMDAE.5.3. Opinions from the questionnaireThe subjects’ opinions on MMDAE, in terms of their freelywritten answers to the questions in the survey, are listed below.There were the following positive opinions.• The efficiency of work was improved considerably by theinput complement function of MMDAE.• Readability using MMDAE is better than that using a texteditor.• Without any knowledge of the FST and with auto-completion of the conversation command, editing waseasy.According to these comments, the subjects felt that MMDAEhad improved efficiency. In particular, there were many positiveopinions on readability. The subjects in this experiment hadno editing experience of the dialog scenario file prior to theexperiment, and had no knowledge of commands of the FST.However, because the input was complemented by MMDAE,the user could learn the FST while editing.There was however the following negative opinion.• Many text boxes are displayed, which is confusing.Opinions were often positive in terms of readability, but therewas also the negative opinion that each user should be presentedwith a format that suits them. The present experiment wasconducted using only the detailed edit mode, and the experi-ment should be repeated using the simple edit mode shown inFig. 4. A user who is confused by the many text boxes in thedetailed edit mode might find it easier to use the simple editmode.There were the following opinions.• When the mouse cursor is over a button, a description ofthe button should be displayed.• There should be a line copy function.For the novice user of MMDAE, the function is displayed oneach button or part of the display, and it is thus possible toedit the dialog scenario while learning how to use the system.Because this feature is important to beginners, it will be addedto all edit modes of MMDAE in the future. Furthermore, theability to copy one or more lines would be useful when describ-ing a dialog scenario structured like a conversation scenariothat has previously been described. This function will also beadded.6. ConclusionTo improve the environment for editing the scenario ofan SDS, we developed a scenario editor (MMDAE). In anexperiment, subjects preferred to edit a scenario in MMDAEthan in Notepad, and the editing duration was about 18% lesswhen using MMDAE. In terms of usability, MMDAE ratedhigher than Notepad in a survey of users.In future work, from the results obtained in the subjectquestionnaire, we will add necessary functionality to the sys-tem. In addition to the items listed in the section “Objectiveevaluation”, the automatic insertion of the FST number, check-ing of the transition destination, and a preview of the systemresponse using speech synthesis will be implemented for thesystem.Additionally, we want to add a function to enter the dialogscenario using speech recognition. Using this function, it willbe possible to create a scenario while checking the behavior ofthe system. In using voices to edit the system response, prosodicinformation such as the speech rate, accent, and pitch can beinput.AcknowledgmentsThis study has been supported by the Core Research forEvolutional Science and Technology (CREST) (11102610) ofthe Japan Science and Technology Agency (JST), by JSPSKAKENHI Grant Number JP 25700009, and by the StrategicInformation and Communications R&D Promotion Programme(SCOPE) (162309001) of Ministry of Internal Affairs andCommunications (MIC), Japan.R. Nishimura, D. Yamamoto, T. Uchiya et al. / ICT Express 5 (2019) 47–51 51Conflict of interestThe authors declare that there is no conflict of interest in thispaper.References[1] P.J. Price, Evaluation of spoken language systems: the ATIS domain, in:Proc. DARPA Speech & Natural Language Workshop, 1990, pp. 91–95.[2] S. Kawamoto, H. Shimodaira, T. Nitta, T. Nishimoto, S. Nakamura, K.Itou, S. Morishima, T. Yotsukura, A. Kai, A. Lee, Y. Yamashita, T.Kobayashi, K. Tokuda, K. Hirose, N. Minematsu, A. Yamada, Y. Den, T.Utsuro, S. Sagayama, Open-source software for developing anthropomor-phic spoken dialog agent, in: Proc. of PRICAI-02, International Workshopon Lifelike Animated Agents, 2002, pp. 64–69.[3] Y. Matsuyama, K. Hosoya, H. Taniyama, H. Tsuboi, S. Fujie, T.Kobayashi, SCHEMA: Multi-party interaction-oriented humanoid robot,in: ACM SIGGRAPH ASIA 2009 Art Gallery & Emerging Technologies:Adaptation, 2009, pp. 82–82.[4] Apple Inc., Siri, http://www.apple.com/ios/siri/.[5] A. Lee, K. Oura, K. Tokuda, MMDAgent - A fully open-sourcetoolkit for voice interaction systems, in: Proc. of ICASSP 2013, 2013,pp. 8382–8385.[6] K. Oura, D. Yamamoto, I. Takumi, A. Lee, K. Tokuda, On-campus, user-participatable, and voice-interactive digital signage, J. Jpn. Soc. Artif.Intell. 28 (1) (2013) 60–67.[7] D. Yamamoto, K. Oura, R. Nishimura, T. Uchiya, A. Lee, I. Takumi,K. Tokuda, Voice interaction system with 3D-CG modeling for stand-alone smartphones, in: Proc. of International Conference on Human-Agent Interaction (IHAI) 2014, 2014, pp. 323–330.[8] T. Uchiya, D. Yamamoto, M. Shibakawa, M. Yoshida, R. Nishimura, I.Takumi, Development of spoken dialogue service based on video callnamed ‘Mobile Mei-chan’, in: Proc. of JAWS2012, Interaction, 2012,pp. 1–3.[9] VoiceXML Forum, VoiceXML: The standard application language forvoice dialogues, http://www.voicexml.org/.[10] K. Katsurada, Y. Nakamura, H. Yamada, T. Nitta, XISL:A language fordescribing multimodal interaction scenarios, in: Proc. of ICMI’03, 2003,pp. 281–284.

MMDAE : Dialog scenario editor for MMDAgent on the web browser

https://repo.lib.tokushima-u.ac.jp/files/public/11/115138/20200826162614363192/icte_5_1_47.pdf

MMDAE : Dialog scenario editor for MMDAgent on the web browser

Abstract

Similar works

Full text

Available Versions

Tokushima University Institutional Repository