Athanasopoulos, G

Beck, A

Cañamero, L

Cosi, P

Cuayáhuitl, H

Demiris, Y

Enescu, V

Hiolle, A

Kiefer, B

Kruijff-Korbayová, I

Racioppa, S

Ros, R

Sahli, H

Schröder, M

Sommavilla, G

Tesser, F

Verhelst, W

Wang, W

English

Spiral - Imperial College Digital Repository

A Conversational System for Multi-SessionChild-Robot Interaction with Several GamesIvana Kruijff-Korbayová1, Heriberto Cuayáhuitl1, Bernd Kiefer1, StefaniaRacioppa1, Piero Cosi2, Giulio Paci2, Giacomo Sommavilla2, Fabio Tesser2Hichem Sahli3, Georgios Athanasopoulos3, Weiyi Wang3, Valentin Enescu3,Werner Verhelst3 Lola Cañamero4, Aryel Beck4, Antoine Hiolle4 Raquel RosEspinoza5, Yiannis Demiris51 Language Technology Lab, DFKI, Saarbrücken, Germanyivana.kruijff@dfki.de2 Istituto di Scienze e Tecnologie della Cognizione, ISTC, C.N.R., Italy3 Interdisciplinary Institute for Broadband Technology - IBBT,Vrije Universiteit Brussel, Dept. of Electronics and Informatics, Belgium4 Adaptive Systems Research Group, School of Computer Science,University of Hertfordshire, United Kingdom5 Department of Electrical and Electronic Engineering, Imperial College London, UK1 IntroductionChildren are keen users of new technologies and new technologies can provideinteresting opportunities to enrich children’s experience, e.g., for educational andtherapeutic purposes.As children are not small adults, it is necessary to researchtheir specific needs and develop systems that address them. The aliz-e project6develops cognitive robots for adaptive social interaction with young users overseveral sessions in real-world settings. We demonstrate a conversational systemdeveloped in aliz-e using the Nao robot7. It engages a user in the followingactivities (Fig.1):– quiz: the child and the robot ask each other series of multiple-choice quizquestions from various domains, the robot provides evaluation feedback;– imitation: either the child or the robot presents a sequence of simple armposes that the other tries to memorize and imitate;– dance: the robot explores various dance moves with the child and thenteaches the child a dance sequence according to its abilitiesThese activities were chosen with regard to the target application domainof the system, namely long-term interaction with children hospitalized due tometabolic disorders, in particular diabetes. Quiz is a knowledge-exchange ativitymeant to support learning of health-related concepts. Due to its prediminantlyverbal character and constrained interaction structure it is a good testbed for6 The EU-FP7 project aliz-e (ICT-248116), http://aliz-e.org/7 http://www.aldebaran-robotics.com/enPreprint version; final version available at KI-2012: Poster and Demo TrackKI (2012), pp: 135-1392Fig. 1. Left to right : Nao in the measurement setup in a sound lab at VUB and playingQuiz, Imitation and Dance during experiments in the San Raffaele hospital in Milan.speech-processing technologies. Dance is an activity that promotes physical exer-cise. At the same time it provides a challenging domain for motion modeling andprocessing. Finally, Imitation on the one hand involves memory-exercise, and onthe other hand provides a gentle introduction to physical movement for thoseusers who are too shy to join the dance activity. It also involves an interestingmixture of verbal and non-verbal interaction, but more structured than Dance.Besides activity-specific conversation, the interactions involve also a socialcomponent (greetings, introductions). During an activity, the robot provides per-formance feedback to the user. The social aspect here requires careful handlingof the evaluation process so as not to discourage the user with negative feedback.As the system is designed to have multiple encounters with a user, the robot’sbehavior differs in various aspects from the first session (meeting for the firsttime) to the subsequent sessions (“knowing” the user and their performance).2 System DescriptionFig. 2 depicts the system components (more details below). We use the Urbimiddleware [3] to implement an event-based approach to integration [10].Motor Control  Control  (start or stop) Speech activity Gesture activity Speech activity Control  (start or stop) Gesture activity ASR result ASR result Text to synthesize Text to synthesize Text to synthesize Comm.goal Motor command NLU result AFE Audio data NVBP Comm.goal Comm.goal Audio segments of interest VAD SSL Source Localization Control  (start or stop) Speech activity TTS GRU GU and Pose result Motor command GU and Pose result Name, history,… Game results, … User Model Question+answers URBI EVENT HANDLER NLG Dialogue Manager(s) for Quiz/Imitation/Dance ASR GMG NLU Quiz DB Fig. 2. Left: The components of the integrated system. Filled boxes: components im-plemented in Java, double-line boxes: C/C++, and plain boxes: UrbiScript. The TTScomponent is either the Acapela TTS on the Nao or the Mary TTS implemented inJava. Right: The Quiz Game Wizard GUI.Preprint version; final version available at KI-2012: Poster and Demo TrackKI (2012), pp: 135-1393Speech Signal Detection and Capture. The Audio Front-End (AFE) componentcaptures the speech signal from the microphones, makes preliminary preprocess-ing such as sample rate conversion, and sends the audio buffers to the VoiceActivity Detection (VAD) component. VAD allows the robot to detect that dy-namically varying sound sources for further analysis are active, using a robustenergy based algorithm [8]. For the Sound Source Localization (SSL) compo-nent we implemented a Generalized Cross-Correlation based method with a setof pre-measured Time Delays On Arrival followed by parabolic interpolation [2].Spoken Input Processing The Automatic Speech Recognition (ASR) componentuses the Open-Source Large Vocabulary CSR Engine Julius8 for which we trainedan Italian child acoustic model. For demonstration purposes we also use an off-the-shelf ASR component for English.Further processing in the Natural Language Understanding (NLU) compo-nent proceeds along two paths: For the recognition of quiz questions, answeroptions and answers we use fuzzy matching of recignized content words againstthe Quiz DB entries. For the recognition of other dialogue acts we use eitherpartial parsing or keyword spotting as a fallback.Visual Input Processing For the Gesture recognition and Understanding com-ponent (GRU) we have been experimenting with various methods. For example,one method to trace hands in the Imitation Game uses skin detection [9] en-hanced with motion history [5]. An alternative method uses face detection andtracking algorithm [1] to define the vertical areas where the hands might move,and either motion detection or various optical flow algorithms.Dialogue Management Speech and gesture interpretations go to the DialogueManager (DM) that bears primary responsibility for controlling the robot’s con-versational behaviour and the game progress. It keeps track of the interactionstate, integrates the interpretations of the user’s input/actions w.r.t. this state,and selects the next action of the system as a transition to another state, makingprogress towards a goal. How exactly this is done depends on the game. For ex-ample, selecting the next suitable question in Quiz is done by a separate GameMove Generator (GMG) component that also accesses the Quiz DB.Inspired by the Zone of Proximal Development theory proposed by Vygotsky,the system takes the user’s performance into account. For example, in the danceactivity, the key point is to propose dance moves with gradually increasing ordecreasing complexity based on the user’s performance. On the one hand, theselected move should be within the user’s capabilities to avoid discouraging thechild, and on the other hand, challenging enough to maintain the child engagedin the task and willing to continue. The dance move selection mechanism thusconsiders: the level of complexity of the dance moves, a hierarchical representa-tion of dance moves and the user’s current potential capability to perform thedifferent moves. The quiz question selection algorithm similarly takes into ac-count the difficulty level of a question, whether it was asked already, and whether8 http://julius.sourceforge.jp/Preprint version; final version available at KI-2012: Poster and Demo TrackKI (2012), pp: 135-1394the user knew the answer. In both quiz and imitation, the robot makes mistakeson purpose, to maintain a performance level approximating that of the user.User-specific information (e.g., name, age) and interaction history (e.g., gamesplayed, achieved performance) are kept in the User Model (UM), which also re-ceives updates from the DM. Game-specific information is also stored.To experiment with the system without relying on fully automatic processing,we developed a Wizard-of-Oz interface (Fig. 2). Given a user input, the wizardcan select the corresponding user dialogue act’. The DM then selects the nextaction. In the automatic mode, the DM passes a dialogue act to the NLG andNVBP components. In a non-automatic mode, the action selected by the DM ishighlighted in the interface for the wizard to aprove or override. It is possible toswitch between automatic and wizarded DM at any time during a session.Spoken Output Production Spoken output is produced by the Natural LanguageGeneration (NLG) and Text-To-Speech Synthesis (TTS) components. The sys-tem action selected by the DM specifies the type of dialogue act and the values ofinformation state variables important for verbalization selection. Verbalizationis determined by an utterance planner using a set of graph rewriting rules. Theoutput is either a string passed directly to the TTS, or a logical form that servesas input to a grammar-based lexical realization component using OpenCCG9.Since repetitive verbalization of system output could be annoying and thusnegatively influence engagement, we implemented a large range of verbal outputvariation. Selection among variants is either random or controlled by selectionalcriteria, taking into account the content to be conveyed and the dialogue context.To foster a sense of familiarity between the robot and the user in interactionsover multiple sessions, the robot explicitly acknowledges and refers to commonground with a given user, thus making it explicit that it is familiar with them.For speech synthesis the commercial Acapela TTS system10 is available bydefault on the Nao. However, we also integrated the open source Mary TTSplatform11, for which we developed a new Italian voice. Mary TTS supportsstate of the art HMM-synthesis technology, and enables us to experiment withthe manipulation of para-verbal parameters (e.g. pitch shape, speech rate, voiceintensity, pause durations) for the purpose of expressive speech synthesis, andthe voice quality and timbre modifications algorithms [11] useful to convert anadult TTS voice into a child like voice.To further enhance the contextual appropriateness of the output speech weexperiment with modifications of the spoken output prosody using the supportfor controling the prosody of TTS voices with symbolic markup of speech rate,pitch and contour. So far, we implemented prosodic prominence modification(stress) on words that realize the focus of a sentence and emotional prosodymodification according to the emotional state of the robot (sad and happy).9 http://openccg.sourceforge.net/10 http://www.acapela-group.com/index.html11 http://mary.dfki.de/Preprint version; final version available at KI-2012: Poster and Demo TrackKI (2012), pp: 135-1395Nonverbal Behavior Production The Non-verbal Behavior Planning (NVBP) andMotor Control (MC) components produce arm gestures and head&body poses.Besides the game-specific moves and poses in the imitation and dance games,static key poses are produced to display emotions, namely anger, sadness, fear,happiness, excitement and pride [4].3 Demonstrated FeaturesThe present demonstration will focus in particular on spoken language inputand output processing. This includes robust natural language understanding,dialogue management based on hierarchical reinforcement learning [6] using flex-ible hierarchical dialogue control [7], varied verbalization production, familiarityacross multiple sessions and contextually controlled speech synthesis.References1. OpenCV library website ((accessed 1052012)), http://opencv.willowgarage.com2. Athanasopoulos, G., Brouckxon, H., Verhelst, W.: Sound source localization forreal-world humanoid robots. In: Proceedings of 11th International Conference onSignal Processing (SIP 2012). pp. 131 – 136. WSEAS (Mar 2012)3. Baillie, J.: URBI: Towards a Universal Robotic Low-Level Programming Language.In: 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems.pp. 3219–3224. IEEE (2005)4. Beck, A., Cañamero, L., Bard, K.: Towards an affect space for robots to displayemotional body language. In: Proceedings of the 19th IEEE international sympo-sium on robot and human interactive communication. pp. 464–469. Ro-Man 2010,IEEE (2010)5. Bradski, G., Davis, J.: Motion segmentation and pose recognition with motionhistory gradients. Machine Vision and Applications 13, 174–184 (2002)6. Cuayáhuitl, H.: Learning dialogue agents with bayesian relational state represen-tations. In: Proceedings of the IJCAI Workshop on Knowledge and Reasoning inPractical Dialogue Systems (IJCAI-KRPDS), Barcelona, Spain. pp. 9–15 (2011)7. Cuayáhuitl, H., Kruijff-Korbayová, I.: An interactive humanoid robot exhibitingflexible sub-dialogues. In: Proceedings of the North American Chapter of the As-sociation for Computational Linguistics: Human Language Technologies (NAACL-HLT), Montreal, Canada (2012)8. Dekens, T., Verhelst, W.: On the noise robustness of voice activity detection al-gorithms. In: Proceedings of 12th Annual Conference of the International SpeechCommunication Assosiation (INTERSPEECH). pp. 2649 – 2652. ISCA (Sep 2011)9. Jones, M.J., Rehg, J.M.: Statistical color models with application to skin detection.Int. J. Comput. Vision 46, 81–96 (2002)10. Kruijff-Korbayová, I., Athanasopoulos, G., Beck, A., Cosi, P., Cuayáhuitl, H.,Dekens, T., Enescu, V., Hiolle, A., Kiefer, B., Sahli, H., Schröder, M., Sommavilla,G., Tesser, F., Verhelst, W.: An event-based conversational system for the naorobot. In: IWSDS 2011. Granada, Spain (Sep 2011)11. Tesser, F., Zovato, E., Nicolao, M., Cosi, P.: Two Vocoder Techniques for Neutralto Emotional Timbre Conversion. In: Yoshinori Sagisaka, Tokuda, K. (eds.) 7thSpeech Synthesis Workshop (SSW). pp. 130–135. ISCA, Kyoto, Japan (2010)Preprint version; final version available at KI-2012: Poster and Demo TrackKI (2012), pp: 135-139

An event-based conversational system for the nao robot. In:

I.: An interactive humanoid robot exhibiting exible sub-dialogues. In:

Learning dialogue agents with bayesian relational state representations. In:

Motion segmentation and pose recognition with motion history gradients.

On the noise robustness of voice activity detection algorithms. In:

Sound source localization for real-world humanoid robots. In:

Statistical color models with application to skin detection.

Towards an aect space for robots to display emotional body language. In:

Two Vocoder Techniques for Neutral to Emotional Timbre Conversion.

URBI: Towards a Universal Robotic Low-Level Programming Language. In:

A conversational system for multi-session child-robot interaction with several games

A conversational system for multi-session child-robot interaction with several games

Abstract

Similar works

Full text

Available Versions

Spiral - Imperial College Digital Repository