Carlmeyer B, Betz S, Wagner P, Wrede B, Schlangen D. The Hesitating Robot - Implementation and First Impressions. In:  Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction  - HRI '18. New York, NY, USA: ACM Press;  2018: 77-78

Betz, Simon 

Carlmeyer, Birte 

Schlangen, David 

Wagner, Petra 

Wrede, Britta

Publications at Bielefeld University

The Hesitating Robot - Implementation and First ImpressionsBirte Carlmeyer∗†‡bcarlmey@techfak.uni-bielefeld.deSimon Betz∗‡§simon.betz@uni-bielefeld.dePetra Wagner∗§petra.wagner@uni-bielefeld.deDavid Schlangen∗‡david.schlangen@uni-bielefeld.deBritta Wrede∗†bwrede@techfak.uni-bielefeld.deABSTRACTIn this paper we present the implementation of a robot, that dynam-ically hesitates, based on the attention of the human interactionpartner. To this end, we outline requirements for a real-time inter-action scenario, describe the realization of a disfluency insertionstrategy, and present observations from the first tests of the system.CCS CONCEPTS• Human-centered computing→ Human computer interac-tion (HCI); •Computingmethodologies→Natural languageprocessing;ACM Reference Format:Birte Carlmeyer, Simon Betz, Petra Wagner, David Schlangen, and BrittaWrede. 2018. The Hesitating Robot - Implementation and First Impressions.In HRI ’18 Companion: 2018 ACM/IEEE International Conference on Human-Robot Interaction Companion, March 5–8, 2018, Chicago, IL, USA. ACM, NewYork, NY, USA, 2 pages. https://doi.org/10.1145/3173386.31769921 INTRODUCTIONHumans observe their interaction partner, among other reasons, inorder to detect communication mistakes or figure out how attentivethe interlocutor is in the conversation. Based on feedback signalswe adapt our behavior to manage conversations. One adaptationstrategy is hesitation, through pauses, repetitions, or fillers, to ob-tain extra time without losing the conversational floor. Additionally,[8] indicate that hesitations heighten the listeners’ attention to up-coming speech. To have a situated human-robot interaction (HRI),it is necessary that the robot can recognize human feedback signals(e.g., social eye-gaze [1]) and is able to react to them, (e.g., withhesitations).Little analysis has been conducted on the impact of hesitationson human-robot interaction. [4] use filled pauses ("So..." "Let’s see...")between dialogue acts to give the system more time for perceptionand decision making. The authors evaluated the disengagementcosts and could show, that the use of hesitations in combination∗Cluster of Excellence Cognitive Interaction Technology (CITEC), Bielefeld University†Applied Informatics Group, Faculty of Technology, Bielefeld University‡Dialogue Systems Group, Faculty of Linguistics and Literary Studies§Phonetics and Phonology Workgroup, Faculty of Linguistics and Literary StudiesPermission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).HRI ’18 Companion, March 5–8, 2018, Chicago, IL, USA© 2018 Copyright held by the owner/author(s).ACM ISBN 978-1-4503-5615-2/18/03.https://doi.org/10.1145/3173386.3176992with a forecasting model of disengagement lead to less costly dis-engagement management, but no evaluation of the effect on thehuman interaction partner, e.g., on task performance or subjectiverating of the robot are conducted.[6, 7] evaluated self-interruptions in an smart-home setting. Theauthors use silence as an attention-regaining strategy whenever theattention of the human interaction partner moves away. They couldshow, that this strategy has a positive effect on the visual attentionof the human, but at the cost of less positive subjective ratings.The attention-regaining strategy was effective but the agent wasperceived as rude and less friendly.Based on a human-human interaction corpus analysis, [3] pro-pose a disfluency insertion strategy for synthetic speech, whichconsist of the following cascade: (i) lengthening: add lengtheningat the next appropriate syllable (ii) silence: insert a silence for amaximum of 1000ms (iii) filler: insert a filler (e.g., "uhm") (iv) silence:insert an additional silence. Whereas in [7] the attention-regainingstrategy consists of a simple pausing of the synthesis (while theinteraction partner is not attentive), this strategy has several levelsof escalation. To counteract the perceived rudeness observed in [6],we integrate this hesitation strategy into our dialogue system forsmoother transitions from fluent delivery to hesitation mode.Figure 1: System overview of the main components.2 REQUIREMENTS & IMPLEMENTATIONTo manage the dialogue between a human and the robot, two mainquestions have to be answered: (i) When to (re-)act? and (ii) Howto (re-)act? To implement hesitations, based on the attention of theinteraction partner, several requirements have to be met.Perception of the interaction partner. The robot needs to beable to perceive the human interaction partner. To answer the firstquestion -when to react? - it is necessary to observe the interlocutor.For this purpose we use, among other things, a gaze detector [9] toassess the current visual focus of attention (VFoA).Model of attention. Based on the perception, it has to decidewhether the interaction partner is attentive or not. The robot needsan internal concept of attention. Here we define attention as a statein which the VFoAmatches with current focus of discourse (FoD)(cf.[6]).Session: Late-Breaking Reports HRI’18 Companion, March 5-8, 2018, Chicago, IL, USA77Figure 2: Production of the wordIU’s of the hesitation module: (t1) module receives hesitation start event; (t2) best entry pointfor the hesitation; (t3) module receives hesitation end signal. Color scheme of the different incremental units: (dark gray)already synthesized; (blue) hesitation insertions; (green) revoked wordIU’s.Incremental processing capabilities. In order to realize the pro-posed hesitation insertion strategy - in this case the answer of thequestion how to react? - we need the possibility to change the ongo-ing speech plan. Therefore the capability of incremental processing,especially on the synthesis level, is mandatory. We realize this byusing the incremental processing framework InproTK [2].Figure 1 shows an overview of the main parts of the currentsystem. The agent can perceive the human interaction partner viaseveral sensors, in this case via a microphone and a small web-cam. The attention management receives information from thegaze detector [9] - the VFoA - and context information from thedialogue management - the current FoD - and provides informationabout the estimated attention state of the human interaction partner(attentive, not-attentive). The dialogue management [5] can triggerthe motor control of the robot, e.g., to show attention at the currentFoD by looking at it and is able to send utterances to text-to-speechsynthesis module (tts). Based on the attention state, the dialoguemanager can also start and stop the hesitation strategy.Figure 2 shows an example of the hesitation strategy, which isimplemented as separate module in InproTK, an implementation ofthe general, abstract model for incremental processing [10]. Themodel consists of a network of processingmodules, which exchangeincremental data in form of incremental units (IU). The IU-modulesreceive information on their left buffer, perform some kind of pro-cessing on these IU’s and provide output on their right buffer. Atthe moment t1 the module receives the event ’start hesitation’. Thestrategy takes the IUs from the left buffer of the synthesis module,in this case a list of wordIUs, each representing a single word. Itsearches for the best entry point and lengthens the most appropri-ate segments. In this example, the synthesis module already playedback the first two wordIU s "my" and "name". The rest of the currentphrase ("is Flobi and I’m the ...") is still in the playback pipeline.According to the proposed strategy, the best entry point for thehesitation strategy is the wordIU "and" (t2 ), which is then stretchedby a factor based on the findings of [3]. Then the synthesis modulewill be paused up to 1000ms (< sil/>). If this is not enough time,the module inserts a filler ("uhm"), also applied with lengthening,followed by a second pause until the dialogue management stopsthe hesitation strategy (t3). In the case the dialogue managementwants to stop the hesitation strategy earlier (e.g., the estimatedattention state of the human interaction partner changed to atten-tive), the strategy can be interrupted at several points: (i) beforethe entry point t2: without any effect in the synthesis (ii) beforethe filler ("uhm"): the lengthening will be produced, but the silencecan be interrupted (iii) during or after the filler: the filler will beproduced, but the silence is again interruptible.3 CURRENT STATE & FUTUREWORKWe tested our system in a small pilot study (n=4) in an interac-tion scenario in a smart apartment to get some insights and firstimpressions of our system. As a platform we use the simulationof the anthropomorphic robot head Flobi. The system is mostlyworking as expected. The hesitation strategy starts if the humanis not attentive (looking away) and stops when the user refocuses.We observed some issues, which need to be addressed before wecan evaluate our system in an interaction study. The insertion ofthe filler sometimes leads to noise inferences, which need to beeliminated. Additionally, producing fillers such as "uhm" is not atrivial issue, because they are normally not part of the trainingcorpus for speech synthesis voices, and at least for German, theycannot be synthesized out-off-the-box with the tts system at handwithout further acoustic modification. We need to investigate ifthe participants correctly interpret the intention of these fillers ashesitations. After these issues are solved, the next step will be anevaluation study to investigate the effect of this hesitation strategyon the attention, task progress, and the subjective ratings of theagent in order to test if the lengthening and the insertion of fillerscan counteract the in [7] perceived rudeness of self-interruptions.4 ACKNOWLEDGMENTSThis work was funded as part of the Cluster of Excellence CognitiveInteraction Technology ’CITEC’ (EXC 277), Bielefeld University.REFERENCES[1] Henny Admoni and Brian Scassellati. 2017. Social Eye Gaze in Human-RobotInteraction: A Review. 6 (03 2017), 25.[2] Timo Baumann and David Schlangen. 2012. The InproTK 2012 release (Proc. ofthe NAACL-HLT Workshop in SDCTD 2012). ACL, 29–32.[3] Simon Betz, Petra Wagner, and Jana VoÃ§e. 2016. Deriving a strategy for synthe-sizing lengthening disfluencies based on spontaneous conversational speech data.In Tagungsband der 12. Tagung Phonetik und Phonologie im deutschsprachigenRaum. LMU, 19–22.[4] Dan Bohus and Eric Horvitz. 2014. Managing Human-Robot Engagement withForecasts and... Um... Hesitations. In Proc. of the 16th ICMI. ACM, 2–9.[5] Birte Carlmeyer, David Schlangen, and Britta Wrede. 2014. Towards ClosedFeedback Loops in HRI. In Proceedings of ICMI-MMRWHRI ’14. ACM, 1–6.[6] Birte Carlmeyer, David Schlangen, and Britta Wrede. 2016. Exploring self-interruptions as a strategy for regaining the attention of distracted users. InProc. of - EISE ’16. ACM.[7] Birte Carlmeyer, David Schlangen, and Britta Wrede. 2016. "Look at Me!": Self-Interruptions as Attention Booster?. In Proc. of HAI ’16. ACM.[8] Philip Collard. 2009. Disfluency and listeners’ attention: An investigation of theimmediate and lasting effects of hesitations in speech. Ph.D. Dissertation. Universityof Edinburgh.[9] L. Schillingmann and Y. Nagai. 2015. Yet another gaze detector. In 2015 IEEE-RAS15th Humanoids. 8–13.[10] David Schlangen and Gabriel Skantze. 2011. A General, Abstract Model ofIncremental Dialogue Processing. Dialogue and Discourse 2, 1 (2011), 83–111.Session: Late-Breaking Reports HRI’18 Companion, March 5-8, 2018, Chicago, IL, USA78

The Hesitating Robot - Implementation and First Impressions

https://pub.uni-bielefeld.de/download/2918555/2918556

The Hesitating Robot - Implementation and First Impressions

Abstract

Similar works

Full text

Available Versions

Publications at Bielefeld University