Advances in animation and sensor technology allow us to engage in face-to-face conversations with virtual agents [1]. One major challenge is to generate the virtual agent’s appropriate, human-like behavior contingent with that of the human conversational partner. Models of (nonverbal) behavior are pre-dominantly learned from corpora of dialogs between human subjects [2], or based on simple observations from literature (e.g. [3,4,5,6]

Heylen, Dirk K.J.

Poppe, Ronald Walter

ter Maat, Mark

English

University of Twente Research Information

Online Behavior Evaluation with theSwitching Wizard of OzRonald Poppe, Mark ter Maat, and Dirk Heylen ?Human Media Interaction Group, University of TwenteP.O. Box 217, 7500 AE, Enschede, The Netherlands{r.w.poppe,m.termaat,d.k.j.heylen}@utwente.nl1 IntroductionAdvances in animation and sensor technology allow us to engage in face-to-faceconversations with virtual agents [1]. One major challenge is to generate thevirtual agent’s appropriate, human-like behavior contingent with that of the hu-man conversational partner. Models of (nonverbal) behavior are pre-dominantlylearned from corpora of dialogs between human subjects [2], or based on simpleobservations from literature (e.g. [3–6]).Humans are particularly sensitive to flaws in the displayed behavior, bothin form and timing [7, 8]. This effect also occurs when certain behaviors are notanimated, which is common in experimental settings where the behavior of thevirtual agent is varied systematically only one or a few modalities [9, 10]. Thisleads to biased perceptual ratings, which hampers progress in the design andimplementation of behavior synthesis algorithms.To this end, we propose a methodology and implementation that combinesideas behind the human Turing test with those of a Wizard of Oz setup. At theheart of our approach is a distributed (video-conferencing) setting with two hu-man conversational partners. Each of the subjects is observed with camera andmicrophone and algorithms are employed to analyze the verbal and nonverbalbehavior in real-time (similar to e.g., [11–13]). These observations are used asinput to a behavior synthesis model. Both subjects are shown a virtual repre-sentation of the other (see Fig. 2), animated based on one of two sources: (1)directly on the observed behavior of the other, or (2) on the output of the syn-thesis model. Both sources share the same behavior animation capabilities andlimitations. We can therefore analyze the effect of the quantity, type and timingof the nonverbal behaviors on the perception thereof. During a conversation, thesource of animation of the representation of each subject switches occasionally.The idea behind the framework is that, when the displayed behavior deviatesfrom what is typically regarded as human-like, the observer should notice. In thiscase, he or she is instructed to press a button (the yuck button [10]). The ratingscan be used to evaluate and improve the behavior synthesis models (e.g. [14]). Asobservations of the subjects are continuously recorded, the framework doublesas a tool for study into nonverbal behavior.? This publication was supported by the Dutch national program COMMIT.22 Switching Wizard of OzIn the Switching Wizard of Oz (SWOZ) setting, two human subjects A andB, seated at distributed locations, are shown virtual representations of eachother. The representation of B displays either the behavior performed by B, orbehavior synthesized by an algorithm, based on audio or video observations ofA. The behaviors displayed by the virtual representations can be discrete (e.g.nods) or continuous (e.g. head movement). During a conversation, the source ofa virtual representation is switched at random time intervals. To evaluate thequality of behavior synthesis models, both subjects are presented with a yuckbutton which they press whenever they believe the displayed behavior does notoriginate from the other subject.AVirtual BVirtual ABehaviorsynthesismodelBehaviorsynthesismodelBSwitchSwitchFig. 1. Schematic representation of the Switching Wizard of Oz framework.Subject observation The conversational partners are observed via sensorssuch as cameras, microphones, Kinects and gaze trackers. The observations areencoded into features in real-time. It should also be possible to regenerate theobserved behavior on the virtual representation of the subject.Behavior synthesis These extracted features are subsequently used in a be-havior synthesis algorithm, to determine whether or not certain behaviors shouldbe animated. These algorithms can be manually engineered sets of classificationrules (e.g. [3, 5]) or machine learning classifiers trained on previously recordedcorpus data (e.g. [6]). Based on the outcome of the algorithm or the observationsof the actual conversational partner, the behavior is animated on a virtual agent.Behaviors can be verbal and nonverbal, discrete and continuous.Behavior switching The framework switches between the two sources atrandom time intervals. The displayed behavior should be continuous. For discreteevents, this implies that the currently animated behavior should be finished anda new behavior should not be directly animated. For continuous behaviors, itshould also be ensured that the displayed behavior is continuous so the switchingmoment will not be perceived as such to the observer. As the switching compo-nent of the framework is presented with the behavior of both the conversationalpartner and the algorithm, the switching time can be selected when the twosources are more or less similar, to allow for interpolation between the two.3References1. Heylen, D., Bevacqua, E., Pelachaud, C., Poggi, I., Gratch, J., Schröder, M.: Gener-ating Listening Behaviour. In: Emotion-Oriented Systems Cognitive Technologies- Part 4. Springer (2011) 321–3472. Martin, J.C., Paggio, P., Kuehnlein, P., Stiefelhagen, R., Pianesi, F.: Introduc-tion to the special issue on multimodal corpora for modeling human multimodalbehavior. Language Resources and Evaluation 42(2) (2008) 253–2643. Ward, N., Tsukahara, W.: Prosodic features which cue back-channel responses inEnglish and Japanese. Journal of Pragmatics 32(8) (2000) 1177–12074. Cathcart, N., Carletta, J., Klein, E.: A shallow model of backchannel continuers inspoken dialogue. In: Proceedings of the Conference of the European chapter of theAssociation for Computational Linguistics - Volume 1, Budapest, Hungary (2003)51–585. Truong, K.P., Poppe, R., Heylen, D.: A rule-based backchannel prediction modelusing pitch and pause information. In: Proceedings of Interspeech, Makuhari,Japan (2010) 490–4936. Morency, L.P., de Kok, I., Gratch, J.: A probabilistic multimodal approach forpredicting listener backchannels. Autonomous Agents and Multi-Agent Systems20(1) (2010) 80–847. McDonnell, R., Ennis, C., Dobbyn, S., O’Sullivan, C.: Talking bodies: Sensitivityto desynchronization of conversations. ACM Transactions on Applied Perception6(4) (2009) A228. Hodgins, J., Jörg, S., O’Sullivan, C., Park, S.I., Mahler, M.: The saliency of anoma-lies in animated human characters. ACM Transactions on Applied Perception 7(4)(2010) A229. Poppe, R., Truong, K.P., Reidsma, D., Heylen, D.: Backchannel strategies forartificial listeners. In: Proceedings of the International Conference on InteractiveVirtual Agents (IVA), Philadelphia, PA (2010) 146–15810. Poppe, R., Truong, K.P., Heylen, D.: Backchannels: Quantity, type and timingmatters. In: Proceedings of the International Conference on Interactive VirtualAgents (IVA), Reykjavik, Iceland (2011) 228–23911. Bailenson, J.N., Yee, N., Patel, K., Beall, A.C.: Detecting digital chameleons.Computers in Human Behavior 24(1) (2008) 66–8712. Edlund, J., Beskow, J.: Mushypeek: A framework for online investigation of au-diovisual dialogue phenomena. Language and Speech 52(2–3) (2009) 351–36713. Huang, L., Morency, L.P., Gratch, J.: Virtual rapport 2.0. In: Proceedings of theInternational Conference on Interactive Virtual Agents (IVA), Reykjavik, Iceland(2011) 68–7914. de Kok, I., Poppe, R., Heylen, D.: Iterative perceptual learning for social behaviorsynthesis. Technical Report TR-CTIT-12-01, University of Twente (2012)

Online behavior evaluation with the switching wizard of Oz

A probabilistic multimodal approach for predicting listener backchannels.

A rule-based backchannel prediction model using pitch and pause information. In:

A.C.: Detecting digital chameleons.

Backchannel strategies for arti listeners. In:

Backchannels: Quantity, type and timing matters. In:

E.: A shallow model of backchannel continuers in spoken dialogue. In:

Generating Listening Behaviour. In: Emotion-Oriented Systems Cognitive Technologies - Part 4.

Introduction to the special issue on multimodal corpora for modeling human multimodal behavior.

Iterative perceptual learning for social behavior synthesis.

Mushypeek: A framework for online investigation of audiovisual dialogue phenomena. Language and Speech 52(2{3)

Prosodic features which cue back-channel responses in English and Japanese.

Talking bodies: Sensitivity to desynchronization of conversations.

The saliency of anomalies in animated human characters.

Virtual rapport 2.0. In:

https://research.utwente.nl/en/publications/193e8165-7e86-421e-bbc5-ee37114c1732

Online behavior evaluation with the switching wizard of Oz

Abstract

Similar works

Full text

Available Versions

University of Twente Research Information