Cafaro, Angelo

Chollet, Mathieu

Pecune, Florian

Pelachaud, Catherine

Philippe, Pierre

German

BieColl - Bielefeld eCollections

Suggestions for Extending SAIBA with the VIB PlatformFlorian Pecune1, Angelo Cafaro1, Mathieu Chollet2, Pierre Philippe2, and Catherine Pelachaud11CNRS-LTCI, Télécom ParisTech and 2Institut Mines-Télécom, Télécom ParisTech, CNRS-LTCI1 Main Research ThemesVirtual Interactive Behavior (VIB) is a SAIBA compliantplatform which supports the creation of socio-emotionalECAs. It takes as input utterances augmented with com-municative functions and emotional states specified inFML-APML (Carolis et al., 2004). The ECA spoken ut-terances are enriched by nonverbal behaviors NVB (gaze,facial expression, gesture). The choice of NVB can bemodulated by the definition of the dynamicline that is as-sociated to each ECA (Mancini and Pelachaud, 2008).The dynamicline specifies the preferred modalities andthe expressive parameters of each modality (Huang andPelachaud, 2012). These parameters act on the quality ofexecution of a behavior such as its speed and acceleration,its fluidity and amplitude.VIB allows the agent to be an active interactant(speaker or listener). The ECA can decide which socialattitudes to display towards its conversation partners (Pe-cune, 2013). These social attitudes can be shown by thechoice of its intentions (Callejas et al., 2014), the typeof the ECAs reactions (e.g. exhibit a polite or amusedsmile (Ochs and Pelachaud, 2013)) and its capacity tobe temporally aligned (Prepin et al., 2013). For exam-ple, an interpersonal attitude can be chosen at intentionallevel and the supporting utterances (Chollet et al., 2014)and multimodal behaviors (Ravenet et al., 2013) are pro-duced. While most of the behavior models integratedwithin VIB are procedural, Ding has used machine learn-ing techniques to drive the multimodal behaviors of theagent when saying emotional speech (Ding et al., 2013)and laughing (Ding et al., 2014).We are currently working on endowing the agent withthe capacity to show its engagement during the interac-tion by choosing its conversation topics (Glas in (Ochset al., 2013)) and by making use of hetero-repetition, onexpanding the expressivity model by analysing a largedatabase of motion capture data of expressive multimodalbehaviors (Fourati and Pelachaud, 2014), and on model-ing group behavior during conversations.2 Current Architectures and StandardsVIB has a modular and extensible architecture. Eachmodule represents an ECA’s functionality, therefore oneECA is defined by a particular set of modules as de-picted with three examples in Fig. 1. Different formatsare adopted to describe the actions that an ECA can per-form, these formats represent information ranging fromthe cognitive level (eg. communicative intentions) to thephysical level (eg. skeleton animations). These informa-tion are exchanged in the form of events and representthe input and output of the modules. Each module in VIBautomatically processes the received events.We developed a graphical user interface (GUI) that al-lows a user to design, instantiate and connect the mod-ules (e.g. FML/BML reader, behavior planner, 3D engineplayer, gesture editor, etc.).Figure 1: Three example architectures created in VIBCore Modules These modules include a SAIBA com-pliant Behavior Planner and Realizer that work with ourown FML-APML specification and extended BML fileformats respectively. The Behavior Realizer outputs arekeyframes performers used to compute the real-time ani-mation of the agent’s body and face. An external player(i.e. 3D engine) can be plugged into the system. Thissupports the rendering of the 3D environment with differ-ent engines (e.g. Ogre3D or Unity3D). Different agentsor users can be represented in VIB and share the sameplayer, therefore the same 3D environment.Other Modules The libraries of movements used arenot fixed and may be updated continuously. For this pur-pose, “editors” modules can be used to define the facialexpressions by action units (AU) (Ekman and Friesen,1978), the gestures (Kendon, 2004; McNeill, 1992), themapping between AU and FAP (MPEG-4, 2014), theshape of hands, etc. Network communications moduleshave been developed to interface with external softwareor to exchange events between different instances of theplatform over several machines. Currently, different APIsare used such as ActiveMQ and Thrift. A Neural Networkmodule provides an opportunity for the user to create neu-rons and connect them via a specific GUI. It can be usedto create real-time expressions by motor resonance.3 Future Architectures and Standards forIVAsUser Behavior Perception Many ECA systems at-tempted to recognize user behavior during the interac-tion (e.g. the SEMAINE project (Bevacqua et al., 2012)).There exist commercial and academic software for be-havior recognition (e.g. head tracking with OpenCV) andmental states (e.g. emotion recognition with SHORE).However, these are often heterogeneous and require ad-hoc development of interfaces for integration into ECAsystems. We advocate for a standard representation of theouptut provided by these applications. The PerceptionMarkup Language (PML) (Scherer et al., 2012), repre-sents a first attempt. However, in addition to the certaintyvalue and the sensory layer, PML redefines behavioraland functional levels (i.e. behavior and function layers),which could simply reuse the BML and FML specifica-tions.Multi-party and Multi-floor Interaction In face-to-face communication a person might be engaged in morethan one conversation at the same time with differentroles. For example, someone could listen to a talk andspeak to the person seated next by. This moves fromdyadic settings towards more complex multi-party sce-narios that should be represented both at functional (i.e.FML) and behavioral (i.e. BML) levels. An FML speci-fication addressing this aspect has been proposed by (Ca-faro et al., 2014) but it has not been adopted yet by anyECA system. We also think that representing more com-plex configurations at functional level and not only withBML may be important to affect the subsequent producedmultimodal behavior. A few examples are 1-to-many(e.g. describing a public speech) and many-to-many (twogroups interacting as a whole with each other).Transforming from FML to BML in SAIBA SAIBAcurrently does not specify how FML should be trans-formed into BML. We believe that this aspect cannot beleft aside of the framework with individual researchersproviding their “home made” solutions, as this may crit-ically impact the flexibility of integration into other sys-tems. Previous ECA systems have mainly adopted twostrategies to solve this problem that are broadly catego-rized as data-driven or procedural approaches (cf. in-troduction of Chapter 6 in (Cafaro, 2014) for a review). Ingeneral, our vision is that SAIBA should not only providestandardized interface languages but also techniques andmodules that enable to properly transfer between SAIBAcomponents the information represented by these inter-face languages.4 Suggestions for DiscussionRaw Perception vs. Attention We emphasized the im-portance of separating low level user’s behavior recogni-tion and its interpretation. An interesting aspect is alsothe distinction between raw environment perception andthe actual information processed by an ECA dependingon its attention level. In (Balint and Allbeck, 2013),agents’ perceptions are limited by their senses (i.e. solelywithin their field of view). However, the agent’s attentionlevel might filter out some raw information. In a crowdedscene, for example, an ECA in face-to-face interactionmight exclude some auditory or visual raw perceptions(e.g. other agents walking by) since its attention is fo-cused on the interactant, but another agent or an object(e.g. a car moving fast close by) might trigger an atten-tion shift. We suggest a discussion on how to separate rawperception and attention with emphasis on how to modelattention level.Dealing with Reactive Behaviors Reactive behaviorsare part of the interaction. For instance, an agent mightbend over to avoid a ball coming to him, or scream af-ter seeing a spider landing on its shoulder. According to(Scherer, 2001), a quicker response is needed for thosebehavior, may be bypassing the functional planning partof the SAIBA pipeline. The question is whether theselow-level quick reactions would fit in SAIBA or shouldbe modeled externally. (Bevacqua et al., 2008) attemptedto model this aspect, but on behavioral level (e.g. BML).Our question is what happens at the functional level? Ifan immediate reaction is required, should the pending in-tention be canceled, or re-scheduled to be accomplishedlater?From BML to Animation Similarly to transformingFML to BML, transforming planned BML into low levelparameters ready to be executed as animations by BMLRealizer might be problematic. Currently there is noguarantee that playing the same BML block on two dif-ferent realizers will lead to the same result. Is there a wayfor designers to be assured that the behaviors they arecreating will be played the same way, whatever the agentcould be? One solution to address this problem mightbe to set an additional standardization layer between thePlanner and the Realizer components in SAIBA whichcould provide lower level information (e.g. joint rotationsor animation parameters).ReferencesT. Balint and J. Allbeck. 2013. Whats going on?multi-sense attention for virtual agents. In R. Aylett,B. Krenn, C. Pelachaud, and H. Shimodaira, editors,Intelligent Virtual Agents, volume 8108 of LectureNotes in Computer Science, pages 349–357. SpringerBerlin Heidelberg.E. Bevacqua, K. Prepin, E. de Sevin, R. Niewiadomski,and C. Pelachaud. 2008. Reactive behaviors in saibaarchitecture. In AAMAS 2009 Workshop Towards aStandard Markup Language for Embodied DialogueActs, pages 9–12.E. Bevacqua, E. Sevin, S. Hyniewska, and C. Pelachaud.2012. A listener model: introducing personality traits.Journal on Multimodal User Interfaces, 6(1-2):27–38.A. Cafaro, H. Vilhjálmsson, T. Bickmore, D. Heylen, andC. Pelachaud. 2014. Representing communicativefunctions in saiba with a unified function markup lan-guage. In Proceedings of the 14th International Con-ference on Intelligent Virtual Agents (to appear), IVA’14.A. Cafaro. 2014. First Impressions in Human-AgentVirtual Encounters. Ph.D. thesis, Center for Analysisand Design of Intelligent Agents, Reykjavik Univer-sity, Iceland.Z. Callejas, B. Ravenet, M. Ochs, and C. Pelachaud.2014. A computational model of social attitudes fora virtual recruiter. In International Conference on Au-tonomous Agent and Multi-Agent Systems (AAMAS).B. D. Carolis, C. Pelachaud, I. Poggi, and M. Steedman.2004. Apml, a markup language for believable be-havior generation. In H. Prendinger and M. Ishizuka,editors, Life-like characters, Cognitive Technologies,pages 65–86. Springer.M. Chollet, M. Ochs, and C. Pelachaud. 2014. Mininga multimodal corpus for non-verbal signals sequencesconveying attitudes. In Language Resources and Eval-uation Conference (LREC).Y. Ding, C. Pelachaud, and T. Artires. 2013. Modelingmultimodal behaviors from speech prosody. In 13thInternational Conference of Intelligent Virtual Agents- IVA.Y. Ding, K. Prepin, J. HUANG, C. Pelachaud, and T. Ar-tires. 2014. Laughter animation synthesis. In Inter-national Conference on Autonomous Agent and Multi-Agent Systems (AAMAS).P. Ekman and W. Friesen. 1978. The Facial Action Cod-ing System: A Technique For The Measurement of Fa-cial Movement. Consulting Psychologists Press, PaloAlto, CA, USA.N. Fourati and C. Pelachaud. 2014. Emilya: Emotionalbody expression in daily actions database. In Lan-guage Resources and Evaluation Conference (LREC).J. Huang and C. Pelachaud. 2012. Expressive bodyanimation pipeline for virtual agent. In proceedingsof 12th International Conference of Intelligent VirtualAgents - IVA, pages 355–362.A. Kendon. 2004. Gesture: Visible action as utterance.Cambridge University Press.M. Mancini and C. Pelachaud. 2008. Distinctiveness inmultimodal behaviors. In 7th International Joint Con-ference on Autonomous Agents and Multiagent Sys-tems (AAMAS 2008), pages 159–166, Estoril, Portugal.D. McNeill. 1992. Hand and mind: What gestures revealabout thought. University of Chicago Press.MPEG-4. 2014. http://mpeg.chiariglione.org/standards/mpeg-4.M. Ochs and C. Pelachaud. 2013. Socially aware virtualcharacters: The social signal of smiles. IEEE SignalProcess. Mag., 30(2):128–132.M. Ochs, Y. Ding, N. Fourati, M. Chollet, B. Ravenet,F. Pecune, N. Glas, K. Prpin, C. Clavel, andC. Pelachaud. 2013. Vers des agents conversationnelsanims socio-affectifs. In Interaction Humain-Machine(IHM’13).F. Pecune. 2013. Toward a computational model of so-cial relations for artificial companions. In HumaineAssociation Conference on Affective Computing andIntelligent Interaction, ACII, pages 677–682.K. Prepin, M. Ochs, and C. Pelachaud. 2013. Be-yond backchannels: co-construction of dyadic stancceby reciprocal reinforcement of smiles between virtualagents. In International Conference CogSci (AnnualConference of the Cognitive Science Society).B. Ravenet, M. Ochs, and C. Pelachaud. 2013. Froma user-created corpus of virtual agent’s non-verbal be-havior to a computational model of interpersonal atti-tudes. In 13th International Conference of IntelligentVirtual Agents - IVA, pages 263–274.S. Scherer, S. Marsella, G. Stratou, Y. Xu, F. Morbini,A. Egan, A. Rizzo, and L.-P. Morency. 2012. Per-ception markup language: Towards a standardizedrepresentation of perceived nonverbal behaviors. InY. Nakano, M. Neff, A. Paiva, and M. Walker, edi-tors, Intelligent Virtual Agents, volume 7502 of LectureNotes in Computer Science, pages 455–463. SpringerBerlin Heidelberg.K. R. Scherer. 2001. Appraisal considered as a processof multilevel sequential checking. Appraisal processesin emotion: Theory, methods, research, 92:120.Biographical SketchesFlorian Pecune is a PhD can-didate at the LTCI laboratoryof Telecom Paristech. His re-search activities are focused ondesigning Embodied Conversa-tional Agents able to adapt theirdecision making in particular so-cial contexts. As part of the ANRMoCA project, which aims at creating a connected worldof artificial companions, he proposed a model to computethe social attitude of an agent according to its goals andbeliefs.Angelo Cafaro is a postdoctoralresearcher at CNRS-LTCI, Tele-com ParisTech. He is doingresearch in the area of embod-ied conversational agents and se-rious game environments withemphasis on social interaction,group behavior and expressionof social attitudes. Angelo is part of the EU FP7 Verveproject, which aims at developing serious games to sup-port the treatment of elderly people who are at risk ofsocial exclusion. He obtained his Ph.D. from ReykjavikUniversity in 2014. His dissertation dealt with analyz-ing and modeling human nonverbal communicative be-havior exhibited by a virtual agent in a first greeting en-counter with the user. In his dissertation he also pro-posed a SAIBA compliant computational model featur-ing a unified specification for the Function Markup Lan-guage (FML). More information is available on his per-sonal webpage: www.angelocafaro.info.Mathieu Chollet is a PhD can-didate at the LTCI laboratory ofTelecom Paristech. His researchactivities are focused on Embod-ied Conversational Agents andtheir applications for social skillstraining. As part of the EU FP7TARDIS project, which aims atimproving youngsters’ job interview skills, he proposedbehavior models of attitude expression for virtual re-cruiters. Additionally, he was involved as a Visit-ing Researcher at the Institute for Creative Technolo-gies where he designed an interactive virtual audi-ence architecture for public speaking training. Personalwebpage: http://perso.telecom-paristech.fr/˜mchollet/Pierre Philippe is a research en-gineer at the LTCI laboratory ofTelecom ParisTech. He receiveda Master degree in ComputerScience and a M.S. degree inArtificial Intelligence from Brus-sels Polytechnic School. Heworked as a consultant in Com-puter Associates, then as a research engineer in IRIDIAlab at Brussels University (Belgium) and in COIN labat Skövde University (Sweden). He is currently mod-elling and programming modules for the Greta platform.His research interest includes Embodied ConversationalAgents, emotions modelling and cognitive architectures.Catherine Pelachaud is a Di-rector of Research at CNRS inthe laboratory LTCI of TelecomParisTech. Her research inter-est includes embodied conversa-tional agent, nonverbal commu-nication (face, gaze, and ges-ture), expressive behaviors andsocio-emotional agents.E-Mail Contacts (@telecom-paristech.fr)Florian Pecune: florian.pecuneAngelo Cafaro: angelo.cafaroMathieu Chollet: mathieu.cholletPierre Philippe: pierre.philippeCatherine Pelachaud: catherine.pelachaud

Suggestions for Extending SAIBA with the VIB Platform

https://biecoll.ub.uni-bielefeld.de/index.php/wasiva/article/download/604/699

Suggestions for Extending SAIBA with the VIB Platform

Abstract

Similar works

Full text

Available Versions

BieColl - Bielefeld eCollections