Bruijnes, Merijn

Falcone, Sara

English

University of Twente Research Information

The Embodied Debate CorpusProposed corpus recordings of multi-party multi-embodiment multi-measure debatesMerijn BruijnesUniversity of TwenteThe Netherlandsm.bruijnes@utwente.nlSara FalconeFondazione Bruno KesslerItalysara.falcone93@gmail.comKEYWORDSCorpus Recordings, Conversational Agents, Embodiment Fidelity,Echoborg, Cognitive PragmaticsACM Reference Format:Merijn Bruijnes and Sara Falcone. 2018. The Embodied Debate Corpus: Pro-posed corpus recordings of multi-party multi-embodiment multi-measuredebates. In Proceedings of 18th ACM International Conference on Intelli-gent Virtual Agents’ workshop on Methodology and/of Evaluation of IVAs(IVA2018 Workshop on Methodology). ACM, New York, NY, USA, 3 pages.https://doi.org/10.1145/nnnnnnn.nnnnnnn1 INTRODUCTIONIn this paper we outline our plans to record a novel corpus thatcan be used to investigate how the pragmatics of the interaction,particularly the embodiment of the interlocutor, influence the in-formation perceived by a human and her/his experience during theinteraction. The purpose of this paper is twofold: 1) discuss ourplans with the community and get feedback on this novel method-ology, and 2) it serves as a preregistration of the work.Wewill create a public corpus that consists of interactions amongthree interlocutors debating an ethical dilemma. The participantsperform specific roles: moderator, proponent, and opponent. Thetopic of the debate is the trolley dilemma as are many of argumentsfor and against, and there is no correct answer [17]. To investigatethe effects of embodiment and agency, we conceive four variantsof interactions in the corpus:• Human-Human-Human (HHH) Corpus: three participantsare instructed before interacting on their role in the debate,their interactions are recorded (audio-video), and afterwardsthey fill out questionnaires about the interaction and theperceptions they have of their interlocutors;• Agent-Agent-Agent (AAA) Corpus: virtual agents debatingthe same topic. The behavior of the agents is based on thelinguistic information and pragmatics in the HHH corpus;• Human-Agent-Agent (HAA) Corpus: one participant (hu-man) interacting with two virtual agents. The agents’ behav-ior is based on the HHH corpus. The human has the role ofmoderator;Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).IVA2018 Workshop on Methodology, November 5, 2018, Sydney, Australia© 2018 Copyright held by the owner/author(s).ACM ISBN 978-x-xxxx-xxxx-x/YY/MM.https://doi.org/10.1145/nnnnnnn.nnnnnnn• Human-Echoborg-Agent (HEA) Corpus: one participant, oneechoborg, and one virtual agent interact. An echoborg is aconfederate who acts as the embodiment of a virtual agent:the virtual agent decides what to say, conveys this covertlyto the confederate, and he/she utters these words verbatim.The participant is unaware of the nature of the echoborg [7].The agent’s and echoborg’s behaviors are based on the HHHcorpus;Each interaction will be recorded (audio-video) and transcribed.After each interaction, each (human) participant fills out question-naires about the interaction and the perceptions that they have oftheir interlocutors. This data will be congregated into a corpus thatwill be made publicly available.Next we give some background about our rationale and after that,in section 2, we describe the procedure that we will use to collectthe data. In section 2.3 we present the questionnaires that we willadminister to the participants in order to compare their perceptionof the interaction. And in section 3 we discuss the methodology.1.1 BackgroundThe interpretation of a particular utterance can be almost infinitelyvariable. It depends on factors such as the identity of the speaker, thephysical context of use, and the preceding discourse. The contextualflexibility of language (its pragmatics) and the systematization of itsstructural features are still open issues, particularly their formaliza-tion [9]. Cognitive pragmatics is the study of the mental structuresand processes involved in the use of language in communicativecontexts. Paradigms of cognitive psychology have been applied tostudy the abilities of humans to go beyond the literal (inference) andderive meaning in relation to context. Contextual manipulationsidentify an overt issue of research in cognitive pragmatics [3]. Thechallenge for cognitive pragmatics, as it is for all psycholinguistics,is to construct experiments that help to identify the obligatory pro-cesses occurring beneath human awareness [8]. Our work can aidthese fields studying and manipulating a particular aspect of thecontext, namely the embodiment (or appearance) of an interlocutor.The physical embodiment has a measurable effect on perfor-mance and impression of social interactions [21]. There are studiesinvestigating how a different embodiment of an interlocutor canaffect the resulting interaction with a human user (e.g. for a hu-manoid virtual agent [22], robot [13], or avatar [2]). For example,when a human user interacts with a non-human, regardless the em-bodiment of the agent, s/he will tend to simplify her/his languageand to have low expectation about the interaction [1]. One thing isclear, all of these experiments have to deal with an apparent artifi-cial embodiment and this is likely to affect the user’s perception ofthe agent. This is in addition to the artificial nature of the ‘mind’ ofIVA2018 Workshop on Methodology, November 5, 2018, Sydney, Australia Merijn Bruijnes and Sara Falconea virtual agent, which is also likely to influence the perception of auser. Yet, without disentangling the artificial embodiment from theartificial mind the contribution of each of these components to theuser’s perception remains unclear. We propose that the perceptionof the ‘natural’ human embodiment of an echoborg does not affectthe perception of the artificial mind. Thus, looking at echoborgs willshow more clearly to what extend an artificial mind is perceived bya user as ‘human-like’. Our corpus will provide researchers a way toanalyze and disentangle the effects of artificialness of embodimentand mind.2 THE METHODOLOGYIn this section we describe our methodology: the planned procedurefor the collection of the corpus, the specifics of the procedure foreach of the four parts of the corpus, and the measures.2.1 Trolley DilemmaThe trolley dilemma is a thought experiment from ethics and willbe debated by the participants. The task is to find the most ethicalsolution to the following problem: “You see a runaway trolley movingtoward five tied-up (or otherwise incapacitated) people lying on thetracks. You are standing next to a lever that controls a switch. If youpull the lever, the trolley will be redirected onto a side track and thefive people on the main track will be saved. However, there is a singleperson lying on the side track.”1 Two options exist:(1) Do nothing and allow the trolley to kill the five people onthe main track.(2) Pull the lever, diverting the trolley onto the side track whereit will kill one person.The debating participants are instructed to defend one of the op-tions, where the proponent defends acting (switching) and theopponent defends doing nothing.2.2 The ProcedureThe total time of a session is expected to be around 20 minutes (10minutes for the debate and 10 minutes to fill the consent form andthe questionnaire). The proponent and the opponent we will sitopposite each other, with the moderator in the middle. For eachparticipant there is a dedicated microphone and a dedicated cameradirected at only that participant. This way we can anonymise/deletethe data of participants who do not want to share (parts of) theirdata without having to remove the data of all participants in thatsession. For each recording where participants are involved thefollowing is procedure is planned:• Recruitment: Participants are recruited from students andstaff from the researchers’ university (convenience sample)on a voluntary basis.• Welcome and briefing: After welcoming the participantthey are briefed: they are explained what will be recorded,why it is recorded, and they are informed about their rights.After informed consent is obtained, we explain that they willtake part in a debate about the Trolley Dilemma and whattheir role is during the debate.1From Wikipedia (October, 2018): https://en.wikipedia.org/wiki/Trolley_problem• Recording: The recording equipment is turned on and synced.The participants have their discussion (max 10 minutes).• Questionnaires: The participants fill out questionnaires.They do this separated from each other.• Data collection and reset.2.2.1 HHH Corpus. The participants are instructed to freely dis-cuss the trolley dilemma, but to try and keep the conversation clearby speaking one-by-one. The roles are assigned randomly.2.2.2 AAA Corpus. The arguments that the participants in theHHH corpus use, are extracted and defined in an agent argumenta-tion engine [15]. The system provides a selection of arguments thatare possible (appropriate) for each of the agents. One argument isselected and translated into behavior by an intent planner imple-mented in Flipper 2.0 [20]. The behavior of the agents is plannedby the ASAP realizer [19] and displayed by Unity3D embodiments[11]. The agents’ interactions will be recorded with screen capturesoftware. From the possible arguments one is selected randomlywhich means that multiple ‘runs’ create different interactions. Cru-cial in this setup is that any of the agents can be replaced by ahuman, and the other agents still will be able to debate as if therewere three agents. This feature will be used in the other corpusvariants, so that a participant can interact with the agents.2.2.3 HAA Corpus. A participant, who has the role of moderator,interacts with two virtual agents. The two agents take the role ofproponent and opponent. The participant will will out the samequestionnaires as the participants in the HHH recordings.2.2.4 HEA Corpus. A participant, who has the role of moderator,interacts with two virtual agents. One virtual agent has a ‘normal’virtual agent embodiment, whereas the other has a human confed-erate as embodiment. This resulting combination of human andmachine is called an echoborg [7]. The virtual agent part of theechoborg decides what it wants to say and do, when to do this,and communicates this covertly to the human embodiment over ahidden ear-piece. The human embodiment uses a technique calledspeech shadowing to repeat the words of the virtual agent verbatim.The participant is not briefed about the nature of the echoborg andis presumed to assume this to be a ’normal’ human counterpart inthe debate.After the interaction, the participant and the human playingthe echoborg embodiment fill out the same questionnaires as theparticipants in the HHH corpus. The answers by the echoborg offera unique insight into the ‘experiences’ that an agent has whileinteracting with a human.2.3 MeasuresWe selected several existing questionnaires that can provide us withinformation about how the participants experienced the interactionand the perceptions they have of their interlocutors. The first ques-tionnaire is the Thomas-Kilman conflict mode instrument that as-sesses the participant’s predisposition to behavior in a conflict situ-ation on two-dimensions (assertiveness and cooperation) [16]. Next,we present a questionnaire which includes the items from the God-speed Questionnaire series [4], the Game Experience Questionnaire[10], the Intrinsic Motivation Inventory [12], the Self-AssessmentThe Embodied Debate Corpus IVA2018 Workshop on Methodology, November 5, 2018, Sydney, AustraliaManikin [5] and it analyzes factors such as the Co-presence (Self-reported co-presence, Perceived other’s co-presence) [14], Rapport[18], Tele-presence [14], Social presence [14] and Trust [6]. Theseitems measure how the participant perceived her/himself duringthe interaction and how s/he perceived the others.3 DISCUSSIONWe will collect four variants of a corpus that will hopefully helpus in understanding the impact of different embodiments on thehuman perception of the information that is exchanged in a debateand the experience of the interaction. We hope that our efforts willlead to a corpus that is useful for the community. For this reason,we invite the community to give critical and constructive feedbackon our design, and come up with additional questions we should(not) ask the participants.REFERENCES[1] RenéAmalberti, Noëlle Carbonell, and Pierre Falzon. 1993. User representations ofcomputer systems in human-computer speech interaction. International Journalof Man-Machine Studies 38, 4 (1993), 547–566.[2] Jeremy N Bailenson, Nick Yee, Dan Merget, and Ralph Schroeder. 2006. Theeffect of behavioral realism and form realism of real-time avatar faces on verbaldisclosure, nonverbal disclosure, emotion recognition, and copresence in dyadicinteraction. Presence: Teleoperators and Virtual Environments 15, 4 (2006), 359–372.[3] Lawrence W Barsalou. 1982. Context-independent and context-dependent infor-mation in concepts. Memory & Cognition 10, 1 (1982), 82–93.[4] Christoph Bartneck, Dana Kulić, Elizabeth Croft, and Susana Zoghbi. 2009. Mea-surement instruments for the anthropomorphism, animacy, likeability, perceivedintelligence, and perceived safety of robots. International journal of social robotics1, 1 (2009), 71–81.[5] Margaret M Bradley and Peter J Lang. 1994. Measuring emotion: the self-assessment manikin and the semantic differential. Journal of behavior therapyand experimental psychiatry 25, 1 (1994), 49–59.[6] Ki-Taek Chun and John B Campbell. 1974. Dimensionality of the Rotter interper-sonal trust scale. Psychological Reports 35, 3 (1974), 1059–1070.[7] Kevin Corti and Alex Gillespie. 2015. Offscreen and in the chair next to your: con-versational agents speaking through actual human bodies. In Intelligent VirtualAgents. Springer, 405–417.[8] G Albyn Davis. 2007. Cognitive pragmatics of language disorders in adults. InSeminars in Speech and Language, Vol. 28. Copyright© 2007 by Thieme MedicalPublishers, Inc., 333 Seventh Avenue, New York, NY 10001, USA., 111–121.[9] Noah D Goodman and Michael C Frank. 2016. Pragmatic language interpretationas probabilistic inference. Trends in Cognitive Sciences 20, 11 (2016), 818–829.[10] WA IJsselsteijn, YAW De Kort, and Karolien Poels. 2013. The game experiencequestionnaire. (2013).[11] Jan Kolkmeier, Merijn Bruijnes, Dennis Reidsma, and Dirk Heylen. 2017. An asaprealizer-unity3d bridge for virtual and mixed reality applications. In InternationalConference on Intelligent Virtual Agents. Springer, 227–230.[12] Edward McAuley, Terry Duncan, and Vance V Tammen. 1989. Psychometricproperties of the Intrinsic Motivation Inventory in a competitive sport setting: Aconfirmatory factor analysis. Research quarterly for exercise and sport 60, 1 (1989),48–58.[13] Jonathan Mumm and Bilge Mutlu. 2011. Human-robot proxemics: physical andpsychological distancing in human-robot interaction. In Proceedings of the 6thinternational conference on Human-robot interaction. ACM, 331–338.[14] Kristine L Nowak and Frank Biocca. 2003. The effect of the agency and anthro-pomorphism on users’ sense of telepresence, copresence, and social presence invirtual environments. Presence: Teleoperators & Virtual Environments 12, 5 (2003),481–494.[15] Mark Snaith, Harm Op Den Akker, Tessa Beinema, Merijn Bruijnes, AlvaroFides-Valero, Gerwin Huizing, Reshmashree Kantharaju, Randy Klaassen, KostasKonsolakis, Dennis Reidsma, and Others. 2018. A demonstration of multi-partydialogue using virtual coaches: the first Council of Coaches demonstrator. InComputational Models of Argument (COMMA) (Frontiers in Artificial Intelligenceand Applications), Vol. 305. 473–474.[16] Kenneth W Thomas. 2008. Thomas-Kilmann Conflict Mode. TKI Profile andInterpretive Report (2008), 1–11.[17] Piercarlo Valdesolo and David DeSteno. 2006. Manipulations of emotional contextshape moral judgment. PSYCHOLOGICAL SCIENCE-CAMBRIDGE- 17, 6 (2006),476.[18] Michel Valstar, Jonathan Gratch, Björn Schuller, Fabien Ringeval, Denis Lalanne,Mercedes Torres Torres, Stefan Scherer, Giota Stratou, Roddy Cowie, and MajaPantic. 2016. Avec 2016: Depression, mood, and emotion recognition workshopand challenge. In Proceedings of the 6th International Workshop on Audio/VisualEmotion Challenge. ACM, 3–10.[19] Herwin Van Welbergen, Ramin Yaghoubzadeh, and Stefan Kopp. 2014. AsapReal-izer 2.0: The next steps in fluent behavior realization for ECAs. In InternationalConference on Intelligent Virtual Agents. Springer, 449–462.[20] Jelte vanWaterschoot, Merijn Bruijnes, Jan Flokstra, Dennis Reidsma, DanielDavison, Mariet Theune, and Dirk Heylen. 2018. Flipper 2.0 - A PragmaticDialogue Engine for Embodied Conversational Agents. In International Conferenceon Intelligent Virtual Agents. ACM.[21] Joshua Wainer, David J Feil-Seifer, Dylan A Shell, and Maja J Mataric. 2007. Em-bodiment and human-robot interaction: A task-based perspective. In Robot andHuman interactive Communication, 2007. RO-MAN 2007. The 16th IEEE Interna-tional Symposium on. IEEE, 872–877.[22] Ran Zhao, Alexandros Papangelis, and Justine Cassell. 2014. Towards a dyadiccomputational model of rapport management for human-virtual agent interaction.In International Conference on Intelligent Virtual Agents. Springer, 514–527.

The Embodied Debate Corpus – Proposed corpus recordings of multi-party multi-embodiment multi-measure debates

NARCIS 

https://ris.utwente.nl/ws/files/75231907/s5_embodied_debate_corpus_1_.pdf

The Embodied Debate Corpus – Proposed corpus recordings of multi-party multi-embodiment multi-measure debates

Abstract

Similar works

Full text

Available Versions

University of Twente Research Information

NARCIS