The recognition of acted interpersonal stance in police interrogations and the influence of actor proficiency

This paper reports on judgement studies regarding the perception of interpersonal stances taken by humans playing the role of a suspect in a police interrogation setting. Our project aims at building believable embodied conversational characters to play the role of suspects in a serious game for learning interrogation strategies. The main question we ask is: do human judges agree on the way they perceive the various aspects of stance taking, such as friendliness and dominance? Four types of stances were acted by eight amateur actors. Short recordings were shown in an online survey to subjects who were asked to describe them using a selection of a number of adjectives. Results of this annotation task are reported in this paper. We explain how we computed the inter-rater agreement with Krippendorff’s alpha statistics using a set theoretical distance metric. Results show that for some of the stance types observers agreed more than for others. Some actors are better than others, but validity (recognizing the intended stance) and inter-rater agreement do not always go hand in hand. We further investigate the effect the expertise of actors has on the perception of the stance that is acted. We compare the fragments from amateur actors to fragments from professional actors taken from popular TV-shows

Age-related differences in the evaluation of a virtual health agent’s appearance and embodiment in a health-related interaction: Experimental lab study

Straßmann C, Krämer NC, Buschmeier H, Kopp S. Age-related differences in the evaluation of a virtual health agent’s appearance and embodiment in a health-related interaction: Experimental lab study. Journal of Medical Internet Research. 2020;22(4): e13726.**Background:** Assistive technologies have become more important owing to the aging population, especially when they foster healthy behaviors. Because of their natural interface, virtual agents are promising assistants for people in need of support. To engage people during an interaction with these technologies, such assistants need to match the users´ needs and preferences, especially with regard to social outcomes. **Objective:** Prior research has already determined the importance of an agent’s appearance in a human-agent interaction. As seniors can particularly benefit from the use of virtual agents to maintain their autonomy, it is important to investigate their special needs. However, there are almost no studies focusing on age-related differences with regard to appearance effects. **Methods:** A 2×4 between-subjects design was used to investigate the age-related differences of appearance effects in a human-agent interaction. In this study, 46 seniors and 84 students interacted in a health scenario with a virtual agent, whose appearance varied (cartoon-stylized humanoid agent, cartoon-stylized machine-like agent, more realistic humanoid agent, and nonembodied agent [voice only]). After the interaction, participants reported on the evaluation of the agent, usage intention, perceived presence of the agent, bonding toward the agent, and overall evaluation of the interaction. **Results:** The findings suggested that seniors evaluated the agent more positively (liked the agent more and evaluated it as more realistic, attractive, and sociable) and showed more bonding toward the agent regardless of the appearance than did students. In addition, interaction effects were found. Seniors reported the highest usage intention for the cartoon-stylized humanoid agent, whereas students reported the lowest usage intention for this agent. The same pattern was found for participant bonding with the agent. Seniors showed more bonding when interacting with the cartoon-stylized humanoid agent or voice only agent, whereas students showed the least bonding when interacting with the cartoon-stylized humanoid agent. **Conclusions:** In health-related interactions, target group–related differences exist with regard to a virtual assistant’s appearance. When elderly individuals are the target group, a humanoid virtual assistant might trigger specific social responses and be evaluated more positively at least in short-term interactions

Dominant and submissive nonverbal behavior of virtual agents and its effects on evaluation and negotiation outcome in different age groups

Rosenthal-von der Pütten AM, Straßmann C, Yaghoubzadeh R, Kopp S, Krämer NC. Dominant and submissive nonverbal behavior of virtual agents and its effects on evaluation and negotiation outcome in different age groups. Computers in Human Behavior. 2019;90:397-409

Relational agents : effecting change through human-computer relationships

Thesis (Ph. D.)--Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2003.Includes bibliographical references (p. 205-219).What kinds of social relationships can people have with computers? Are there activities that computers can engage in that actively draw people into relationships with them? What are the potential benefits to the people who participate in these human-computer relationships? To address these questions this work introduces a theory of Relational Agents, which are computational artifacts designed to build and maintain long-term, social-emotional relationships with their users. These can be purely software humanoid animated agents--as developed in this work--but they can also be non-humanoid or embodied in various physical forms, from robots, to pets, to jewelry, clothing, hand-helds, and other interactive devices. Central to the notion of relationship is that it is a persistent construct, spanning multiple interactions; thus, Relational Agents are explicitly designed to remember past history and manage future expectations in their interactions with users. Finally, relationships are fundamentally social and emotional, and detailed knowledge of human social psychology--with a particular emphasis on the role of affect--must be incorporated into these agents if they are to effectively leverage the mechanisms of human social cognition in order to build relationships in the most natural manner possible. People build relationships primarily through the use of language, and primarily within the context of face-to-face conversation. Embodied Conversational Agents--anthropomorphic computer characters that emulate the experience of face-to-face conversation--thus provide the substrate for this work, and so the relational activities provided by the theory will primarily be specific types of verbal and nonverbal conversational behaviors used by people to negotiate and maintain relationships.(cont.) This work also provides an analysis of the types of applications in which having a human-computer relationship is advantageous to the human participant. In addition to applications in which the relationship is an end in itself (e.g., in entertainment systems), human-computer relationships are important in tasks in which the human is attempting to undergo some change in behavior or cognitive or emotional state. One such application is explored here: a system for assisting the user through a month-long health behavior change program in the area of exercise adoption. This application involves the research, design and implementation of relational agents as well as empirical evaluation of their ability to build relationships and effect change over a series of interactions with users.by Timothy Wallace Bickmore.Ph.D

Modelling cultural dimensions and social relationships to create cultural synthetic characters

The work presented in this thesis investigates studies and theories of culture, social power and the relationship between culture and emotion studied by psychologists and anthropology. We operationalised a Cultural Dimension model, proposed by Hofstede, and Social Power and integrated them into an already existing architecture for autonomous agents called “FAtiMA”. The purpose of the adapted system is to generate culturally-specific behaviour in character interaction which is recognisably different to users. Two different experiments, with human participants, were conducted to investigate the perceived differences between two different groups of characters: with and without cultural parameters. The main result shows that users do recognise the differences in character behaviour between the two experimental cases, which demonstrates that our model is able to create culturally-specific synthetic characters

Virgil Ortiz: American Indian Artist, Representational Trickster, and Identity Shapeshifter

This study opens the door for a re-thinking of how discourse shapes American Indian representation and identity. As such, contemporary American Indian artist, Virgil Ortiz, his art, and the discourse surrounding both art and artist are examined to reveal the strategies and tactics employed in his constitution of a politics of representation that broaden the spectrum of considerations of American Indian identity. Critical invention is the orientation through which two methodological approaches are intertextually applied. A critical rhetorical approach is employed to analyze both the vernacular discourse produced by Ortiz and the dominant discourse constructed by the dominant culture. Sorrells (1999) theoretical and methodological approach to reading intercultural imagery is also applied to conduct a visual analysis of Ortiz’s art. To contextually frame an understanding of Ortiz and his work, a literature review and a historical chapter are included. The literature review details the linking of American Indian cultural identity, collective identity, and cultural sovereignty to the production of American Indian art; examines art and American Indian identity; and investigates art and the production of a politics of representation. The historical chapter reveals the poetics and politics of American Indian discursive constructions by both the dominant culture and American Indians. The theme of sadomasochistic dominance and submission (SMDS) is explored in Ortiz’s art to understand how it communicatively operates through vernacular discourse. Ortiz’s marketing through branding and personal branding is analyzed to understand how Ortiz both subverts and complies with the dominant culture’s current entrenchment in commodity capitalism and in stale American Indian representations. The measure of representational sovereignty that Ortiz asserts is evident in the mediums and the media in which he participates. This study reveals that Ortiz produces a counter discourse that disturbs hegemonic notions of American Indians; promotes more prismatic considerations of American Indian identity, rather than one-dimensional stale stereotypes or two-dimensional restraining binaries; and offers alternative American Indian archetypes for consideration. Ortiz draws from the mainstream to the margins and the surface to the subterranean to create a politics of representation that promotes an understanding of multi-faceted, multi-dimensional, and multiple American Indian identity articulations, which move American Indians closer to signification self-sovereignty

Trust in interdependent and task-oriented human-computer cooperation

Kulms P. Trust in interdependent and task-oriented human-computer cooperation. Bielefeld: Universität Bielefeld; 2018.This thesis presents a new paradigm for the modeling of cooperative human–computer interaction in order to evaluate the antecedents, formation, and regulation of human–computer trust. Human–computer trust is the degree to which human users trust computers to help them achieve their goals, and functions as powerful psychological variable that governs user behavior. The modeling framework presented in this thesis aims to extend predominant methods for the study of trust and cooperation by building on competent problemsolving and equal goal contributions by users and computers. Specifically, the framework permits users to participate in interactive and interdependent decision-making games with autonomous computer agents. The main task is to solve a two-dimensional puzzle, similar to the popular game Tetris. The games derived from this framework include cooperative interaction factors known from interpersonal cooperation: the duality of competence and selfishness, anthropomorphism, task advice, and social blame. One validation study (68 participants) and four experiments (318 participants) investigate how these cooperative interaction factors influence human–computer trust. In particular, the results show how trust in computers is mediated by warmth as universal dimension of social cognition, how anthropomorphism of computers influences trust formation over time, and how expressive anthropomorphic cues can be used to regulate trust. We explain how these findings can be applied to design trustworthy computer agents for successful cooperation

Perception of Facial Emotional Expressions and Touch in Virtual Face-to-Face Interaction

Emotional expressions as manifested in facial movements, voice, and touch are a crucial part of face-to-face interaction. The majority of existing neuroscientific research on emotional expressions concerns the perception of unimodal emotional cues, such as facial emotional expressions. In natural face-to-face interaction, however, emotions are often expressed as compounds of facial, tactile, prosodic, and postural cues. How the brain processes such multimodal emotional information remains poorly understood. The aim of the current dissertation is to investigate how emotional expressions conveyed consecutively via face and touch are integrated in perceptual processing and decision-making. Four studies were conducted to measure event-related brain potentials (ERPs) and autonomic nervous system responses to simulated touches and facial emotional expression stimuli. The first two studies used virtual reality to investigate how a virtual agent’s facial emotional expressions influenced the way the agent’s subsequent touch was perceived (Study I) and whether the receiver’s individual characteristics influenced this visuo-tactile affective modulation (Study II). Touch perception was measured using self-reports, somatosensory-evoked potentials (SEPs), and cardiac orienting responses (ORs), and the individual characteristics were indexed by behavioural inhibition system sensitivity (BIS) and gender. Study III investigated whether receiving a touch influenced the processing of a subsequent emotional face picture presented on the computer screen. Here, face-evoked ERPs, ORs, and facial electromyography were measured. Finally, the Study IV examined whether a virtual agent’s touch and emotional facial expressions influence receivers’ decision-making and offer-related ORs in an economic decision-making game. Additionally, the study examined whether the receivers’ behavioural inhibition/approach system (BAS/BIS) sensitivities and sensitivity to unfair treatment moderated persuasiveness of nonverbal cues. Study I revealed that happy, angry, and sad facial expressions resulted in amplified SEPs around 20–50 ms after touch onset, whereas in later latencies (250–650 ms), the angry facial expression amplified and the happy expression decreased the SEP amplitudes. In Study II, men with high BIS were found to perceive touch from male agents as especially intense if accompanied by happy, angry, of fearful facial expressions, and they showed pronounced cardiac OR to all the touches. Study III demonstrated that receiving a computer-generated touch did not modulate emotional face processing in any of the measured indices. Finally, in Study IV, people were found to accept unfair offers more often if the agent smiled or touched them before making the offer. The touch had a stronger persuasive influence in people with low sensitivity to unfairness and low approach tendency, whereas the effect of facial expressions was moderated by BIS. Altogether, the findings of the dissertation reveal that a sender’s facial emotional expressions modulate subsequent touch perception at a very early stage and that the modulation is based on different emotional information in different temporal stages. In addition, the findings suggest that motivational tendencies and gender influence the manner in which people perceive a sender’s multimodal emotional expressions and make decisions thereafter. These findings are valuable for basic research, but their implications extend also to the development of novel clinical interventions and social virtual reality applications.Kasvonilmeiden, asentojen, äänenpainojen ja kosketuksen kautta välittyvät tunneilmaisut ovat keskeinen osa sosiaalista vuorovaikutusta. Suuri osa olemassa olevasta tunneilmaisujen neurotieteellisestä tutkimuksesta koskee unimodaalisten tunneilmaisuärsykkeiden, kuten kasvonilmeiden prosessointia. Kasvokkaisessa vuorovaikutuksessa tunneilmaisut esiintyvät kuitenkin usein kasvonilmeiden, äänenpainojen, asentojen ja kosketuksen sarjamaisina yhdistelminä. Aivojen tavasta prosessoida tällaisia moniaistisia tunneilmaisuja tiedetään vielä vähän. Tässä väitöskirjatutkimuksessa tutkittiin, miten peräkkäin esitetyt kasvonilmeet ja kosketus integroituvat toisiinsa aistiprosessoinnin tasolla ja vaikuttavat vastaanottajan sosiaaliseen päätöksentekoon. Väitöskirja koostuu neljästä tutkimuksesta, joissa mitattiin aivosähkökäyrän herätevasteita ja autonomisen hermoston reaktioita kosketukseen ja kasvonilmeisiin. Kahdessa ensimmäisessä tutkimuksessa selvitettiin, kuinka virtuaalitodellisuuteen heijastetun ihmishahmon kasvonilmeet (iloinen, suuttunut, pelokas, surullinen ja neutraali) vaikuttavat hahmon lähettämän koneellisesti tuotetun kosketuksen havaitsemiseen (tutkimus I) ja kuinka vastaanottajan välttämistaipumus ja sukupuoli säätelevät kasvonilmeiden vaikutuksia kosketuksen havaitsemiseen (tutkimus II). Kosketuksen havaitsemista mitattiin itseraportoinnin, somatosensoristen herätevasteiden ja sydämen orientaatiorefleksin avulla. Tutkimuksessa III selvitettiin, vaikuttaako koneellisesti tuotettu kosketus sitä seuraavan emotionaalisen kasvokuvan prosessointiin. Kasvonilmeiden prosessointia mitattiin kasvoärsykkeiden aikaansaamien näköherätevasteiden, kasvolihasaktivaation ja sydämen orientaatiorefleksin avulla. Tutkimuksessa IV tarkasteltiin virtuaalihahmon kasvonilmeiden ja kosketuksen yhteisvaikutuksia hahmon taloudellisiin tarjouksiin suostumiseen ja tarjousten aiheuttamaan sydämen orientaatiorefleksiin. Samalla tarkasteltiin, mikäli vastaanottajan lähestymis- ja välttämistaipumus sekä herkkyys epäreilulle kohtelulle säätelevät tunneilmaisuiden suostuttelevaa vaikutusta. Tutkimuksen I tulokset osoittivat, että kosketukset, joita edelsi iloinen, suuttunut tai surullinen ilme johtivat voimakkaampiin varhaisiin herätevasteisiin 20 - 50 millisekuntia kosketuksen esittämisestä. Vastaavasti myöhemmät vasteet (250-650 ms) olivat korostuneempia, jos hahmolla oli suuttunut ilme, ja latistuneita jos hahmo hymyili. Tutkimuksessa II havaittiin, että erityisesti miehillä, joilla oli korkea välttämistaipumus, hahmon kosketus sai aikaan voimakkaan sydämen orientaatiorefleksin. He myös kokivat kosketuksen voimakkaana, kun hahmo ilmaisi kasvoillaan suuttumusta, pelkoa tai iloa koskettaessaan. Vastoin oletuksia, tutkimuksessa III koneellisesti tuotettu kosketus ei vaikuttanut kasvonilmeärsykkeen prosessointiin millään mitatuista vasteista. Tutkimuksessa IV havaittiin, että virtuaalihahmon kosketus ja iloinen/neutraali kasvonilme lisäsivät suostuvaisuutta hahmon esittämiin epäreiluihin taloudellisiin tarjouksiin. Kosketus johti suurempaan suostuvaisuuteen henkilöillä, joilla oli alhainen herkkyys epäreiluudelle ja alhainen lähestymistaipumus. Kasvonilmeiden vaikutusta suostuvaisuuteen sääteli vastaanottajan välttämistaipumus. Yhteenvetona tulokset osoittavat, että vuorovaikutuksessa ilmaistut kasvonilmeet ja kosketus integroituvat toisiinsa hyvin varhain ja että integraatioprosessi on monivaiheinen. Myös persoonallisuuspiirteillä ja sukupuolella näyttää olevan merkittävä rooli multimodaalisten tunneilmaisuiden havaitsemisessa ja vaikutuksessa päätöksentekoon