Affective reactions towards socially interactive agents and their computational modeling

Over the past 30 years, researchers have studied human reactions towards machines applying the Computers Are Social Actors paradigm, which contrasts reactions towards computers with reactions towards humans. The last 30 years have also seen improvements in technology that have led to tremendous changes in computer interfaces and the development of Socially Interactive Agents. This raises the question of how humans react to Socially Interactive Agents. To answer these questions, knowledge from several disciplines is required, which is why this interdisciplinary dissertation is positioned within psychology and computer science. It aims to investigate affective reactions to Socially Interactive Agents and how these can be modeled computationally. Therefore, after a general introduction and background, this thesis first provides an overview of the Socially Interactive Agent system used in this work. Second, it presents a study comparing a human and a virtual job interviewer, which shows that both interviewers induce shame in participants to the same extent. Thirdly, it reports on a study investigating obedience towards Socially Interactive Agents. The results indicate that participants obey human and virtual instructors in similar ways. Furthermore, both types of instructors evoke feelings of stress and shame to the same extent. Fourth, a stress management training using biofeedback with a Socially Interactive Agent is presented. The study shows that a virtual trainer can teach coping techniques for emotionally challenging social situations. Fifth, it introduces MARSSI, a computational model of user affect. The evaluation of the model shows that it is possible to relate sequences of social signals to affective reactions, taking into account emotion regulation processes. Finally, the Deep method is proposed as a starting point for deeper computational modeling of internal emotions. The method combines social signals, verbalized introspection information, context information, and theory-driven knowledge. An exemplary application to the emotion shame and a schematic dynamic Bayesian network for its modeling are illustrated. Overall, this thesis provides evidence that human reactions towards Socially Interactive Agents are very similar to those towards humans, and that it is possible to model these reactions computationally.In den letzten 30 Jahren haben Forschende menschliche Reaktionen auf Maschinen untersucht und dabei das “Computer sind soziale Akteure”-Paradigma genutzt, in dem Reaktionen auf Computer mit denen auf Menschen verglichen werden. In den letzten 30 Jahren hat sich ebenfalls die Technologie weiterentwickelt, was zu einer enormen Veränderung der Computerschnittstellen und der Entwicklung von sozial interaktiven Agenten geführt hat. Dies wirft Fragen zu menschlichen Reaktionen auf sozial interaktive Agenten auf. Um diese Fragen zu beantworten, ist Wissen aus mehreren Disziplinen erforderlich, weshalb diese interdisziplinäre Dissertation innerhalb der Psychologie und Informatik angesiedelt ist. Sie zielt darauf ab, affektive Reaktionen auf sozial interaktive Agenten zu untersuchen und zu erforschen, wie diese computational modelliert werden können. Nach einer allgemeinen Einführung in das Thema gibt diese Arbeit daher, erstens, einen Überblick über das Agentensystem, das in der Arbeit verwendet wird. Zweitens wird eine Studie vorgestellt, in der eine menschliche und eine virtuelle Jobinterviewerin miteinander verglichen werden, wobei sich zeigt, dass beide Interviewerinnen bei den Versuchsteilnehmenden Schamgefühle in gleichem Maße auslösen. Drittens wird eine Studie berichtet, in der Gehorsam gegenüber sozial interaktiven Agenten untersucht wird. Die Ergebnisse deuten darauf hin, dass Versuchsteilnehmende sowohl menschlichen als auch virtuellen Anleiterinnen ähnlich gehorchen. Darüber hinaus werden durch beide Instruktorinnen gleiche Maße von Stress und Scham hervorgerufen. Viertens wird ein Biofeedback-Stressmanagementtraining mit einer sozial interaktiven Agentin vorgestellt. Die Studie zeigt, dass die virtuelle Trainerin Techniken zur Bewältigung von emotional herausfordernden sozialen Situationen vermitteln kann. Fünftens wird MARSSI, ein computergestütztes Modell des Nutzeraffekts, vorgestellt. Die Evaluation des Modells zeigt, dass es möglich ist, Sequenzen von sozialen Signalen mit affektiven Reaktionen unter Berücksichtigung von Emotionsregulationsprozessen in Beziehung zu setzen. Als letztes wird die Deep-Methode als Ausgangspunkt für eine tiefer gehende computergestützte Modellierung von internen Emotionen vorgestellt. Die Methode kombiniert soziale Signale, verbalisierte Introspektion, Kontextinformationen und theoriegeleitetes Wissen. Eine beispielhafte Anwendung auf die Emotion Scham und ein schematisches dynamisches Bayes’sches Netz zu deren Modellierung werden dargestellt. Insgesamt liefert diese Arbeit Hinweise darauf, dass menschliche Reaktionen auf sozial interaktive Agenten den Reaktionen auf Menschen sehr ähnlich sind und dass es möglich ist diese menschlichen Reaktion computational zu modellieren.Deutsche Forschungsgesellschaf

ForDigitStress: A multi-modal stress dataset employing a digital job interview scenario

We present a multi-modal stress dataset that uses digital job interviews to induce stress. The dataset provides multi-modal data of 40 participants including audio, video (motion capturing, facial recognition, eye tracking) as well as physiological information (photoplethysmography, electrodermal activity). In addition to that, the dataset contains time-continuous annotations for stress and occurred emotions (e.g. shame, anger, anxiety, surprise). In order to establish a baseline, five different machine learning classifiers (Support Vector Machine, K-Nearest Neighbors, Random Forest, Long-Short-Term Memory Network) have been trained and evaluated on the proposed dataset for a binary stress classification task. The best-performing classifier achieved an accuracy of 88.3% and an F1-score of 87.5%

Transference of relationship qualities to a virtual world

This dissertation investigates to which extent internal working models of relationships and personality characteristics of participants are transferred to computer-generated agents in a virtual social environment (VSE). For this purpose a VSE was created that was populated with several autonomous agents which interacted with each other. Participants can control one of the the agents (the "protagonist"); one of the other agents is the spouse of the protagonist. Using this setup, a potential projection screen for the dynamics of the real-life relationship, as well as for generalized internal working models was created. Chapter 1 summarizes the research question, gives an overview of the development of the VSE called "Simoland", and provides a general summary and conclusion about the dissertation. Chapters 2 - 4 each are self-contained manuscripts. Chapter 2 („The challenge of constructing psychologically believable agents“) is a theoretical article which describes challenges and new possibilities in the implementation of psychological models into autonomous agents. Chapter 3 („Virtual social environments as a tool for psychological assessment: Dynamics of interaction with a virtual spouse“) describes the first empirical study conducted in Simoland. This study demonstrates how intrapersonal changes in interaction behavior can be investigated in VSEs. With a sample of 236 participants I could show that both the relationship satisfaction with the real life partner, and the intimacy motive of the participant had an influence on the behavior towards the virtual spouse. Hence, it could be shown that virtual behavior indeed is connected to conditions of the real world. While the first study investigated unrestricted behavior in an open world, in Chapter 4 („Transference of adult attachment dynamics to a virtual spouse“) specific, theoretically derived scenes were created which were supposed to activate the attachment system. With a sample of 422 participants we could demonstrate interindividual differences in attachment behavior as a reaction to one of three scenes: a separation, a conflict with the virtual spouse, and a threatening situation. Both attachment anxiety and attachment avoidance significantly and meaningfully correlated with the behavior in the VSE, the physical distance between protagonist and virtual spouse, and emotional ratings. To summarize, this dissertation for the first time shows that qualities of real-life relationships, as well as internal working models of attachment, are transferred to virtual agents. Thereby new ways are opened to investigate behavior in close relationships and representations of significant others through virtual scenarios.Diese Dissertation untersucht, inwieweit Beziehungsrepräsentationen und Persönlichkeitsmerkmale von Personen auf computergenerierte Agenten in einer virtuellen sozialen Umgebung (VSU) übertragen werden. Dazu wurde eine VSU erstellt, in der Teilnehmer eine virtuelle Figur (den „Protagonisten“) steuern können, welche wiederum mit anderen autonomen Agenten interagiert. Einer dieser anderen Agenten ist der virtuelle Partner/ die virtuelle Partnerin des Protagonisten, womit eine mögliche Projektionsfläche für die Beziehungsdynamik der realen Partnerschaft sowie für generalisierte interne Arbeitsmodelle von Beziehungen hergestellt wurde. Kapitel 1 fasst die Forschungsfrage zusammen und beschreibt den Entstehungsprozess der VSU namens „Simoland“, und gibt eine kurze Zusammenfassung der Gesamtergebnisse. Kapitel 2 – 4 stellen jeweils eigenständige Manuskripte dar. Kapitel 2 („The challenge of constructing psychologically believable agents“) ist ein theoretischer Artikel, der Herausforderungen und neue Wege beschreibt wie psychologische Modelle in autonome Agent implementiert werden können. Kapitel 3 („Virtual social environments as a tool for psychological assessment: Dynamics of interaction with a virtual spouse“) beschreibt die erste psychologische Untersuchung in Simoland, welche demonstriert wie intraindividuelle Entwicklung im Interaktionsverhalten untersucht werden kann. An einer Stichprobe von 236 Teilnehmern konnte gezeigt werden, dass sowohl die Beziehungszufriedenheit zum realen Partner, als auch das Intimitätsmotiv der Teilnehmer einen Einfluss auf das Verhalten in der VSU hatten, und somit das virtuelle Verhalten tatsächlich verknüpft ist mit der realen Welt. Während die erste Studie freies Verhalten untersucht hat, wurden in der zweiten Studie theoriegeleitet verschiedene Szenarien hergestellt, welche das Bindungssystem aktivieren sollen. (Kapitel 4: „Transference of adult attachment dynamics to a virtual spouse“). Dadurch konnten an 422 Teilnehmern interindividuelle Unterschiede im Bindungsverhalten als Reaktion auf folgende Szenen gezeigt werden: eine Trennungssituation, eine Konfliktsituation, sowie eine Bedrohungssituation. Die dimensionalen Bindungsstile (Bindungsvermeidung und Bindungsängstlichkeit) korrelierten sinnvoll und signifikant mit dem Verhalten in der VSU, der physikalischen Distanz zwischen Protagonist und Partner/in, sowie Emotionseinschätzungen. Zusammenfasst zeigt diese Dissertation zum ersten Mal, dass Qualitäten der realen Partnerschaft, sowie interne Arbeitsmodelle der Bindung auf virtuelle Agenten übertragen werden. Somit werden neue Wege eröffnet, Beziehungsverhalten und -repräsentationen in virtuellen Szenarien zu untersuchen

Semi-Automated & Collaborative Online Training Module For Improving Communication Skills

This paper presents a description and evaluation of the ROC Speak system, a platform that allows ubiquitous access to communication skills training. ROC Speak (available at rocspeak.com) enables anyone to go to a website, record a video, and receive feedback on smile intensity, body movement, volume modulation, filler word usage, unique word usage, word cloud of the spoken words, in addition to overall assessment and subjective comments by peers. Peer comments are automatically ranked and sorted for usefulness and sentiment (i.e., positive vs. negative). We evaluated the system with a diverse group of 56 online participants for a 10-day period. Participants submitted responses to career oriented prompts every other day. The participants were randomly split into two groups: 1) treatment - full feedback from the ROC Speak system; 2) control - written feedback from online peers. When judged by peers (p<.001) and independent raters (p<.05), participants from the treatment group demonstrated statistically significant improvement in overall speaking skills rating while the control group did not. Furthermore, in terms of speaking attributes, treatment group showed an improvement in friendliness (p<.001), vocal variety (p<.05) and articulation (p<.01)

Semi-Automated & Collaborative Online Training Module For Improving Communication Skills

This is the author accepted manuscript. The final version is available from ACM via the DOI in this recordThis paper presents a description and evaluation of the ROC Speak system, a platform that allows ubiquitous access to communication skills training. ROC Speak (available at rocspeak.com) enables anyone to go to a website, record a video, and receive feedback on smile intensity, body movement, volume modulation, filler word usage, unique word usage, word cloud of the spoken words, in addition to overall assessment and subjective comments by peers. Peer comments are automatically ranked and sorted for usefulness and sentiment (i.e., positive vs. negative). We evaluated the system with a diverse group of 56 online participants for a 10-day period. Participants submitted responses to career oriented prompts every other day. The participants were randomly split into two groups: 1) treatment - full feedback from the ROC Speak system; 2) control - written feedback from online peers. When judged by peers (p<.001) and independent raters (p<.05), participants from the treatment group demonstrated statistically significant improvement in overall speaking skills rating while the control group did not. Furthermore, in terms of speaking attributes, treatment group showed an improvement in friendliness (p<.001), vocal variety (p<.05) and articulation (p<.01)

Interactive Virtual Training: Implementation for Early Career Teachers to Practice Classroom Behavior Management

Teachers that are equipped with the skills to manage and prevent disruptive behaviors increase the potential for their students to achieve academically and socially. Student success increases when prevention strategies and effective classroom behavior management (CBM) are implemented in the classroom. However, teachers with less than 5 years of experience, early career teachers (ECTs), are ill equipped to handle disruptive students. ECTs describe disruptive behaviors as a major factor for stress given their limited training in CBM. As a result, disruptive behaviors are reported by ECTs as one of the main reasons for leaving the field. Virtual training environments (VTEs) combined with advances in virtual social agents can support the training of CBM. Although VTEs for teachers already exist, requirements to guide future research and development of similar training systems have not been defined. We propose a set of six requirements for VTEs for teachers. Our requirements were established from a survey of the literature and from iterative lifecycle activities to build our own VTE for teachers. We present different evaluations of our VTE using methodologies and metrics we developed to assess whether all requirements were met. Our VTE simulates interactions with virtual animated students based on real classroom situations to help ECTs practice their CBM. We enhanced our classroom simulator to further explore two aspects of our requirements: interaction devices and emotional virtual agents. Interactions devices were explored by comparing the effect of immersive technologies on users\u27 experience (UX) such as presence, co-presence, engagement and believability. We adapted our VTE originally built for desktop computer, to be compatible with two immersive VR platforms. Results show that our VTE generates high levels of UX across all VR platforms. Furthermore, we enhanced our virtual students to display emotions using facial expressions as current studies do not address whether emotional virtual agents provide the same level of UX across different VR platforms. We assessed the effects of VR platforms and display of emotions on UX. Our analysis shows that facial expressions have greater impact when using a desktop computer. We propose future work on immersive VTEs using emotional virtual agents