Affect and Social Processes in Online Communication- Experiments with an Affective Dialog System

Abstract—This paper presents an integrated view on a series of experiments conducted with an affective dialog system, applied as a tool in studies of emotions and social processes in online communication. The different realizations of the system are evaluated in three experimental setups in order to verify effects of affective profiles, as well as of fine-grained communication scenarios on users’ expressions of affective states, experienced emotional changes, and interaction patterns. Results demonstrate that the system applied in virtual reality settings matches a Wizard-of-Oz in terms of chatting enjoyment, dialog coherence and realism. Variants of the system’s affective profile significantly influence the rating of chatting enjoyment and an emotional connection. Self-reported emotional changes experienced by participants during an interaction with the system are in line with the type of applied profile. Analysis of interaction patterns, i.e., usage of particular dialog act classes, word categories, and textual expressions of affective states for different scenarios, demonstrates that a communication scenario for social sharing of emotions was successfully established. The experimental evidence provides valuable input for applications of affective dialog systems and strengthens them as valid tools for studying affect and social aspects in online communication. Index Terms—Affective dialog system, human-computer interaction, affect sensing and analysis, structuring affective interactions.

How to improve TTS systems for emotional expressivity

Several experiments have been carried out that revealed weaknesses of the current Text-To-Speech (TTS) systems in their emotional expressivity. Although some TTS systems allow XML-based representations of prosodic and/or phonetic variables, few publications considered, as a pre-processing stage, the use of intelligent text processing to detect affective information that can be used to tailor the parameters needed for emotional expressivity. This paper describes a technique for an automatic prosodic parameterization based on affective clues. This technique recognizes the affective information conveyed in a text and, accordingly to its emotional connotation, assigns appropriate pitch accents and other prosodic parameters by XML-tagging. This pre-processing assists the TTS system to generate synthesized speech that contains emotional clues. The experimental results are encouraging and suggest the possibility of suitable emotional expressivity in speech synthesis

A virtual diary companion

Chatbots and embodied conversational agents show turn based conversation behaviour. In current research we almost always assume that each utterance of a human conversational partner should be followed by an intelligent and/or empathetic reaction of chatbot or embodied agent. They are assumed to be alert, trying to please the user. There are other applications which have not yet received much attention and which require a more patient or relaxed attitude, waiting for the right moment to provide feedback to the human partner. Being able and willing to listen is one of the conditions for being successful. In this paper we have some observations on listening behaviour research and introduce one of our applications, the virtual diary companion

Exploring the Affective Loop

Research in psychology and neurology shows that both body and mind are involved when experiencing emotions (Damasio 1994, Davidson et al. 2003). People are also very physical when they try to communicate their emotions. Somewhere in between beings consciously and unconsciously aware of it ourselves, we produce both verbal and physical signs to make other people understand how we feel. Simultaneously, this production of signs involves us in a stronger personal experience of the emotions we express. Emotions are also communicated in the digital world, but there is little focus on users' personal as well as physical experience of emotions in the available digital media. In order to explore whether and how we can expand existing media, we have designed, implemented and evaluated /eMoto/, a mobile service for sending affective messages to others. With eMoto, we explicitly aim to address both cognitive and physical experiences of human emotions. Through combining affective gestures for input with affective expressions that make use of colors, shapes and animations for the background of messages, the interaction "pulls" the user into an /affective loop/. In this thesis we define what we mean by affective loop and present a user-centered design approach expressed through four design principles inspired by previous work within Human Computer Interaction (HCI) but adjusted to our purposes; /embodiment/ (Dourish 2001) as a means to address how people communicate emotions in real life, /flow/ (Csikszentmihalyi 1990) to reach a state of involvement that goes further than the current context, /ambiguity/ of the designed expressions (Gaver et al. 2003) to allow for open-ended interpretation by the end-users instead of simplistic, one-emotion one-expression pairs and /natural but designed expressions/ to address people's natural couplings between cognitively and physically experienced emotions. We also present results from an end-user study of eMoto that indicates that subjects got both physically and emotionally involved in the interaction and that the designed "openness" and ambiguity of the expressions, was appreciated and understood by our subjects. Through the user study, we identified four potential design problems that have to be tackled in order to achieve an affective loop effect; the extent to which users' /feel in control/ of the interaction, /harmony and coherence/ between cognitive and physical expressions/,/ /timing/ of expressions and feedback in a communicational setting, and effects of users' /personality/ on their emotional expressions and experiences of the interaction

MINDtouch embodied ephemeral transference: Mobile media performance research

This is the post-print version of the final published article that is available from the link below. Copyright @ Intellect Ltd 2011.The aim of the author's media art research has been to uncover any new understandings of the sensations of liveness and presence that may emerge in participatory networked performance, using mobile phones and physiological wearable devices. To practically investigate these concepts, a mobile media performance series was created, called MINDtouch. The MINDtouch project proposed that the mobile videophone become a new way to communicate non-verbally, visually and sensually across space. It explored notions of ephemeral transference, distance collaboration and participant as performer to study presence and liveness emerging from the use of wireless mobile technologies within real-time, mobile performance contexts. Through participation by in-person and remote interactors, creating mobile video-streamed mixes, the project interweaves and embodies a daisy chain of technologies through the network space. As part of a practice-based Ph.D. research conducted at the SMARTlab Digital Media Institute at the University of East London, MINDtouch has been under the direction of Professor Lizbeth Goodman and sponsored by BBC R&D. The aim of this article is to discuss the project research, conducted and recently completed for submission, in terms of the technical and aesthetic developments from 2008 to present, as well as the final phase of staging the events from July 2009 to February 2010. This piece builds on the article (Baker 2008) which focused on the outcomes of phase 1 of the research project and initial developments in phase 2. The outcomes from phase 2 and 3 of the project are discussed in this article

Machine Understanding of Human Behavior

A widely accepted prediction is that computing will move to the background, weaving itself into the fabric of our everyday living spaces and projecting the human user into the foreground. If this prediction is to come true, then next generation computing, which we will call human computing, should be about anticipatory user interfaces that should be human-centered, built for humans based on human models. They should transcend the traditional keyboard and mouse to include natural, human-like interactive functions including understanding and emulating certain human behaviors such as affective and social signaling. This article discusses a number of components of human behavior, how they might be integrated into computers, and how far we are from realizing the front end of human computing, that is, how far are we from enabling computers to understand human behavior

Enkinaesthetic polyphony: the underpinning for first-order languaging

We contest two claims: (1) that language, understood as the processing of abstract symbolic forms, is an instrument of cognition and rational thought, and (2) that conventional notions of turn-taking, exchange structure, and move analysis, are satisfactory as a basis for theorizing communication between living, feeling agents. We offer an enkinaesthetic theory describing the reciprocal affective neuro-muscular dynamical flows and tensions of co- agential dialogical sense-making relations. This “enkinaesthetic dialogue” is characterised by a preconceptual experientially recursive temporal dynamics forming the deep extended melodies of relationships in time. An understanding of how those relationships work, when we understand and are ourselves understood, when communication falters and conflict arises, will depend on a grasp of our enkinaesthetic intersubjectivity

The Neurocognitive Process of Digital Radicalization: A Theoretical Model and Analytical Framework

Recent studies suggest that empathy induced by narrative messages can effectively facilitate persuasion and reduce psychological reactance. Although limited, emerging research on the etiology of radical political behavior has begun to explore the role of narratives in shaping an individual’s beliefs, attitudes, and intentions that culminate in radicalization. The existing studies focus exclusively on the influence of narrative persuasion on an individual, but they overlook the necessity of empathy and that in the absence of empathy, persuasion is not salient. We argue that terrorist organizations are strategic in cultivating empathetic-persuasive messages using audiovisual materials, and disseminating their message within the digital medium. Therefore, in this paper we propose a theoretical model and analytical framework capable of helping us better understand the neurocognitive process of digital radicalization