Cross-Corpora Study of Smiles and Laughter Mimicry in Dyadic Interactions

In this paper, we present preliminary results of our ongoing work on cross-corpora analyses of smiles and laughter mimicry. For this, instead of recording new data, we leverage the ones produced and available. We analyze smiles and laughs mimicry in three different datasets and show results similar to our previous work.The data used here can be accessed at: https://doi.org/10.5281/zenodo.3820510

Cross-Corpora Study of Smiles and Laughter Mimicry in Dyadic Interactions

In this paper, we present preliminary results of our ongoing work on cross-corpora analyses of smiles and laughter mimicry. For this, instead of recording new data, we leverage the ones produced and available. We analyze smiles and laughs mimicry in three different datasets and show results similar to our previous work.The data used here can be accessed at: https://doi.org/10.5281/zenodo.3820510

A Quantum-like Multimodal Network Framework for Modeling Interaction Dynamics in Multiparty Conversational Sentiment Analysis

Sentiment analysis in conversations is an emerging yet challenging artificial intelligence (AI) task. It aims to discover the affective states and emotional changes of speakers involved in a conversation on the basis of their opinions, which are carried by different modalities of information (e.g., a video associated with a transcript). There exists a wealth of intra- and inter-utterance interaction information that affects the emotions of speakers in a complex and dynamic way. How to accurately and comprehensively model complicated interactions is the key problem of the field. To fill this gap, in this paper, we propose a novel and comprehensive framework for multimodal sentiment analysis in conversations, called a quantum-like multimodal network (QMN), which leverages the mathematical formalism of quantum theory (QT) and a long short-term memory (LSTM) network. Specifically, the QMN framework consists of a multimodal decision fusion approach inspired by quantum interference theory to capture the interactions within each utterance (i.e., the correlations between different modalities) and a strong-weak influence model inspired by quantum measurement theory to model the interactions between adjacent utterances (i.e., how one speaker influences another). Extensive experiments are conducted on two widely used conversational sentiment datasets: the MELD and IEMOCAP datasets. The experimental results show that our approach significantly outperforms a wide range of baselines and state-of-the-art models

Quantum-Inspired Interactive Networks for Conversational Sentiment Analysis.

Conversational sentiment analysis is an emerging, yet challenging Artificial Intelligence (AI) subtask. It aims to discover the affective state of each participant in a conversation. There exists a wealth of interaction information that affects the sentiment of speakers. However, the existing sentiment analysis approaches are insufficient in dealing with this task due to ignoring the interactions and dependency relationships between utterances. In this paper, we aim to address this issue by modeling intrautterance and inter-utterance interaction dynamics. We propose an approach called quantum-inspired interactive networks (QIN), which leverages the mathematical formalism of quantum theory (QT) and the long short term memory (LSTM) network, to learn such interaction dynamics. Specifically, a density matrix based convolutional neural network (DM-CNN) is proposed to capture the interactions within each utterance (i.e., the correlations between words), and a strong-weak influence model inspired by quantum measurement theory is developed to learn the interactions between adjacent utterances (i.e., how one speaker influences another). Extensive experiments are conducted on the MELD and IEMOCAP datasets. The experimental results demonstrate the effectiveness of the QIN model

A Little Bird Told Me So...:the Emotional, Attributional, Relational And Team-Level Outcomes Of Engaging In Gossip

In this paper, I examine the consequences, both positive and negative, of initiating and participating in gossip in work-related contexts. While a commonly held perspective is that gossip is harmful in that it hurts relational interactions by encouraging coalition-building and engendering divisiveness, an alternative hypothesis is that gossip\u27s emotional attributes, can also help to foster stronger relationships and help individuals navigate complex environments. Specifically, I explore the influence of gossip at multiple levels of analysis: individual, dyadic and group. In Study 1, a laboratory experiment that looks at the short-term benefits of engaging in gossip (versus two control conditions, self-disclosure and task discussion), I find that individuals who engage in gossip experience higher positive emotions, energy and motivation but lower levels of state self-esteem. These gossiping dyads also experience dyadic benefits of relationship closeness and cooperation. Study 2 explored both the reputational and team-level outcomes of gossip. This study showed that team members who engaged in gossip were seen as being less trustworthy. Furthermore, gossip centrality had an inverted U-shaped curvilinear relationship with perceptions of competence. Study 2 showed that gossip about team members negatively influenced team outcomes such as psychological safety, cooperation and viability and increased team-level perceptions of politics while gossip about individuals outside the team has a positive effect on these outcomes, enhancing levels of team cooperation and decreasing perceptions of politics at the team-level. More detailed mediation analyses showed that team process variables, psychological safety and perceptions of politics measured halfway through the course of the team, mediated the negative relationship between intra-team gossip density and team cooperation and team viability measured at the end of the team\u27s lifecycle. In terms of the relationship between extra-team gossip density and team cooperation, it was mediated by decreased team perceptions of politics. This research contributes to the emerging field of inquiry on gossip by providing a comprehensive model of the consequences of gossip at three different levels of analysis as well as a strong empirical test of the effect of gossip on organizationally-relevant outcomes

An Actor-Centric Approach to Facial Animation Control by Neural Networks For Non-Player Characters in Video Games

Game developers increasingly consider the degree to which character animation emulates facial expressions found in cinema. Employing animators and actors to produce cinematic facial animation by mixing motion capture and hand-crafted animation is labor intensive and therefore expensive. Emotion corpora and neural network controllers have shown promise toward developing autonomous animation that does not rely on motion capture. Previous research and practice in disciplines of Computer Science, Psychology and the Performing Arts have provided frameworks on which to build a workflow toward creating an emotion AI system that can animate the facial mesh of a 3d non-player character deploying a combination of related theories and methods. However, past investigations and their resulting production methods largely ignore the emotion generation systems that have evolved in the performing arts for more than a century. We find very little research that embraces the intellectual process of trained actors as complex collaborators from which to understand and model the training of a neural network for character animation. This investigation demonstrates a workflow design that integrates knowledge from the performing arts and the affective branches of the social and biological sciences. Our workflow begins at the stage of developing and annotating a fictional scenario with actors, to producing a video emotion corpus, to designing training and validating a neural network, to analyzing the emotion data annotation of the corpus and neural network, and finally to determining resemblant behavior of its autonomous animation control of a 3d character facial mesh. The resulting workflow includes a method for the development of a neural network architecture whose initial efficacy as a facial emotion expression simulator has been tested and validated as substantially resemblant to the character behavior developed by a human actor

Storytelling, self, and affiliation : conversation analysis of interactions between neurotypical participants and participants with Asperger syndrome

https://helda.helsinki.fi/handle/10138/341931This dissertation examines interpersonal affiliation and the reciprocal protecting of selves and their worthiness, i.e., face-work, during conversational storytelling and story reception. The method utilized is Conversation Analysis (CA), which is a qualitative method for studying audio and video recorded interactions. CA’s purpose is unravelling recurring interactional practices through which social actions are constructed. The dataset analyzed in the study consists of ten video recordings of 45- to 60-minute dyadic conversations, where one participant has been diagnosed with Asperger syndrome (AS) and the other participant is neurotypical (NT), and nine video recordings, in which both participants are neurotypical. The participants were adult males, aged between 18-40 years. The participants received instructions to talk about happy events and losses in their lives in a freely chosen way. Storytelling and story reception practices have previously gained considerable attention in CA, as have the interactional practices of participants diagnosed with autism spectrum disorder or AS. The investigation in the current study, however, involves a unique combination of these elements. Studying AS–NT interactions can increase our understanding of the underlying structures and norms of conversational storytelling and help reveal the taken for granted aspects of ‘commonsense’ that usually go unquestioned. The aim for the study is thus twofold: to investigate the face-work, storytelling and story reception practices of individuals diagnosed with AS, and to increase our understanding of these phenomena in general. More specifically, the focus of the study is on the displays of (non-)affiliation and on the differing degrees of affiliation conveyed by different interactional practices. Since the study compares the interactional practices of NT and AS participants in the same interactional setting, it inherently involves categorizing the participants. CA has generally followed the policy of ‘ethnomethodological indifference’ toward the participants’ identities and predominantly focused on how participants themselves categorize each other in their talk. However, in this study the empirical observations of the participants’ talk have been interpreted in the light of different contextual factors, which include the participants’ neurological statuses. The dissertation consists of four research articles. The first concerns stories in which the AS participants are in the spontaneously assumed role of the recipient. The results are discussed in relation to earlier CA findings on story reception and affiliation in typical interaction, as well as on AS and its specific interactional features. The second article compares the affiliation and topicality of the questions that AS and NT story recipients ask after their co-participants’ tellings. The article shows that the affiliative import of story-responsive questions can only really be seen in retrospect, because the questioner can cast their action in an affiliative or non-affiliative light in subsequent turns. The third article investigates how story recipients manage to display the right level of access to the events the teller describes in order to achieve affiliation. The article describes two main ways to accomplish this in a responsive utterance: fine-tuning the strength of one’s access claim and adjusting the degree of generalization. The fourth article explores the differences in the ways in which the AS and NT participants recognize and manage face threats in interaction, in their role as both storytellers and story recipients. The study shows how affiliation and the establishment of empathic communion between participants has several intersecting levels, as refraining from endorsing the affective stance displayed in the co-participant’s telling can sometimes be a prosocial move that protects the selves of the participants. In addition, the study suggests that the difference between the NT and AS participants lies not in the amount of affiliation per se but in the subtle use of conversational practices to manage their non-affiliation. The study proposes that future CA studies of asymmetric interactions may consider more theory-laden approaches in addition to the traditional ‘ethnomethodologically indifferent’ perspectives

The APRACE: Six Components of Social Interactions

In the pursuit to advance research on situations of social interactions, this dissertation comprises the creation of a taxonomy of social interactions and its application to two research issues, both concerning subjective well-being in social interactions and age differences. For the development of the hierarchical taxonomy of social interactions, an integrative method of combining a bottom-up data-based approach and a top-down approach integrating existing empirical and theoretical literature was employed. The resulting taxonomy consists of the six components Actor, Partner, Relation, Activities, Context, and Evaluation (APRACE), each divided into features at three lower hierarchical levels. In order to confirm the comprehensiveness and generalizability, the APRACE taxonomy was implemented in another dataset. In addition to this theoretical contribution to social interaction research, the APRACE is intended to be a flexible measurement tool to assess social interactions. In the first application study with the APRACE as a measurement tool, the features of the social interactions were transformed in closed questions to investigate age differences in satisfying social interactions. Evidence was found that older adults are more likely to experience social interactions associated with high levels of subjective well-being than are younger adults. Emotion and control-related features were critical, which indicates that older adults employ early-stage emotion regulation strategies, such as situation selection. Almost no evidence was found for the use of late-stage regulation strategies in older adults insofar as they did not process social information differently from younger adults when they were already in a certain social interaction. Nevertheless, in general there was high comparability in the social worlds of younger and older adults. In the second application study, to examine whether descriptions of social interactions portray subjective well-being, they were analyzed by coding social-interaction features of the APRACE. The results of two studies indicated that the Partner component is the most important in reflecting situational well-being. The (non)mention of the Partner component differentiated between low and high subjective well-being in social interactions and was moderated by valence and age. Almost no other components were associated with situational well-being. This finding can be interpreted in terms of the fundamental need to belong in that the (non)focus on the interaction partner strengthens and protects the social bond. Further, it supports the recent theoretical argument that other people might be the most important situations for us. In sum, this cumulative dissertation provides the APRACE, an approach to depict social interactions holistically and parsimoniously. The APRACE offers not only a common language for communication about social interactions but also a measurement tool for their systematic assessment. Hence, the APRACE might stimulate diverse research on social interactions and open variform applications in practice

Improving conversational dynamics with reactive speech synthesis

The active exchange of ideas and/or information is a crucial feature of human-human conversation. Yet it is a skill that present-day ‘conversational’ interfaces are lacking, which effectively hampers the dynamics of interaction and makes it feel artificial. In this paper, we present a reactive speech synthesis system that can handle user’s interruptions. Initial results of evaluation of our interactive experiment indicate that participants prefer a reactive system to a non-reactive one. Based on participants’ feedback, we suggest potential applications for reactive speech synthesis systems (i.e. interactive tutor and adventure game) and propose further interactive user experiments to evaluate them. We anticipate that the reactive system can offer more engaging and dynamic interaction and improve user experience by making it feel more like a natural human-human conversation