Speech-based recognition of self-reported and observed emotion in a dimensional space

The differences between self-reported and observed emotion have only marginally been investigated in the context of speech-based automatic emotion recognition. We address this issue by comparing self-reported emotion ratings to observed emotion ratings and look at how differences between these two types of ratings affect the development and performance of automatic emotion recognizers developed with these ratings. A dimensional approach to emotion modeling is adopted: the ratings are based on continuous arousal and valence scales. We describe the TNO-Gaming Corpus that contains spontaneous vocal and facial expressions elicited via a multiplayer videogame and that includes emotion annotations obtained via self-report and observation by outside observers. Comparisons show that there are discrepancies between self-reported and observed emotion ratings which are also reflected in the performance of the emotion recognizers developed. Using Support Vector Regression in combination with acoustic and textual features, recognizers of arousal and valence are developed that can predict points in a 2-dimensional arousal-valence space. The results of these recognizers show that the self-reported emotion is much harder to recognize than the observed emotion, and that averaging ratings from multiple observers improves performance

Discourse Analysis of the 2022 Australian Tennis Open: A Multimodal Appraisal Perspective

This article presents a preliminary analysis of a corpus of texts relating to the 2022 Australian Tennis Open using a multimodal appraisal framework. The study utilises quantitative and qualitative content analysis to examine media reports, official statements, and public reactions to the incident, which centred around Novak Djokovic's vaccination status. The analysis focusses on assessing how evaluative language contributes to community-building and identifies the underlying values, beliefs, and evaluations that shape stakeholders' emotional, cognitive, and behavioural responses.The appraisal framework, encompassing attitude, engagement, and graduation, serves as a comprehensive tool for categorising resources that express evaluation. Furthermore, the article delves into the application of appraisal analysis within the context of multimodal and online discourse, encompassing various platforms such as newspapers, television, radio, YouTube, Twitter, Instagram, blogs, official political statements, and court rulings. By examining these diverse media, the study seeks to investigate the dynamic discourse interplay surrounding the 2022 Australian Open, highlighting the pivotal role of evaluative communication in fostering alignment among readers through shared values and attitudes.The preliminary findings suggest that access to greater semiotic recourses increases consensus. The gains from using this interpretative framework are an asset, facilitating the coding of a large data set and attending the different manifestations of discourses around the player’s participation. As discourse continues to shape societal narratives, this multimodal appraisal investigation contributes to our understanding of the complex dynamics inherent in discourse construction and the influence of evaluative language in shaping collective perception

Investigating Context Awareness of Affective Computing Systems: A Critical Approach

AbstractIntelligent Human Computer Interaction systems should be affective aware and Affective Computing systems should be context aware. Positioned in the cross-section of the research areas of Interaction Context and Affective Computing current paper investigates if and how context is incorporated in automatic analysis of human affective behavior. Several related aspects are discussed ranging from modeling, acquiring and annotating issues in affectively enhanced corpora to issues related to incorporating context information in a multimodal fusion framework of affective analysis. These aspects are critically discussed in terms of the challenges they comprise while, in a wider framework, future directions of this recently active, yet mainly unexplored, research area are identified. Overall, the paper aims to both document the present status as well as comment on the evolution of the upcoming topic of Context in Affective Computing

An Actor-Centric Approach to Facial Animation Control by Neural Networks For Non-Player Characters in Video Games

Game developers increasingly consider the degree to which character animation emulates facial expressions found in cinema. Employing animators and actors to produce cinematic facial animation by mixing motion capture and hand-crafted animation is labor intensive and therefore expensive. Emotion corpora and neural network controllers have shown promise toward developing autonomous animation that does not rely on motion capture. Previous research and practice in disciplines of Computer Science, Psychology and the Performing Arts have provided frameworks on which to build a workflow toward creating an emotion AI system that can animate the facial mesh of a 3d non-player character deploying a combination of related theories and methods. However, past investigations and their resulting production methods largely ignore the emotion generation systems that have evolved in the performing arts for more than a century. We find very little research that embraces the intellectual process of trained actors as complex collaborators from which to understand and model the training of a neural network for character animation. This investigation demonstrates a workflow design that integrates knowledge from the performing arts and the affective branches of the social and biological sciences. Our workflow begins at the stage of developing and annotating a fictional scenario with actors, to producing a video emotion corpus, to designing training and validating a neural network, to analyzing the emotion data annotation of the corpus and neural network, and finally to determining resemblant behavior of its autonomous animation control of a 3d character facial mesh. The resulting workflow includes a method for the development of a neural network architecture whose initial efficacy as a facial emotion expression simulator has been tested and validated as substantially resemblant to the character behavior developed by a human actor

Humour support and emotive stance in comments on Korean TV Drama

Viewers on viki.com comment on Korean television drama series while watching: They produce timed comments tied to the timecode of the audiovisual stream. Among the functions these comments have in the community, the expression of emotive stance is central. Importantly, this includes humour support encoded in a variety of linguistic and paralinguistic ways. Our study identifies a range of humour support indicators, which allow us to find comments that are responses to humour. Accordingly, our study explores how commenters make use of the affordances of the Viki timed comment feature to linguistically and paralinguistically encode their humorous reaction to fictional events and to previous comments. We do this both quantitatively e based on a multilingual corpus of all 320,118 timed comments that accompany five Korean dramas we randomly selected (80 episodes in total), and qualitatively based on the in-depth analysis of two episodes. What we contribute is a typology and the distribution of humour support indicators used in a novel genre of technology-mediated communication as well as insights into how the viewing community collectively does humour support. Finally, we also present the semi-automatic detection of humour support as a viable strategy to objectively identify humour-relevant scenes in Korean TV drama

The Language of League: Making Sense of Multimodal Meaning in Twitch Live Streams

Though there has been a good deal of research on digital discourse and online gaming, there has been relatively little research on 1) the social structure of specific groups within the large online gaming community, 2) the multimodal structure of the online gaming live stream, and 3) the impact that these structures have on the final communicative event. One noteworthy component of the social characteristics of online streams is the streamer gender and size of the stream’s audience. In addition, one difference that sets the live stream apart from other online communications is its intense technological complexity. This study then, will examine both of these social and technological characteristics, in an effort to understand how the participants themselves influence language use and how that language use is further impacted by the availability of multiple mediums, each of which houses multiple modes for communication. The data for this study consists of a corpus of 32,397 messages posted in the public chat area of 12 League of Legends live streamers, collected between July and September of 2019. Once collected, however, there was no prior convention in place for organizing and transcribed the data for analytical purposes. Therefore, this study also examines multiple transcription vi protocols and outlines the model developed by Graham and Arendall for an online gaming digital corpus. For this study, I take an interactional approach to explore the communicative strategies employed by participants in a complex multimedium-based multimodal event. Using quantitative analysis, I examine patterns of communicative strategies as related to streamer gender and stream size (participant population). In addition, I examine the qualitative characteristics of those patterns, as well as the influences that multiple available mediums and modes have on those patterns. The results of this analysis indicate that both social and technological characteristics of the live stream heavily impact the communicative strategies employed by participants and is often tailored to the specific needs of each community, especially where the use of graphic images is concerned. These results have implications for the further study of online gaming, live streams, and visual communications within multimediumbased multimodal events

Acoustic Features of Different Types of Laughter in North Sami Conversational Speech

Peer reviewe

An ecological method for the sampling of nonverbal signalling behaviours of young children with profound and multiple learning disabilities (PMLD)

- Background: Profound and multiple learning disabilities (PMLD) are a complex range of disabilities that affect the general health and wellbeing of the individual and their capacity to interact and learn. - Method: We developed a new methodology to capture the nonsymbolic signalling behaviours of children with PMLD within the context of a face-to-face interaction with a caregiver to provide analysis at a micro-level of descriptive detail incorporating the use of the ELAN digital video software. - Conclusion: The signalling behaviours of participants in a natural, everyday interaction can be better understood with the use of this innovation in methodology, which is predicated on the ecology of communication. Recognition of the developmental ability of the participants is an integral factor within that ecology. The method presented establishes an advanced account of the modalities through which a child affected by PMLD is able to communicate