Speech-based recognition of self-reported and observed emotion in a dimensional space

The differences between self-reported and observed emotion have only marginally been investigated in the context of speech-based automatic emotion recognition. We address this issue by comparing self-reported emotion ratings to observed emotion ratings and look at how differences between these two types of ratings affect the development and performance of automatic emotion recognizers developed with these ratings. A dimensional approach to emotion modeling is adopted: the ratings are based on continuous arousal and valence scales. We describe the TNO-Gaming Corpus that contains spontaneous vocal and facial expressions elicited via a multiplayer videogame and that includes emotion annotations obtained via self-report and observation by outside observers. Comparisons show that there are discrepancies between self-reported and observed emotion ratings which are also reflected in the performance of the emotion recognizers developed. Using Support Vector Regression in combination with acoustic and textual features, recognizers of arousal and valence are developed that can predict points in a 2-dimensional arousal-valence space. The results of these recognizers show that the self-reported emotion is much harder to recognize than the observed emotion, and that averaging ratings from multiple observers improves performance

Correlating Facial Expressions and Subjective Player Experiences in Competitive Hearthstone

In this study, we used recordings of players’ facial expressions that are captured during competitive Hearthstone games to analyse the correlation between in-game player affective responses and subjective post-game self-reports. With this, we aimed to examine whether eye gaze, head pose and emotions gathered as objective data from face recordings would be associated with subjective experiences of players which were collected in the form of a post-game survey. Data was collected during a live offline Hearthstone competition, which involved a total of 17 players and 31 matches played. Correlation analyses between in-game and post-game variables show that players’ facial expressions and eye gaze measurements are associated with both players’ attention to the opponent and their mood influenced by the opponent. In future research, these results may be used to implement predictive player models

How to Create Suitable Augmented Reality Application to Teach Social Skills for Children with ASD

Autism spectrum disorders (ASDs) are characterized by a reduced ability to appropriately express social greetings. Studies have indicated that individuals with ASD might not recognize the crucial nonverbal cues that usually aid social interaction. This study applied augmented reality (AR) with tabletop role-playing game (AR-RPG) to focus on the standard nonverbal social cues to teach children with ASD, how to appropriately reciprocate when they socially interact with others. The results showed that intervention system provides an AR combined with physical manipulatives and presents corresponding specific elements in an AR 3D animation with dialogue; thus, it can be used to help them increase their social interaction skills and drive their attention toward the meaning and social value of greeting behavior in specific social situations. We conclude that AR-RPG of social situations helped children with ASD recognize and better understand these situations and moderately effective in teaching the target greeting responses

Play Experience Enhancement Using Emotional Feedback

Innovations in computer game interfaces continue to enhance the experience of players. Affective games - those that adapt or incorporate a player’s emotional state - have shown promise in creating exciting and engaging user experiences. However, a dearth of systematic exploration into what types of game elements should adapt to affective state leaves game designers with little guidance on how to incorporate affect into their games. We created an affective game engine, using it to deploy a design probe into how adapting the player’s abilities, the enemy’s abilities, or variables in the environment affects player performance and experience. Our results suggest that affectively adapting games can increase player arousal. Furthermore, we suggest that reducing challenge by adapting non-player characters is a worse design choice than giving players the tools that they need (through enhancing player abilities or a supportive environment) to master greater challenges

Mapping Beyond the Uncanny Valley: A Delphi Study on Aiding Adoption of Realistic Digital Faces

Developers and HCI researchers have long strived to create digital agents that are more realistic. Voice-only versions are now common, but there has been a lack of visually realistic agents. A key barrier is the “Uncanny Valley”, referring to aversion being triggered if agents are not quite realistic. To gain understanding of the challenges of the Uncanny Valley in creating realistic agents, we conducted a Delphi study. For the Delphi panel, we recruited 13 leading international experts in the area of digital humans. They participated in three rounds of qualitative interviews. We aimed to transfer their knowledge from the entertainment industry to HCI researchers. Our findings include the unexpected conclusion that the panel considered the challenges of final rendering was not a key problem. Instead, modeling and rigging were highlighted, and a new dimension of interactivity was revealed as important. Our results provide a set of research directions for those engaged in HCI-oriented information systems using realistic digital humans

Mapping Beyond the Uncanny Valley: A Delphi Study on Aiding Adoption of Realistic Digital Faces

Developers and HCI researchers have long strived to create digital agents that are more realistic. Voice-only versions are now common, but there has been a lack of visually realistic agents. A key barrier is the “Uncanny Valley”, referring to aversion being triggered if agents are not quite realistic. To gain understanding of the challenges of the Uncanny Valley in creating realistic agents, we conducted a Delphi study. For the Delphi panel, we recruited 13 leading international experts in the area of digital humans. They participated in three rounds of qualitative interviews. We aimed to transfer their knowledge from the entertainment industry to HCI researchers. Our findings include the unexpected conclusion that the panel considered the challenges of final rendering was not a key problem. Instead, modeling and rigging were highlighted, and a new dimension of interactivity was revealed as important. Our results provide a set of research directions for those engaged in HCI-oriented information systems using realistic digital humans