Predicting Human Interpretations of Affect and Valence in a Social Robot

In this paper we seek to understand how people interpret a social robot’s performance of an emotion, what we term ‘affective display,’ and the positive or negative valence of that affect. To this end, we tasked annotators with observing the Anki Cozmo robot perform its over 900 pre-scripted behaviors and labeling those behaviors with 16 possible affective display labels (e.g., interest, boredom, disgust, etc.). In our first experiment, we trained a neural network to predict annotated labels given multimodal information about the robot’s movement, face, and audio. The results suggest that pairing affects to predict the valence between them is more informative, which we confirmed in a second experiment. Both experiments show that certain modalities are more useful for predicting displays of affect and valence. For our final experiment, we generated novel robot behaviors and tasked human raters with assigning scores to valence pairs instead of applying labels, then compared our model’s predictions of valence between the affective pairs and compared the results to the human ratings. We conclude that some modalities have information that can be contributory or inhibitive when considered in conjunction with other modalities, depending on the emotional valence pair being considered

Do You Feel Me?: Learning Language from Humans with Robot Emotional Displays

In working towards accomplishing a human-level acquisition and understanding of language, a robot must meet two requirements: the ability to learn words from interactions with its physical environment, and the ability to learn language from people in settings for language use, such as spoken dialogue. The second requirement poses a problem: If a robot is capable of asking a human teacher well-formed questions, it will lead the teacher to provide responses that are too advanced for a robot, which requires simple inputs and feedback to build word-level comprehension. In a live interactive study, we tested the hypothesis that emotional displays are a viable solution to this problem of how to communicate without relying on language the robot doesn\u27t--indeed, cannot--actually know. Emotional displays can relate the robot\u27s state of understanding to its human teacher, and are developmentally appropriate for the most common language acquisition setting: an adult interacting with a child. For our study, we programmed a robot to independently explore the world and elicit relevant word references and feedback from the participants who are confronted with two robot settings: a setting in which the robot displays emotions, and a second setting where the robot focuses on the task without displaying emotions, which also tests if emotional displays lead a participant to make incorrect assumptions regarding the robot\u27s understanding. Analyzing the results from the surveys and the Grounded Semantics classifiers, we discovered that the use of emotional displays increases the number of inputs provided to the robot, an effect that\u27s modulated by the ratio of positive to negative emotions that were displayed

Expression of Grounded Affect: How Much Emotion Can Arousal Convey?

Springer: © 2020 Springer Nature Switzerland AG. This is a post-peer-review, pre-copyedit version of Hickton L., Lewis M., Cañamero L. (2020) Expression of Grounded Affect: How Much Emotion Can Arousal Convey?. In: Mohammad A., Dong X., Russo M. (eds) Towards Autonomous Robotic Systems. TAROS 2020. Lecture Notes in Computer Science, vol 12228. Springer, Cham. The final authenticated version is available online at: https://doi.org/10.1007/978-3-030-63486-5_26.In this paper we consider how non-humanoid robots can communicate their affective state via bodily forms of communication (kinesics), and the extent to which this influences how humans respond to them. We propose a simple model of grounded affect and kinesic expression before presenting the qualitative findings of an exploratory study (N=9), during which participants were interviewed after watching expressive and non-expressive hexapod robots perform different ‘scenes’. A summary of these interviews is presented and a number of emerging themes are identified and discussed. Whilst our findings suggest that the expressive robot did not evoke significantly greater empathy or altruistic intent in humans than the control robot, the expressive robot stimulated greater desire for interaction and was also more likely to be attributed with emotion

A Study of Non-Linguistic Utterances for Social Human-Robot Interaction

The world of animation has painted an inspiring image of what the robots of the future could be. Taking the robots R2D2 and C3PO from the Star Wars films as representative examples, these robots are portrayed as being more than just machines, rather, they are presented as intelligent and capable social peers, exhibiting many of the traits that people have also. These robots have the ability to interact with people, understand us, and even relate to us in very personal ways through a wide repertoire of social cues. As robotic technologies continue to make their way into society at large, there is a growing trend toward making social robots. The field of Human-Robot Interaction concerns itself with studying, developing and realising these socially capable machines, equipping them with a very rich variety of capabilities that allow them to interact with people in natural and intuitive ways, ranging from the use of natural language, body language and facial gestures, to more unique ways such as expression through colours and abstract sounds. This thesis studies the use of abstract, expressive sounds, like those used iconically by the robot R2D2. These are termed Non-Linguistic Utterances (NLUs) and are a means of communication which has a rich history in film and animation. However, very little is understood about how such expressive sounds may be utilised by social robots, and how people respond to these. This work presents a series of experiments aimed at understanding how NLUs can be utilised by a social robot in order to convey affective meaning to people both young and old, and what factors impact on the production and perception of NLUs. Firstly, it is shown that not all robots should use NLUs. The morphology of the robot matters. People perceive NLUs differently across different robots, and not always in a desired manner. Next it is shown that people readily project affective meaning onto NLUs though not in a coherent manner. Furthermore, people's affective inferences are not subtle, rather they are drawn to well established, basic affect prototypes. Moreover, it is shown that the valence of the situation in which an NLU is made, overrides the initial valence of the NLU itself: situational context biases how people perceive utterances made by a robot, and through this, coherence between people in their affective inferences is found to increase. Finally, it is uncovered that NLUs are best not used as a replacement to natural language (as they are by R2D2), rather, people show a preference for them being used alongside natural language where they can play a supportive role by providing essential social cues

Continuous Analysis of Affect from Voice and Face

Human affective behavior is multimodal, continuous and complex. Despite major advances within the affective computing research field, modeling, analyzing, interpreting and responding to human affective behavior still remains a challenge for automated systems as affect and emotions are complex constructs, with fuzzy boundaries and with substantial individual differences in expression and experience [7]. Therefore, affective and behavioral computing researchers have recently invested increased effort in exploring how to best model, analyze and interpret the subtlety, complexity and continuity (represented along a continuum e.g., from −1 to +1) of affective behavior in terms of latent dimensions (e.g., arousal, power and valence) and appraisals, rather than in terms of a small number of discrete emotion categories (e.g., happiness and sadness). This chapter aims to (i) give a brief overview of the existing efforts and the major accomplishments in modeling and analysis of emotional expressions in dimensional and continuous space while focusing on open issues and new challenges in the field, and (ii) introduce a representative approach for multimodal continuous analysis of affect from voice and face, and provide experimental results using the audiovisual Sensitive Artificial Listener (SAL) Database of natural interactions. The chapter concludes by posing a number of questions that highlight the significant issues in the field, and by extracting potential answers to these questions from the relevant literature. The chapter is organized as follows. Section 10.2 describes theories of emotion, Sect. 10.3 provides details on the affect dimensions employed in the literature as well as how emotions are perceived from visual, audio and physiological modalities. Section 10.4 summarizes how current technology has been developed, in terms of data acquisition and annotation, and automatic analysis of affect in continuous space by bringing forth a number of issues that need to be taken into account when applying a dimensional approach to emotion recognition, namely, determining the duration of emotions for automatic analysis, modeling the intensity of emotions, determining the baseline, dealing with high inter-subject expression variation, defining optimal strategies for fusion of multiple cues and modalities, and identifying appropriate machine learning techniques and evaluation measures. Section 10.5 presents our representative system that fuses vocal and facial expression cues for dimensional and continuous prediction of emotions in valence and arousal space by employing the bidirectional Long Short-Term Memory neural networks (BLSTM-NN), and introduces an output-associative fusion framework that incorporates correlations between the emotion dimensions to further improve continuous affect prediction. Section 10.6 concludes the chapter

Extending the Affective Technology Acceptance Model to Human-Robot Interactions: A Multi-Method Perspective

The current study sought to extend the Affective Technology Acceptance (ATA) model to human-robot interactions. We tested the direct relationship between affect and technology acceptance of a security robot. Affect was measured using a multi-method approach, which included a self-report survey, as well as sentiment analysis, and response length of written responses. Results revealed that participants who experienced positive affect were more likely to accept technology. However, the significance and direction of the relationship between negative affect and technology acceptance was measurement dependent. Additionally, positive and negative sentiment words accounted for unique variance in technology acceptance, after controlling for self-reported affect. This study demonstrates that affect is an important contributing factor in human-robot interaction research, and using a multi-method approach allows for a richer, more complete understanding of how human feelings influence robot acceptance

Examining affective structure in chickens: valence, intensity, persistence and generalization measured using a conditioned place preference test

When measuring animals’ valenced behavioural responses to stimuli, the Conditioned Place Preference (CPP) test goes a step further than many approach-based and avoidance-based tests by establishing whether a learned preference for, or aversion to, the location in which the stimulus was encountered can be generated. We designed a novel, four-chambered CPP test to extend the capability of the usual CPP paradigm to provide information on four key features of animals’ affective responses: valence, scale, persistence and generalization. Using this test, we investigated the affective responses of domestic chickens (Gallus gallus domesticus) to four potentially aversive stimuli: 1. Puffs of air; 2. Sight of (robotic) snake; 3. Sprays of water; 4. Sound of conspecific alarm calls. We found conditioned avoidance of locations associated with the air puffs and water sprays (Friedman’s χ2(3) = 13.323 p > .005; χ2(3) = 14.235 p > .005), but not with the snake and alarm calls. The scale of the learned avoidance was similar for the air puff and water spray stimuli, but persistence and generalization differed. We conclude that the four chambered CPP test can have a valuable role to play in making multi-feature measurements of stimulus-generated affective responses, and we highlight the value of such measurements for improving our understanding of the structure of affect in chickens and other animals