Perceptually Valid Facial Expressions for Character-Based Applications

This paper addresses the problem of creating facial expression of mixed emotions in a perceptually valid way. The research has been done in the context of a “game-like” health and education applications aimed at studying social competency and facial expression awareness in autistic children as well as native language learning, but the results can be applied to many other applications such as games with need for dynamic facial expressions or tools for automating the creation of facial animations. Most existing methods for creating facial expressions of mixed emotions use operations like averaging to create the combined effect of two universal emotions. Such methods may be mathematically justifiable but are not necessarily valid from a perceptual point of view. The research reported here starts by user experiments aiming at understanding how people combine facial actions to express mixed emotions, and how the viewers perceive a set of facial actions in terms of underlying emotions. Using the results of these experiments and a three-dimensional emotion model, we associate facial actions to dimensions and regions in the emotion space, and create a facial expression based on the location of the mixed emotion in the three-dimensional space. We call these regionalized facial actions “facial expression units.

Considerations for believable emotional facial expression animation

Facial expressions can be used to communicate emotional states through the use of universal signifiers within key regions of the face. Psychology research has identified what these signifiers are and how different combinations and variations can be interpreted. Research into expressions has informed animation practice, but as yet very little is known about the movement within and between emotional expressions. A better understanding of sequence, timing, and duration could better inform the production of believable animation. This paper introduces the idea of expression choreography, and how tests of observer perception might enhance our understanding of moving emotional expressions

Multispace behavioral model for face-based affective social agents

This paper describes a behavioral model for affective social agents based on three independent but interacting parameter spaces: knowledge, personality, andmood. These spaces control a lower-level geometry space that provides parameters at the facial feature level. Personality and mood use findings in behavioral psychology to relate the perception of personality types and emotional states to the facial actions and expressions through two-dimensional models for personality and emotion. Knowledge encapsulates the tasks to be performed and the decision-making process using a specially designed XML-based language. While the geometry space provides an MPEG-4 compatible set of parameters for low-level control, the behavioral extensions available through the triple spaces provide flexible means of designing complicated personality types, facial expression, and dynamic interactive scenarios

Text-based Editing of Talking-head Video

Editing talking-head video to change the speech content or to remove filler words is challenging. We propose a novel method to edit talking-head video based on its transcript to produce a realistic output video in which the dialogue of the speaker has been modified, while maintaining a seamless audio-visual flow (i.e. no jump cuts). Our method automatically annotates an input talking-head video with phonemes, visemes, 3D face pose and geometry, reflectance, expression and scene illumination per frame. To edit a video, the user has to only edit the transcript, and an optimization strategy then chooses segments of the input corpus as base material. The annotated parameters corresponding to the selected segments are seamlessly stitched together and used to produce an intermediate video representation in which the lower half of the face is rendered with a parametric face model. Finally, a recurrent video generation network transforms this representation to a photorealistic video that matches the edited transcript. We demonstrate a large variety of edits, such as the addition, removal, and alteration of words, as well as convincing language translation and full sentence synthesis

Relating Objective and Subjective Performance Measures for AAM-based Visual Speech Synthesizers

We compare two approaches for synthesizing visual speech using Active Appearance Models (AAMs): one that utilizes acoustic features as input, and one that utilizes a phonetic transcription as input. Both synthesizers are trained using the same data and the performance is measured using both objective and subjective testing. We investigate the impact of likely sources of error in the synthesized visual speech by introducing typical errors into real visual speech sequences and subjectively measuring the perceived degradation. When only a small region (e.g. a single syllable) of ground-truth visual speech is incorrect we find that the subjective score for the entire sequence is subjectively lower than sequences generated by our synthesizers. This observation motivates further consideration of an often ignored issue, which is to what extent are subjective measures correlated with objective measures of performance? Significantly, we find that the most commonly used objective measures of performance are not necessarily the best indicator of viewer perception of quality. We empirically evaluate alternatives and show that the cost of a dynamic time warp of synthesized visual speech parameters to the respective ground-truth parameters is a better indicator of subjective quality

A framework for automatic and perceptually valid facial expression generation

Facial expressions are facial movements reflecting the internal emotional states of a character or in response to social communications. Realistic facial animation should consider at least two factors: believable visual effect and valid facial movements. However, most research tends to separate these two issues. In this paper, we present a framework for generating 3D facial expressions considering both the visual the dynamics effect. A facial expression mapping approach based on local geometry encoding is proposed, which encodes deformation in the 1-ring vector. This method is capable of mapping subtle facial movements without considering those shape and topological constraints. Facial expression mapping is achieved through three steps: correspondence establishment, deviation transfer and movement mapping. Deviation is transferred to the conformal face space through minimizing the error function. This function is formed by the source neutral and the deformed face model related by those transformation matrices in 1-ring neighborhood. The transformation matrix in 1-ring neighborhood is independent of the face shape and the mesh topology. After the facial expression mapping, dynamic parameters are then integrated with facial expressions for generating valid facial expressions. The dynamic parameters were generated based on psychophysical methods. The efficiency and effectiveness of the proposed methods have been tested using various face models with different shapes and topological representations

The Evaluation of Stylized Facial Expressions

Stylized rendering aims to abstract information in an image making it useful not only for artistic but also for visualization purposes. Recent advances in computer graphics techniques have made it possible to render many varieties of stylized imagery efficiently. So far, however, few attempts have been made to characterize the perceptual impact and effectiveness of stylization. In this paper, we report several experiments that evaluate three different stylization techniques in the context of dynamic facial expressions. Going beyond the usual questionnaire approach, the experiments compare the techniques according to several criteria ranging from introspective measures (subjective preference) to task-dependent measures (recognizability, intensity). Our results shed light on how stylization of image contents affects the perception and subjective evaluation of facial expressions

Cultural differences in the decoding and representation of facial expression signals

Summary. In this thesis, I will challenge one of the most fundamental assumptions of psychological science – the universality of facial expressions. I will do so by first reviewing the literature to reveal major flaws in the supporting arguments for universality. I will then present new data demonstrating how culture has shaped the decoding and transmission of facial expression signals. A summary of both sections are presented below. Review of the Literature To obtain a clear understanding of how the universality hypothesis developed, I will present the historical course of the emotion literature, reviewing relevant works supporting notions of a ‘universal language of emotion.’ Specifically, I will examine work on the recognition of facial expressions across cultures as it constitutes a main component of the evidence for universality. First, I will reveal that a number of ‘seminal’ works supporting the universality hypothesis are critically flawed, precluding them from further consideration. Secondly, by questioning the validity of the statistical criteria used to demonstrate ‘universal recognition,’ I will show that long-standing claims of universality are both misleading and unsubstantiated. On a related note, I will detail the creation of the ‘universal’ facial expression stimulus set (Facial Action Coding System -FACS- coded facial expressions) to reveal that it is in fact a biased, culture-specific representation of Western facial expressions of emotion. The implications for future cross-cultural work are discussed in relation to the limited FACS-coded stimulus set. Experimental Work In reviewing the literature, I will reveal a latent phenomenon which has so far remained unexplained – the East Asian (EA) recognition deficit. Specifically, EA observers consistently perform significantly poorer when categorising certain ‘universal’ facial expressions compared to Western Caucasian (WC) observers – a surprisingly neglected finding given the importance of emotion communication for human social interaction. To address this neglected issue, I examined both the decoding and transmission of facial expression signals in WC and EA observers. Experiment 1: Cultural Decoding of ‘Universal’ Facial Expressions of Emotion To examine the decoding of ‘universal’ facial expressions across cultures, I used eye tracking technology to record the eye movements of WC and EA observers while they categorised the 6 ‘universal’ facial expressions of emotion. My behavioural results demonstrate the robustness of the phenomenon by replicating the EA recognition deficit (i.e., EA observers are significantly poorer at recognizing facial expressions of ‘fear’ and ‘disgust’). Further inspection of the data also showed that EA observers systematically miscategorise ‘fear’ as ‘surprise’ and ‘disgust’ as ‘anger.’ Using spatio-temporal analyses of fixations, I will show that WC and EA observers use culture-specific fixation strategies to decode ‘universal’ facial expressions of emotion. Specifically, while WC observers distribute fixations across the face, sampling the eyes and mouth, EA observers persistently bias fixations towards the eyes and neglect critical features, especially for facial expressions eliciting significant confusion (i.e., ‘fear,’ ‘disgust,’ and ‘anger’). My behavioural data showed that EA observers systematically miscategorise ‘fear’ as ‘surprise’ and ‘disgust’ as ‘anger.’ Analysis of my eye movement data also showed that EA observers repetitively sample information from the eye region during facial expression decoding, particularly for those eliciting significant behavioural confusions (i.e., ‘fear,’ ‘disgust,’ and ‘anger’). To objectively examine whether the EA culture-specific fixation pattern could give rise to the reported behavioural confusions, I built a model observer that samples information from the face to categorise facial expressions. Using this model observer, I will show that the EA decoding strategy is inadequate to distinguish ‘fear’ from ‘surprise’ and ‘disgust’ from ‘anger,’ thus giving rise to the reported EA behavioural confusions. For the first time, I will reveal the origins of a latent phenomenon - the EA recognition deficit. I discuss the implications of culture-specific decoding strategies during facial expression categorization in light of current theories of cross-cultural emotion communication. Experiment 2: Cultural Internal Representations of Facial Expressions of Emotion In the previous two experiments, I presented data that questions the universality of facial expressions. As replicated in Experiment 1, WC and EA observers differ significantly in their recognition performance for certain ‘universal’ facial expressions. In Experiment 1, I showed culture-specific fixation patterns, demonstrating cultural differences in the predicted locations of diagnostic information. Together, these data predict cultural specificity in facial expression signals, supporting notions of cultural ‘accents’ and/or ‘dialects.’ To examine whether facial expression signals differ across cultures, I used a powerful reverse correlation (RC) technique to reveal the internal representations of the 6 ‘basic’ facial expressions of emotion in WC and EA observers. Using complementary statistical image processing techniques to examine the signal properties of each internal representation, I will directly reveal cultural specificity in the representations of the 6 ‘basic’ facial expressions of emotion. Specifically, I will show that while WC representations of facial expressions predominantly featured the eyebrows and mouth, EA representations were biased towards the eyes, as predicted by my eye movement data in Experiment 1. I will also show gaze avoidance as unique feature of the EA group. In sum, this data shows clear cultural contrasts in facial expression signals by showing that culture shapes the internal representations of emotion. Future Work My review of the literature will show that pivotal concepts such as ‘recognition’ and ‘universality’ are currently flawed and have misled both the interpretation of empirical work the direction of theoretical developments. Here, I will examine each concept in turn and propose more accurate criteria with which to demonstrate ‘universal recognition’ in future studies. In doing so, I will also detail possible future studies designed to address current gaps in knowledge created by use of inappropriate criteria. On a related note, having questioned the validity of FACS-coded facial expressions as ‘universal’ facial expressions, I will highlight an area for empirical development – the creation of a culturally valid facial expression stimulus set – and detail future work required to address this question. Finally, I will discuss broader areas of interest (i.e., lexical structure of emotion) which could elevate current knowledge of cross-cultural facial expression recognition and emotion communication in the future