Adaptive 3D facial action intensity estimation and emotion recognition

Automatic recognition of facial emotion has been widely studied for various computer vision tasks (e.g. health monitoring, driver state surveillance and personalized learning). Most existing facial emotion recognition systems, however, either have not fully considered subject-independent dynamic features or were limited to 2D models, thus are not robust enough for real-life recognition tasks with subject variation, head movement and illumination change. Moreover, there is also lack of systematic research on effective newly arrived novel emotion class detection. To address these challenges, we present a real-time 3D facial Action Unit (AU) intensity estimation and emotion recognition system. It automatically selects 16 motion-based facial feature sets using minimal-redundancy–maximal-relevance criterion based optimization and estimates the intensities of 16 diagnostic AUs using feedforward Neural Networks and Support Vector Regressors. We also propose a set of six novel adaptive ensemble classifiers for robust classification of the six basic emotions and the detection of newly arrived unseen novel emotion classes (emotions that are not included in the training set). A distance-based clustering and uncertainty measures of the base classifiers within each ensemble model are used to inform the novel class detection. Evaluated with the Bosphorus 3D database, the system has achieved the best performance of 0.071 overall Mean Squared Error (MSE) for AU intensity estimation using Support Vector Regressors, and 92.2% average accuracy for the recognition of the six basic emotions using the proposed ensemble classifiers. In comparison with other related work, our research outperforms other state-of-the-art research on 3D facial emotion recognition for the Bosphorus database. Moreover, in on-line real-time evaluation with real human subjects, the proposed system also shows superior real-time performance with 84% recognition accuracy and great flexibility and adaptation for newly arrived novel (e.g. ‘contempt’ which is not included in the six basic emotions) emotion detection

Modelling, Classification and Synthesis of Facial Expressions

The field of computer vision endeavours to develop automatic approaches to the interpretation of images from the real world. Over the past number of decades researchers within this field have created systems specifically for the automatic analysis of facial expression. The most successful of these approaches draw on the tools from behavioural science. In this chapter we examine facial expression analysis from both a behavioural science and a computer vision perspective. First we will provide details of the principal approach used in behavioural science to analyze facial expressions. This will include an overview of the evolution of facial expression analysis, where we introduce the field of facial expression analysis with Darwin’s initial findings (Darwin, 1872). We then go on to show how his findings were confirmed nearly 100 years later by Ekman et al. (Ekman et al., 1969). Following on from this we provide details of recent works investigating the appearance and dynamics of facial expressions

Teardrops on My Face: Automatic Weeping Detection from Nonverbal Behavior

Human emotional tears are a powerful socio-emotional signal. Yet, they have received relatively little attention in empirical research compared to facial expressions or body posture. While humans are highly sensitive to others' tears, to date, no automatic means exist for detecting spontaneous weeping. This paper employed facial and postural features extracted using four pre-trained classifiers (FACET, Affdex, OpenFace, OpenPose) to train a Support Vector Machine (SVM) to distinguish spontaneous weepers from non-weepers. Results showed that weeping can be accurately inferred from nonverbal behavior. Importantly, this distinction can be made before the appearance of visible tears on the face. However, features from at least two classifiers need to be combined, with the best models blending three or four classifiers to achieve near-perfect performance (97% accuracy). We discuss how direct and indirect tear detection methods may help to yield important new insights into the antecedents and consequences of emotional tears and how affective computing could benefit from the ability to recognize and respond to this uniquely human signal