Ensemble of Hankel Matrices for Face Emotion Recognition

In this paper, a face emotion is considered as the result of the composition of multiple concurrent signals, each corresponding to the movements of a specific facial muscle. These concurrent signals are represented by means of a set of multi-scale appearance features that might be correlated with one or more concurrent signals. The extraction of these appearance features from a sequence of face images yields to a set of time series. This paper proposes to use the dynamics regulating each appearance feature time series to recognize among different face emotions. To this purpose, an ensemble of Hankel matrices corresponding to the extracted time series is used for emotion classification within a framework that combines nearest neighbor and a majority vote schema. Experimental results on a public available dataset shows that the adopted representation is promising and yields state-of-the-art accuracy in emotion classification.Comment: Paper to appear in Proc. of ICIAP 2015. arXiv admin note: text overlap with arXiv:1506.0500

A temporal latent topic model for facial expression recognition

Posters: no. 128LNCS v. 6495 is conference proceedings of the 10th Asian Conference on Computer Vision, Queens, ACCVIn this paper we extend the latent Dirichlet allocation (LDA) topic model to model facial expression dynamics. Our topic model integrates the temporal information of image sequences through redefining topic generation probability without involving new latent variables or increasing inference difficulties. A collapsed Gibbs sampler is derived for batch learning with labeled training dataset and an efficient learning method for testing data is also discussed. We describe the resulting temporal latent topic model (TLTM) in detail and show how it can be applied to facial expression recognition. Experiments on CMU expression database illustrate that the proposed TLTM is very efficient in facial expression recognition. © 2011 Springer-Verlag Berlin Heidelberg.postprintThe 10th Asian Conference on Computer Vision (ACCV 2010), Queenstown, New Zealand, 8-12 November 2010. In Lecture Notes in Computer Science, 2010, v. 6495, p. 51-6

Improving Facial Action Unit Recognition Using Convolutional Neural Networks

Recognizing facial action units (AUs) from spontaneous facial expression is a challenging problem, because of subtle facial appearance changes, free head movements, occlusions, and limited AU-coded training data. Most recently, convolutional neural networks (CNNs) have shown promise on facial AU recognition. However, CNNs are often overfitted and do not generalize well to unseen subject due to limited AU-coded training images. In order to improve the performance of facial AU recognition, we developed two novel CNN frameworks, by substituting the traditional decision layer and convolutional layer with the incremental boosting layer and adaptive convolutional layer respectively, to recognize the AUs from static image. First, in order to handle the limited AU-coded training data and reduce the overfitting, we proposed a novel Incremental Boosting CNN (IB-CNN) to integrate boosting into the CNN via an incremental boosting layer that selects discriminative neurons from the lower layer and is incrementally updated on successive mini-batches. In addition, a novel loss function that accounts for errors from both the incremental boosted classifier and individual weak classifiers was proposed to fine-tune the IBCNN. Experimental results on four benchmark AU databases have demonstrated that the IB-CNN yields significant improvement over the traditional CNN and the boosting CNN without incremental learning, as well as outperforming the state-of-the-art CNN-based methods in AU recognition. The improvement is more impressive for the AUs that have the lowest frequencies in the databases. Second, all current CNNs use predefined and fixed convolutional filter size. However, AUs activated by different facial muscles cause facial appearance changes at different scales and thus favor different filter sizes. The traditional strategy is to experimentally select the best filter size for each AU in each convolutional layer, but it suffers from expensive training cost, especially when the networks become deeper and deeper. We proposed a novel Optimized Filter Size CNN (OFS-CNN), where the filter sizes and weights of all convolutional layers are learned simultaneously from the training data along with learning convolutional filters. Specifically, the filter size is defined as a continuous variable, which is optimized by minimizing the training loss. Experimental results on four AU-coded databases and one spontaneous facial expression database outperforms traditional CNNs with fixed filter sizes and achieves state-of-the-art recognition performance. Furthermore, the OFS-CNN also beats traditional CNNs using the best filter size obtained by exhaustive search and is capable of estimating optimal filter size for varying image resolution

A study of the temporal relationship between eye actions and facial expressions

A dissertation submitted in ful llment of the requirements for the degree of Master of Science in the School of Computer Science and Applied Mathematics Faculty of Science August 15, 2017Facial expression recognition is one of the most common means of communication used for complementing spoken word. However, people have grown to master ways of ex- hibiting deceptive expressions. Hence, it is imperative to understand di erences in expressions mostly for security purposes among others. Traditional methods employ machine learning techniques in di erentiating real and fake expressions. However, this approach does not always work as human subjects can easily mimic real expressions with a bit of practice. This study presents an approach that evaluates the time related dis- tance that exists between eye actions and an exhibited expression. The approach gives insights on some of the most fundamental characteristics of expressions. The study fo- cuses on nding and understanding the temporal relationship that exists between eye blinks and smiles. It further looks at the relationship that exits between eye closure and pain expressions. The study incorporates active appearance models (AAM) for feature extraction and support vector machines (SVM) for classi cation. It tests extreme learn- ing machines (ELM) in both smile and pain studies, which in turn, attains excellent results than predominant algorithms like the SVM. The study shows that eye blinks are highly correlated with the beginning of a smile in posed smiles while eye blinks are highly correlated with the end of a smile in spontaneous smiles. A high correlation is observed between eye closure and pain in spontaneous pain expressions. Furthermore, this study brings about ideas that lead to potential applications such as lie detection systems, robust health care monitoring systems and enhanced animation design systems among others.MT 201

Developing a Computer System for the Generation of Unique Wrinkle Maps for Human Faces. Generating 2D Wrinkle Maps using Various Image Processing Techniques and the Design of 3D Facial Ageing System using 3D Modelling Tools.

Facial Ageing (FA) is a very fundamental issue, as ageing in general, is part of our daily life process. FA is used in security, finding missing children and other applications. It is also a form of Facial Recognition (FR) that helps identifying suspects. FA affects several parts of the human face under the influence of different biological and environmental factors. One of the major facial feature changes that occur as a result of ageing is the appearance and development of wrinkles. Facial wrinkles are skin folds; their shapes and numbers differ from one person to another, therefore, an advantage can be taken over these characteristics if a system is implemented to extract the facial wrinkles in a form of maps. This thesis is presenting a new technique for three-dimensional facial wrinkle pattern information that can also be utilised for biometric applications, which will back up the system for further increase of security. The procedural approaches adopted for investigating this new technique are the extraction of two-dimensional wrinkle maps of frontal human faces for digital images and the design of three-dimensional wrinkle pattern formation system that utilises the generated wrinkle maps. The first approach is carried out using image processing tools so that for any given individual, two wrinkle maps are produced; the first map is in a binary form that shows the positions of the wrinkles on the face while the other map is a coloured version that indicates the different intensities of the wrinkles. The second approach of the 3D system development involves the alignment of the binary wrinkle maps on the corresponding 3D face models, followed by the projection of 3D curves in order to acquire 3D representations of the wrinkles. With the aid of the coloured wrinkle maps as well as some ageing parameters, simulations and predictions for the 3D wrinkles are performed