A spatial-temporal framework based on histogram of gradients and optical flow for facial expression recognition in video sequences

Facial expression causes different parts of the facial region to change over time and thus dynamic descriptors are inherently more suitable than static descriptors for recognising facial expressions. In this paper, we extend the spatial pyramid histogram of gradients to spatio-temporal domain to give 3-dimensional facial features and integrate them with dense optical flow to give a spatio-temporal descriptor which extracts both the spatial and dynamic motion information of facial expressions. A multi-class support vector machine based classifier with one-to-one strategy is used to recognise facial expressions. Experiments on the CK+ and MMI datasets using leave-one-out cross validation scheme demonstrate that the integrated framework achieves a better performance than using individual descriptor separately. Compared with six state of the art methods, the proposed framework demonstrates a superior performance

A Neural Network Based Classifier for a Segmented Facial Expression Recognition System Based on Haar Wavelet Transform

Automatic recognition of facial expressions is a vital component of natural human-machine interfaces. Facial expressions convey information about one's emotional state and helps regulate our social norms by helping detect and interpret a scene. In this paper, we propose a novel face expression recognition scheme based on Haar discrete wavelet transform and a neural network classifier. First, the sample image undergoes preprocessing where noise is removed using binary image processing techniques. Then feature vectors are extracted using DWT from corresponding pixels in the image. The extracted image pixel data are used as the input to the neural network. We demonstrate experimentally that when wavelet coefficients are fed into a back-propagation based neural network for classification, a high recognition rate can be achieved by using a very small proportion of transform coefficients. Based on our experimental results, the proposed scheme gives satisfactory results

Facial Expression Recognition Using New Feature Extraction Algorithm

This paper proposes a method for facial expression recognition. Facial feature vectors are generated from keypoint descriptors using Speeded-Up Robust Features. Each facial feature vector is then normalized and next the probability density function descriptor is generated. The distance between two probability density function descriptors is calculated using Kullback Leibler divergence. Mathematical equation is employed to select certain practicable probability density function descriptors for each grid, which are used as the initial classification. Subsequently, the corresponding weight of the class for each grid is determined using a weighted majority voting classifier. The class with the largest weight is output as the recognition result. The proposed method shows excellent performance when applied to the Japanese Female Facial Expression database

Fusing dynamic deep learned features and handcrafted features for facial expression recognition

The automated recognition of facial expressions has been actively researched due to its wide-ranging applications. The recent advances in deep learning have improved the performance facial expression recognition (FER) methods. In this paper, we propose a framework that combines discriminative features learned using convolutional neural networks and handcrafted features that include shape- and appearance-based features to further improve the robustness and accuracy of FER. In addition, texture information is extracted from facial patches to enhance the discriminative power of the extracted textures. By encoding shape, appearance, and deep dynamic information, the proposed framework provides high performance and outperforms state-of-the-art FER methods on the CK+ dataset

Facial Landmark Based Region of Interest Localization for Deep Facial Expression Recognition

Automated facial expression recognition has gained much attention in the last years due to growing application areas such as computer animated agents, sociable robots and human computer interaction. The realization of a reliable facial expression recognition system through machine learning is still a challenging task particularly on databases with large number of images. Convolutional Neural Network (CNN) architectures have been proposed to deal with large numbers of training data for better accuracy. For CNNs, a task related best achieving architectural structure does not exist. In addition, the representation of the input image is equivalently important as the architectural structure and the training data. Therefore, this study focuses on the performances of various CNN architectures trained by different region of interests of the same input data. Experiments are performed on three distinct CNN architectures with three different crops of the same dataset. Results show that by appropriately localizing the facial region and selecting the correct CNN architecture it is possible to boost the recognition rate from 84% to 98% while decreasing the training time for proposed CNN architectures

Intelligent facial emotion recognition using moth-firefly optimization

In this research, we propose a facial expression recognition system with a variant of evolutionary firefly algorithm for feature optimization. First of all, a modified Local Binary Pattern descriptor is proposed to produce an initial discriminative face representation. A variant of the firefly algorithm is proposed to perform feature optimization. The proposed evolutionary firefly algorithm exploits the spiral search behaviour of moths and attractiveness search actions of fireflies to mitigate premature convergence of the Levy-flight firefly algorithm (LFA) and the moth-flame optimization (MFO) algorithm. Specifically, it employs the logarithmic spiral search capability of the moths to increase local exploitation of the fireflies, whereas in comparison with the flames in MFO, the fireflies not only represent the best solutions identified by the moths but also act as the search agents guided by the attractiveness function to increase global exploration. Simulated Annealing embedded with Levy flights is also used to increase exploitation of the most promising solution. Diverse single and ensemble classifiers are implemented for the recognition of seven expressions. Evaluated with frontal-view images extracted from CK+, JAFFE, and MMI, and 45-degree multi-view and 90-degree side-view images from BU-3DFE and MMI, respectively, our system achieves a superior performance, and outperforms other state-of-the-art feature optimization methods and related facial expression recognition models by a significant margin

Video copy-move forgery detection scheme based on displacement paths

Sophisticated digital video editing tools has made it easier to tamper real videos and create perceptually indistinguishable fake ones. Even worse, some post-processing effects, which include object insertion and deletion in order to mimic or hide a specific event in the video frames, are also prevalent. Many attempts have been made to detect such as video copy-move forgery to date; however, the accuracy rates are still inadequate and rooms for improvement are wide-open and its effectiveness is confined to the detection of frame tampering and not localization of the tampered regions. Thus, a new detection scheme was developed to detect forgery and improve accuracy. The scheme involves seven main steps. First, it converts the red, green and blue (RGB) video into greyscale frames and treats them as images. Second, it partitions each frame into non-overlapping blocks of sized 8x8 pixels each. Third, for each two successive frames (S2F), it tracks every block’s duplicate using the proposed two-tier detection technique involving Diamond search and Slantlet transform to locate the duplicated blocks. Fourth, for each pair of the duplicated blocks of the S2F, it calculates a displacement using optical flow concept. Fifth, based on the displacement values and empirically calculated threshold, the scheme detects existence of any deleted objects found in the frames. Once completed, it then extracts the moving object using the same threshold-based approach. Sixth, a frame-by-frame displacement tracking is performed to trace the object movement and find a displacement path of the moving object. The process is repeated for another group of frames to find the next displacement path of the second moving object until all the frames are exhausted. Finally, the displacement paths are compared between each other using Dynamic Time Warping (DTW) matching algorithm to detect the cloning object. If any pair of the displacement paths are perfectly matched then a clone is found. To validate the process, a series of experiments based on datasets from Surrey University Library for Forensic Analysis (SULFA) and Video Tampering Dataset (VTD) were performed to gauge the performance of the proposed scheme. The experimental results of the detection scheme were very encouraging with an accuracy rate of 96.86%, which markedly outperformed the state-of-the-art methods by as much as 3.14%

Reconnaissance des expressions faciales pour l’assistance ambiante

Au cours de ces dernières décennies, le monde a connu d’importants changements démographiques et notamment au niveau de la population âgée qui a fortement augmenté. La prise d’âge a comme conséquence directe non seulement une perte progressive des facultés cognitives, mais aussi un risque plus élevé d’être atteint de maladies neurodégénératives telles qu’Alzheimer et Parkinson. La perte des facultés cognitives cause une diminution de l’autonomie et par conséquent, une assistance quotidienne doit être fournie à ces individus afin d’assurer leur bien-être. Les établissements ainsi que le personnel spécialisé censés les prendre en charge représentent un lourd fardeau pour l’économie. Pour cette raison, d’autres solutions moins coûteuses et plus optimisées doivent être proposées. Avec l’avènement des nouvelles technologies de l’information et de la communication, il est devenu de plus en plus aisé de développer des solutions permettant de fournir une assistance adéquate aux personnes souffrant de déficiences cognitives. Les maisons intelligentes représentent l’une des solutions les plus répandues. Elles exploitent différents types de capteurs pour la collecte de données, des algorithmes et méthodes d’apprentissage automatique pour l’extraction/traitement de l’information et des actionneurs pour le déclenchement d’une réponse fournissant une assistance adéquate. Parmi les différentes sources de données qui sont exploitées, les images/vidéos restent les plus riches en termes de quantité. Les données récoltées permettent non seulement la reconnaissance d’activités, mais aussi la détection d’erreur durant l’exécution de tâches/activités de la vie quotidienne. La reconnaissance automatique des émotions trouve de nombreuses applications dans notre vie quotidienne telles que l’interaction homme-machine, l’éducation, la sécurité, le divertissement, la vision robotique et l’assistance ambiante. Cependant, les émotions restent un sujet assez complexe à cerner et de nombreuses études en psychologie et sciences cognitives continuent d’être effectuées. Les résultats obtenus servent de base afin de développer des approches plus efficaces. Les émotions humaines peuvent être perçues à travers différentes modalités telle que la voix, la posture, la gestuelle et les expressions faciales. En se basant sur les travaux de Mehrabian, les expressions faciales représentent la modalité la plus pertinente pour la reconnaissance automatique des émotions. Ainsi, l’un des objectifs de ce travail de recherche consistera à proposer des méthodes permettant l’identification des six émotions de base à savoir : la joie, la peur, la colère, la surprise, le dégoût et la tristesse. Les méthodes proposées exploitent des données d’entrée statiques et dynamiques, elles se basent aussi sur différents types de descripteurs/représentations (géométrique, apparence et hybride). Après avoir évalué les performances des méthodes proposées avec des bases de données benchmark à savoir : JAFFE, KDEF, RaFD, CK+, MMI et MUG. L’objectif principal de ce travail de recherche réside dans l’utilisation des expressions faciales afin d’améliorer les performances des systèmes d’assistance existants. Ainsi, des expérimentations ont été conduites au sein de l’environnement intelligent LIARA afin de collecter des données de validation, et ce, en suivant un protocole d’expérimentation spécifique. Lors de l’exécution d’une tâche de la vie quotidienne (préparation du café), deux types de données ont été récoltés. Les données RFID ont permis de valider la méthode de reconnaissance automatique des actions utilisateurs ainsi que la détection automatique d’erreurs. Quant aux données faciales, elles ont permis d’évaluer la contribution des expressions faciales afin d’améliorer les performances du système d’assistance en termes de détection d’erreurs. Avec une réduction du taux de fausses détections dépassant les 20%, l’objectif fixé a été atteint avec succè