Eigenface algorithm-based facial expression recognition in conversations - an experimental study

Recognising facial expressions is important in many fields such as computer-human interface. Though different approaches have been widely used in facial expression recognition systems, there are still many problems in practice to achieve the best implementation outcomes. Most systems are tested via the lab-based facial expressions, which may be unnatural. Particularly many systems have problems when they are used for recognising the facial expressions being used during conversation. This paper mainly conducts an experi-mental study on Eigenface algorithm-based facial expression recognition. It primarily aims to investigate the performance of both lab-based facial expressions and facial expressions used during conversation. The experiment also aims to probe the problems arising from the recognition of facial expression in conversations. The study is carried out using both the author’s facial expression as the basis for the lab-based expressions and the facial expression from one elderly person during conversation. The experiment showed a good result in lab-based facial expressions, but there are some issues observed when using the case of facial expressions obtained in conversation. By analysing the experimental results, future research focus has been highlighted as the investigation of how to recognise special emotions such as a wry smile and how to deal with the interferences in the lower part of face when speaking

Improving the accuracy of automatic facial expression recognition in speaking subjects with deep learning

When automatic facial expression recognition is applied to video sequences of speaking subjects, the recognition accuracy has been noted to be lower than with video sequences of still subjects. This effect known as the speaking effect arises during spontaneous conversations, and along with the affective expressions the speech articulation process influences facial configurations. In this work we question whether, aside from facial features, other cues relating to the articulation process would increase emotion recognition accuracy when added in input to a deep neural network model. We develop two neural networks that classify facial expressions in speaking subjects from the RAVDESS dataset, a spatio-temporal CNN and a GRU cell RNN. They are first trained on facial features only, and afterwards both on facial features and articulation related cues extracted from a model trained for lip reading, while varying the number of consecutive frames provided in input as well. We show that using DNNs the addition of features related to articulation increases classification accuracy up to 12%, the increase being greater with more consecutive frames provided in input to the model