Towards the improvement of self-service systems via emotional virtual agents

Affective computing and emotional agents have been found to have a positive effect on human-computer interactions. In order to develop an acceptable emotional agent for use in a self-service interaction, two stages of research were identified and carried out; the first to determine which facial expressions are present in such an interaction and the second to determine which emotional agent behaviours are perceived as appropriate during a problematic self-service shopping task. In the first stage, facial expressions associated with negative affect were found to occur during self-service shopping interactions, indicating that facial expression detection is suitable for detecting negative affective states during self-service interactions. In the second stage, user perceptions of the emotional facial expressions displayed by an emotional agent during a problematic self-service interaction were gathered. Overall, the expression of disgust was found to be perceived as inappropriate while emotionally neutral behaviour was perceived as appropriate, however gender differences suggested that females perceived surprise as inappropriate. Results suggest that agents should change their behaviour and appearance based on user characteristics such as gender

Recognition of Facial Expressions by Cortical Multi-scale Line and Edge Coding

Face-to-face communications between humans involve emotions, which often are unconsciously conveyed by facial expressions and body gestures. Intelligent human-machine interfaces, for example in cognitive robotics, need to recognize emotions. This paper addresses facial expressions and their neural correlates on the basis of a model of the visual cortex: the multi-scale line and edge coding. The recognition model links the cortical representation with Paul Ekman's Action Units which are related to the different facial muscles. The model applies a top-down categorization with trends and magnitudes of displacements of the mouth and eyebrows based on expected displacements relative to a neutral expression. The happy vs. not-happy categorization yielded a. correct recognition rate of 91%, whereas final recognition of the six expressions happy, anger, disgust, fear, sadness and surprise resulted in a. rate of 78%

Recognition of facial expressions by cortical multi-scale line and edge coding

Empirical studies concerning face recognition suggest that faces may be stored in memory by a few canonical representations. Models of visual perception are based on image representations in cortical area V1 and beyond, which contain many cell layers for feature extraction. Simple, complex and end-stopped cells provide input for line, edge and keypoint detection. Detected events provide a rich, multi-scale object representation, and this representation can be stored in memory in order to identify objects. In this paper, the above context is applied to face recognition. The multi-scale line/edge representation is explored in conjunction with keypoint-based saliency maps for Focus-of-Attention. Recognition rates of up to 96% were achieved by combining frontal and 3/4 views, and recognition was quite robust against partial occlusions

A temporal latent topic model for facial expression recognition

Posters: no. 128LNCS v. 6495 is conference proceedings of the 10th Asian Conference on Computer Vision, Queens, ACCVIn this paper we extend the latent Dirichlet allocation (LDA) topic model to model facial expression dynamics. Our topic model integrates the temporal information of image sequences through redefining topic generation probability without involving new latent variables or increasing inference difficulties. A collapsed Gibbs sampler is derived for batch learning with labeled training dataset and an efficient learning method for testing data is also discussed. We describe the resulting temporal latent topic model (TLTM) in detail and show how it can be applied to facial expression recognition. Experiments on CMU expression database illustrate that the proposed TLTM is very efficient in facial expression recognition. © 2011 Springer-Verlag Berlin Heidelberg.postprintThe 10th Asian Conference on Computer Vision (ACCV 2010), Queenstown, New Zealand, 8-12 November 2010. In Lecture Notes in Computer Science, 2010, v. 6495, p. 51-6

Analysis of Range Images Used in 3D Facial Expression Recognition Systems

With the creation of BU-3DFE database the research on 3D facial expression recognition has been fostered; however, it is limited by the development of 3D algorithms. Range image is the strategy for solving the problems of 3D recognition based on 2D algorithms. Recently, there are some methods to capture range images, but they are always combined with the preprocess, registration, etc. stages, so it is hard to tell which of these generated range images is of higher quality. This paper introduces two kinds of range images and selects different kinds of features based on different levels of expressions to validate the performances of proposed range images; two other kinds of range images based on previously used nose tip detection methods are applied to compare the quality of generated range images; and finally some recently published works on 3D facial expression recognition are listed for comparison. With the experimental results, we can see that the performances of two proposed range images with different kinds of features are all higher than 88 % which is remarkable compared with the most recently published methods for 3D facial expression recognition; the analysis of the different kinds of facial expressions shows that the proposed range images do not lose primary discriminative information for recognition; the performances of range images using different kinds of nose tip detection methods are almost the same what means that the nose tip detection is not decisive to the quality of range images; moreover, the proposed range images can be captured without any manual intervention what is eagerly required in safety systems

Real time facial expression recognition App development on mobile phones

Facial expression has made significant progress in recent years with many commercial systems are available for real-world applications. It gains strong interest to implement a facial expression system on a portable device such as tablet and smart phone device using the camera already integrated in the devices. It is very common to see face recognition phone unlocking app in new smart phones which are proven to be hassle free way to unlock a phone. Implementation a facial expression system in a smart phone would provide fun applications that can be used to measure the mood of the user in their daily life or as a tool for their daily monitoring of the motion in phycology studies. However, traditional facial expression algorithms are normally computing extensive and can only be implemented offline at a computer. In this paper, a novel automatic system has been proposed to recognize emotions from face images on a smart phone in real-time. In our system, the camera of the smart phone is used to capture the face image, BRIEF features are extracted and k-nearest neighbor algorithm is implemented for the classification. The experimental results demonstrate that the proposed facial expression recognition on mobile phone is successful and it gives up to 89.5% recognition accuracy.The work of Hongying Meng was supported by Brunel Research Initiative & Enterprise Fund on the project entitled “Automatic Emotional State Detection and Analysis on Embedded Devices”. This research is also partially supported by the 973 project on Network Big Data Analytics funded by the Ministry of Science and Technology, China. No. 2014CB340404

Pose-invariant facial expression recognition using variable-intensity templates

Abstract. In this paper, we propose a method for pose-invariant facial expression recognition from monocular video sequences. The advantage of our method is that, unlike existing methods, our method uses a very simple model, called the variable-intensity template, for describing different facial expressions, making it possible to prepare a model for each person with very little time and effort. Variable-intensity templates describe how the intensity of multiple points defined in the vicinity of facial parts varies for different facial expressions. By using this model in the framework of a particle filter, our method is capable of estimating facial poses and expressions simultaneously. Experiments demonstrate the effectiveness of our method. A recognition rate of over 90 % was achieved for horizontal facial orientations on a range of ±40 degrees from the frontal view.

Automatic Facial Expression Recognition For Arbitrary Head Pose

Výraz tváře je přirozeným prostředkem při mezilidské komunikaci. Automatické rozpoznání výrazu tváře je proto důležité pro vývoj systému komunikace mezi člověkem a počítačem. Stávající systémy pro rozpoznávání výrazu tváře pracují převážně s daty pořízenými v kontrolovaném prostředí. Osoba je vyzvána k provedení výrazu tváře s omezeným pohybem hlavy v prostoru. Natočení tváře mimo čelní pohled je tedy minimální. Systémy navržené pro tento typ dat se nedokáží vyrovnat s reálnými daty obsahující libovolné natočení tváře. Jejich úspěšnost se výrazně snižuje s natočením tváře mimo čelní pohled. Disertační práce se zaměřuje na řešení problému libovolného natočení tváře v prostoru. Nově navržená metoda pro předzpracování vstupních dat je schopná provést 3D rekonstrukci objektu tváře s libovolným výrazem z jediného 2D snímku. Klasifikátoru jsou předkládány snímky v čelním natočením tváře. V této práci byl navržen a experimentálně ověřen systém schopný rozpoznat výraz tváře pro sklon hlavy 30 stupňů a otočením do stran 45 stupňů. Vytvořený systém byl experimentálně ověřen na datech ze soutěže FERA 2011 (First Facial Expression Analysis Challange). Data z FERA 2011 se nejvíce přibližují reálným datům, obsahují libovolné natočení tváře v rozsahu 30° ve všech směrech od čelního pohledu. Pro rozpoznávání kategorických emocí dosáhl základní systém 63% a systém s 3D rekonstrukcí 73%. Nově navržená metoda pro normalizaci tváře na čelní pohled s využitím 3D rekonstrukce z jediného 2D snímku nabízí alternativní řešení ke klasickým metodám pro pořízení 3D dat.Katedra kybernetikyObhájenoFacial expressions are a vital part of human communication. Automatic facial expression recognition is essential while designing system for human computer interaction. Existing systems for expression recognition works in controlled environment. Subject is asked to perform an expression while his head is looking straight into camera. Other views than the frontal view are absent. System designed for such data are unable to work with real data containing arbitrary pose. This work is focused on the problem of facial expression recognition with arbitrary view. A new method able to perform 3D reconstruction of facial expression with arbitrary view from single 2D image has been developed. The method is able to work with face with tilt view 30° and side view 45°. Designed system was experimentally tested on data from The First Facial Expression Recognition Challenge (FERA 2011). This data are very close to real one. Data contains various head pose that vary 30° from frontal view. For the task of emotion recognition the basic system obtain recognition rate of 63%, and system with 3D reconstruction obtained 73%. The new method for face normalization to frontal view using the 3D reconstruction of face from single 2D image offers an alternative way to classical method for acquisition of 3D data

Videobasierte Verfahren zur Schätzung des Interaktionsinteresses bei der Mensch-Roboter-Interaktion mittels Analyse durch Synthese

To realize the operation of mobile service robots in everyday life, it isnecessary to develop intelligent and adaptive dialog systems. Such dialogsystems must be designed in a way that allows an easy and intuitive operationeven for untrained users.For that purpose, it is necessary to detect the mood and intentions of a user.In this thesis, methods for the detection and estimation of the attentionand/or interaction interest of a user of a mobile service robot will bedeveloped and presented. For this purpose, three subsystems are presented: the estimation of theorientation of the upper body, the estimation of the head pose, and theanalysis of the facial expression of a user.Each subsystem is realized by using an Analysis by Synthesis approach.More precisely, Active Shape Models and Active Appearance Modelsare utilized within the three subsystems.Furthermore, different classification and function approximation systems willbe applied to estimate the different features. For that, different methodslike linear regression, Multi Layer Perceptrons, Support Vector Machines,and Self-organizing Maps will be compared.This thesis shows that it is possible to estimate the requested featuresin a sufficient quality and robustness by using the proposed subsystems.Hence it is possible, to estimate the attention and interaction interestby using the upper body orientation, the head pose and the facial expression.Each subsystem was tested with different data sets. Besides own data basesalso foreign data sets were utilized to show the robustness and to measurethe detection rates of the proposed methods.Additionally, this thesis shows, that a selection of the relevant modelparameters leads to better results or at least to equal results, whichcan be achieved by easier classifiers. For this parameter selection theMutal Information is applied in this thesis. Furthermore, an overall system, which integrates the results of the differentsubsystems, is presented in this thesis. The fusion of the results isrealized by using methods from the domain of probabilistic robotics.Based on some easy experiments (performed by briefed subjects) it is shown,that all subsystems can deliver feasible results, which can be integratedin an overall estimation of the attention and/or interaction interest ofa user. Thus, the work presented in this thesis can be used for furthersocioscientific experiments, which are not part of this thesis.Um den Einsatz von mobilen Servicerobotern im Alltag zu realisieren, istes notwendig, intelligente und adaptive Dialogsysteme zu entwickeln, die es auch einem nicht-eingewiesenen Benutzer erlauben, einen Serviceroboter intuitiv bedienen und nutzen zu können. Dazu ist es erforderlich, die Stimmung und den Gemütszustand des Benutzers zu erfassen, um entsprechend darauf reagieren zu können. Im Rahmen dieser Dissertation werden Methoden entwickelt und vorgestellt, die als Indikatoren zur Schätzung des Interaktionsinteresses (bzw. der Aufmerksamkeit) eines Benutzers auf einem mobilen Serviceroboter unter Realweltbedingungen verwendet werden können.Hierfür werden drei Teilsysteme präsentiert, die die Orientierung des Oberkörpers, die Blickrichtung und die Mimik des Benutzers schätzen können. Alle drei Teilsysteme werden mittels Analysis-by-Synthesis Verfahren realisiert. Dabei kommen Active Shape Models und Active Appearance Modelszum Einsatz. Zur anschließenden Klassifikation bzw. Schätzung der gesuchten Merkmale werden u.a. Verfahren der linearen Regression, Multi Layer Perceptrons, Support Vector Machines und Self-organizing Maps miteinander verglichen. Es wird gezeigt, dass es mit den drei Teilsystemen möglich ist, die gesuchten Informationen zu bestimmen und damit Indizien für Interesse und Aufmerksamkeit gewonnen werden können. Die Tests wurden dabei jeweils mit bekanntem und unbekanntem Datenmaterial durchgeführt. Zusätzlich wird gezeigt, dass eine Vorauswahl relevanter Parameter auf Basis der Mutual Information zu besseren Ergebnissen führt bzw. gleich gute Ergebnisse mittels einfacherer Klassifikatoren erreicht werden können. Weiterhin wird ein Gesamtsystem vorgestellt, in dem alle drei Teilsysteme miteinander kombiniert werden. Zur Schätzung von Interesse und Aufmerksamkeit kommen dabei Methoden aus der probabilistischen Robotik zum Einsatz. Anhand durchgeführter Experimente mit eingewiesenen Probanden wird gezeigt, dass die Ergebnisse der drei Teilmodule plausibel sind und die Resultate zur Schätzung von Interesse und Aufmerksamkeit verwendet werden können. Das prototypische Gesamtsystem kann daher als Grundlage und Basis fürzukünftige sozialwissenschaftliche Untersuchungen zur Bestimmung des Interaktionsinteresses genutzt werden, die nicht Bestandteil dieser Dissertation sind