A novel method based on shape and texture information is proposed in this paper for facial expression recognition from video sequences. The Discriminant Non-negative Matrix Factorization (DNMF) algorithm is applied at the image corresponding to the greatest intensity of the facial expression (last frame of the video sequence), extracting that way the texture information. A Support Vector Machines (SVMs) system is used for the classi cation of the shape information derived from tracking the Candide grid over the video sequence. The shape information consists of the di erences of the node coordinates between the  rst (neutral) and last (fully expressed facial expression) video frame. Subsequently, fusion of texture and shape information obtained is performed using Radial Basis Function (RBF) Neural Networks (NNs). The accuracy achieved is equal to 98,2% when recognizing the six basic facial expressionsIFIP International Conference on Artificial Intelligence in Theory and Practice - Machine VisionRed de Universidades con Carreras en Informática (RedUNCI

Kotsia, Irene

Pitas, Ioannis

English

El Servicio de Difusión de la Creación Intelectual

1Facial expression recognition using shape and
texture information
I. Kotsia1 and I. Pitas1
Aristotle University of Thessaloniki
pitas@aiia.csd.auth.gr
Department of Informatics
Box 451 54124
Thessaloniki, Greece
Summary. A novel method based on shape and texture information is proposed in
this paper for facial expression recognition from video sequences. The Discriminant
Non-negative Matrix Factorization (DNMF) algorithm is applied at the image cor-
responding to the greatest intensity of the facial expression (last frame of the video
sequence), extracting that way the texture information. A Support Vector Machines
(SVMs) system is used for the classification of the shape information derived from
tracking the Candide grid over the video sequence. The shape information consists
of the differences of the node coordinates between the first (neutral) and last (fully
expressed facial expression) video frame. Subsequently, fusion of texture and shape
information obtained is performed using Radial Basis Function (RBF) Neural Net-
works (NNs). The accuracy achieved is equal to 98,2% when recognizing the six
basic facial expressions.
1.1 Introduction
During the past two decades, many studies regarding facial expression recog-
nition, which plays a vital role in human centered interfaces, have been
conducted. Psychologists have defined the following basic facial expressions:
anger, disgust, fear, happiness, sadness and surprise [?]. A set of muscle move-
ments, known as Action Units, was created. These movements form the so
called Facial Action Coding System (FACS) [?]. A survey on auto-
matic facial expression recognition can be found in [?].
In the current paper, a novel method for video based facial expression
recognition by fusing texture and shape information is proposed. The texture
information is obtained by applying the DNMF algorithm [?] on the last
frame of the video sequence, i.e. the one that corresponds to the greatest
intensity of the facial expression depicted. The shape information is calculated
as the difference of Candide facial model grid node coordinates between the
first and the last frame of a video sequence [?]. The decision made regarding
the class the sample belongs to, is obtained using a SVM system. Both the
DNMF and SVM algorithms have as an output the distances of the sample
under examination from each of the six classes (facial expressions). Fusion
of the distances obtained from DNMF and SVMs applications is attempted
using a RBF NN system. The experiments performed using the Cohn-Kanade
database indicate a recognition accuracy of 98,2% when recognizing the six
basic facial expressions. The novelty of this method lies in the combination of
both texture and geometrical information for facial expression recognition.
1.2 System description
The diagram of the proposed system is shown in Figure ??.
distance r
6 class SVM
system
Grid node
displacements
gj
grid
First frame
Last  frame
Grid tracking
Deformed
grid
Deformed
grid
-
DNMF
Original
images
Basis
images
distance h
Fusion
SVMs
Final classified
facial expression
cj
Facial texture information
Geometrical displacement information
Fig. 1.1. System architecture for facial expression recognition in facial videos
The system is composed of three subsystems: two responsible for texture
and shape information extraction and a third one responsible for the fusion
of their results. Figure ?? shows the two sources of information (texture and
shape) used by the system.
1.3 Texture information extraction
Let U be a database of facial videos. The facial expression depicted in each
video sequence is dynamic, evolving through time as the video progresses. We
Texture Shape
Recognized facial expression
Fig. 1.2. Fusion of texture and shape.
take under consideration the frame that depicts the facial expression in its
greatest intensity, i.e. the last frame, to create a facial image database Y .
Thus, Y consists of images where the depicted facial expression obtains its
greatest intensity . Each image y ∈ Y belongs to one of the 6 basic facial
expression classes{Y1,Y2, . . . ,Y6} with Y =
⋃6
r=1 Yr. Each image y ∈ <K×G+
of dimension F = K ×G forms a vector x ∈ <F+. The vectors x ∈ <F+ will be
used in our algorithm.
The algorithm used was the DNMF algorithm, which is a extension of the
Non-negative Matrix Factorization (NMF) algorithm. The NMF algorithm al-
gorithm is an object decomposition algorithm that allows only additive com-
binations of non negative components. DNMF was the result of an attempt
to introduce discriminant information to the NMF decomposition. Both NMF
and DNMF algorithms will be presented analytically below.
1.3.1 The Non-negative Matrix Factorization Algorithm
A facial image xj after the NMF decomposition can be written as xj ≈ Zhj ,
where hj is the j-th column of H. Thus, the columns of the matrix Z can
be considered as basis images and the vector hj as the corresponding weight
vectors. Vectors hj can also be considered as the projections vectors of the
original facial vectors xj on a lower dimensional feature space .
In order to apply NMF in the database Y, the matrix X ∈ <F×G+ = [xi,j ]
should be constructed, where xi,j is the i-th element of the j-th image, F is
the number of pixels and G is the number of images in the database. In other
words the j-th column of X is the xj facial image in vector form (i.e. xj ∈ <F+).
NMF aims at finding two matrices Z ∈ <F×M+ = [zi,k] and H ∈ <M×L+ = [hk,j ]
such that :
X ≈ ZH. (1.1)
where M is the number of dimensions taken under consideration (usually
M  F ).
The NMF factorization is the outcome of the following optimization prob-
lem :
min
Z,H
DN (X||ZH) subject to (1.2)
zi,k ≥ 0, hk,j ≥ 0,
∑
i
zi,j = 1, ∀j.
The update rules for the weight matrix H and the bases matrix Z can be
found in [?].
1.3.2 The Discriminant Non-negative Matrix Factorization
Algorithm
In order to incorporate discriminants constraints inside the NMF cost function
(??), we should use the information regarding the separation of the vectors
hj into different classes. Let us assume that the vector hj that corresponds
to the jth column of the matrix H, is the coefficient vector for the ρth facial
image of the rth class and will be denoted as η
(r)
ρ = [η
(r)
ρ,1 . . . η
(r)
ρ,M ]
T . The mean
vector of the vectors η
(r)
ρ for the class r is denoted as µ(r) = [µ
(r)
1 . . . µ
(r)
M ]
T
and the mean of all classes as µ = [µ1 . . . µM ]
T . The cardinality of a facial
class Yr is denoted by Nr. Then, the within scatter matrix for the coefficient
vectors hj is defined as:
Sw =
6∑
r=1
Nr∑
ρ=1
(η(r)ρ − µ(r))(η(r)ρ − µ(r))T (1.3)
whereas the between scatter matrix is defined as:
Sb =
6∑
r=1
Nr(µ
(r) − µ)(µ(r) − µ)T . (1.4)
The discriminant constraints are incorporated by requiring tr[Sw] to be as
small as possible while tr[Sb] is required to be as large as possible.
Dd(X||ZDH) = DN (X||ZDH) + γtr[Sw]− δtr[Sb]. (1.5)
where γ and δ are constants and D is the measure of the cost for factoring X
into ZH [?].
Following the same Expectation Maximization (EM) approach used by
NMF techniques [?], the following update rules for the weight coefficients hk,j
that belong to the r-th facial class become:
h
(t)
k,j =
T1 +
√
T 21 + 4(2γ − (2γ + 2δ) 1Nr )h
(t−1)
k,j
2(2γ − (2γ + 2δ) 1
Nr
)∑
i z
(t−1)
i,k
xi,j∑
l z
(t−1)
i,l
h
(t−1)
l,j
2(2γ − (2γ + 2δ) 1
Nr
)
. (1.6)
where T1 is given by:
T1 = (2γ + 2δ)(
1
Nr
∑
λ,λ6=l
hk,λ)− 2δµk − 1. (1.7)
The update rules for the bases ZD, are given by:
z´
(t)
i,k = z
(t−1)
i,k
∑
j h
(t)
k,j
xi,j∑
l z
(t−1)
i,l h
(t)
l,j∑
j h
(t)
k,j
(1.8)
and
z
(t)
i,k =
z´
(t)
i,k∑
l z´
(t)
l,k
. (1.9)
The above decomposition is a supervised non-negative matrix factorization
method that decomposes the facial images into parts while, enhancing the class
separability. The matrix Z†D = (Z
T
DZD)
−1ZTD, which is the pseudo-inverse
of ZD, is then used for extracting the discriminant features as x´ = Z
†
Dx.
The most interesting property of DNMF algorithm is that it decomposes the
image to facial areas, i.e. mouth, eyebrows, eyes, and focuses on extracting
the information hiding in them. Thus, the new representation of the image is
a better one compared to the one acquired when the whole image was taken
under consideration.
For testing, the facial image xj is projected on the low dimensional feature
space produced by the application of the DNMF algorithm:
x´j = Z
†
Dxj (1.10)
For the projection of the facial image x´j , one distance from each center class
is calculated. The smallest distance defined as:
rj = min
k=1,...,6
‖x´j − µ(r)‖ (1.11)
is the one that is taken as the output of the DNMF system.
1.4 Shape information extraction
The geometrical information extraction is done by a grid tracking system,
based on deformable models [?]. The tracking is performed using a pyramidal
implementation of the well-known Kanade-Lucas-Tomasi (KLT) algorithm.
The user has to place manually a number of Candide grid nodes on the corre-
sponding positions of the face depicted at the first frame of the image sequence.
The algorithm automatically adjusts the grid to the face and then tracks it
through the image sequence, as it evolves through time. At the end, the grid
tracking algorithm produces the deformed Candide grid that corresponds to
the last frame i.e. the one that depicts the greatest intensity of the facial
expression.
The shape information used from the j video sequence is the displace-
ments dij of the nodes of the Candide grid, defined as the difference between
coordinates of this node in the first and last frame [?]:
dij = [∆x
i
j∆y
i
j ]
T i ∈ {1, . . . ,K} and j ∈ {1, . . . , N} (1.12)
where i is an index that refers to the node under consideration. In our case
K = 104 nodes were used.
For every facial video in the training set, a feature vector gj of F = 2·104 =
208 dimensions, containing the geometrical displacements of all grid nodes is
created:
gj = [d
1
j d
2
j . . . d
K
j ]
T . (1.13)
Let U be the video database that contains the facial videos, that are clus-
tered into 6 different classes Uk, k = 1, . . . , 6, each one representing one of 6
basic facial expressions. The feature vectors gj ∈ <F labelled properly with
the true corresponding facial expression are used as an input to a multi class
SVM and will be described in the following section.
1.4.1 Support Vector Machines
Consider the training data:
(g1, l1), . . . , (gN , lN ) (1.14)
where gj ∈ <F j = 1, . . . , N are the deformation feature vectors and
lj ∈ {1, . . . , 6} j = 1, . . . , N are the facial expression labels of the feature vec-
tor. The approach implemented for multiclass problems used for direct facial
expression recognition is the one described in [?] that solves only one opti-
mization problem for each class (facial expression). This approach constructs
6 two-class rules where the k−th function wTk φ(gj) + bk separates training
vectors of the class k from the rest of the vectors. Here φ is the function that
maps the deformation vectors to a higher dimensional space (where the data
are supposed to be linearly or near linearly separable) and b = [b1 . . . b6]
T is
the bias vector. Hence, there are 6 decision functions, all obtained by solving
a different SVM problem for each class. The formulation is as follows:
min
w,b,ξ
1
2
6∑
k=1
wTk wk + C
N∑
j=1
∑
k 6=lj
ξkj (1.15)
subject to the constraints:
wTljφ(gj) + blj ≥ wTk φ(gj) + bk + 2− ξkj (1.16)
ξkj ≥ 0, j = 1, . . . , N, k ∈ {1, . . . , 6}\lj .
where C is the penalty parameter for non linear separability and
ξ = [. . . , ξmi , . . .]
T is the slack variable vector. Then, the function used to
calculate the distance of a sample from each center class is defined as:
s(g) = max
k=1,...,6
(wTk φ(g) + bk). (1.17)
That distance was considered as the output of the SVM based shape extraction
procedure. A linear kernel was used for the SVM system in order to avoid
search for appropriate kernels.
1.5 Fusion of texture and shape information
The application of the DNMF algorithm on the images of the database re-
sulted in the extraction of the texture information of the facial expressions
depicted. Similarly, the classification procedure performed using the SVM sys-
tem on the grid following the facial expression through time resulted in the
extraction of the shape information .
More specifically, the image xj and the corresponding vector of geomet-
rical displacements gj were taken into consideration. The DNMF algorithm,
applied to the xj image, produces the distance rj as a result, while SVMs
applied to the vector of geometrical displacements gj , produces the distance
sj as the equivalent result. The distances rj and sj were normalized in [0, 1]
using Gaussian normalization. Thus, a new feature vector cj , defined as:
cj = [rj sj ]
T . (1.18)
containing information from both sources was created.
1.5.1 Radial Basis Function (RBF) Neural Networks (NNs)
A RBF NN was used for the fusion of texture and shape results. The RBF
function is approximated as a linear combination of a set of basis functions
[?]:
pk(cj) =
M∑
n=1
wk,nφn(cj) (1.19)
where M is the number of kernel functions and wk,n are the weights of the
hidden unit to output connection. Each hidden unit implements a Gaussian
function:
φn(cj) = exp[−(mn − cj)TΣ−1n (mn − cj)] (1.20)
where j = 1, . . .M , mn is the mean vector and Σn is the covariance matrix
[?].
Each pattern cj is considered assigned only to one class lj . The decision
regarding the class lj of cj is taken as:
lj = argmax
k=1,...,6
pk(cj) (1.21)
The feature vector cj was used as an input to the RBF NN that was
created. The output of that system was the label lj that classified the sample
under examination (pair of texture and shape information) to one of the 6
classes (facial expressions).
1.6 Experimental results
In order to create the training set, the last frames of the video sequences
used were extracted. By doing so, two databases were created, one for texture
extraction using DNMF and another one for shape extraction using SVMs.
The texture database consisted of images that corresponded to the last frame
of every video sequence studied, while the shape database consisted of the
grid displacements that were noticed between the first and the last frame of
every video sequence.
The databases were created using a subset of the Cohn-Kanade database
that consists of 222 image sequences, 37 samples per facial expression. The
leave-one-out method was used for the experiments [?]. For the implementa-
tion of the RBF NN, 25 neurons were used for the output layer and 35 for the
hidden layer.
The accuracy achieved when only DNMF was applied was equal to 86,5%,
while the equivalent one when SVMs along with shape information were used
was 93,5%. The obtained accuracy after performing fusion of the two informa-
tion sources was equal to 98,2%. By fusing texture information into the shape
results certain confusions are resolved. For example, some facial expressions
involve subtle facial movements. That results in confusion with other facial
expressions when only shape information is used. By introducing texture in-
formation, those confusions are eliminated. For example, in the case of anger,
a subtle eyebrow movement is involved which cannot probably be identified
as movement, but would most probably be noticed if texture is available.
Therefore, the fusion of shape and texture information results in correctly
classifying most of the confused cases, thus increasing the accuracy rate.
The confusion matrix [?] has been also computed. It is a n×n matrix con-
taining information about the actual class label lj , j = 1, .., n (in its columns)
and the label obtained through classification oj , j = 1, .., n (in its rows). The
diagonal entries of the confusion matrix are the numbers of facial expressions
that are correctly classified, while the off-diagonal entries correspond to mis-
classification. The confusions matrices obtained when using DNMF on texture
information, SVM on shape information and when the proposed fusion is ap-
plied are presented in Table ??.
Table 1.1. Confusion matrices for DNMF results, SVMs results and fusion results,
respectively.
labcl\labac anger disgust fear happiness sadness surprise
anger 13 0 0 0 0 0
disgust 10 37 0 0 0 0
fear 4 0 37 0 0 1
happiness 2 0 0 37 0 0
sadness 7 0 0 0 37 5
surprise 1 0 0 0 0 31
labcl\labac anger disgust fear happiness sadness surprise
anger 24 0 0 0 0 0
disgust 5 37 0 0 0 0
fear 0 0 37 0 0 1
happiness 0 0 0 37 0 0
sadness 8 0 0 0 37 0
surprise 0 0 0 0 0 36
labcl\labac anger disgust fear happiness sadness surprise
anger 33 0 0 0 0 0
disgust 2 37 0 0 0 0
fear 0 0 37 0 0 0
happiness 0 0 0 37 0 0
sadness 2 0 0 0 37 0
surprise 0 0 0 0 0 37
1.7 Conclusions
A novel method for facial expression recognition is proposed in this paper.
The recognition is performed by fusing the texture and the shape informa-
tion extracted from a video sequence. The DNMF algorithm is applied at the
last frames of every video sequence corresponding to the greatest intensity of
the facial expression, extracting that way the texture information. Simultane-
ously, a SVM system classifies the shape information obtained by tracking the
Candide grid between the first (neutral) and last (fully expressed facial expres-
sion) video frame. The results obtained from the above mentioned methods
are then fused using RNF NNs. The system achieves an accuracy of 98,2%
when recognizing the six basic facial expressions.
1.8 Acknowledgment
This work has been conducted in conjunction with the ”SIMILAR” European
Network of Excellence on Multimodal Interfaces of the IST Programme of the
European Union (www.similar.cc).
References
1. P. Ekman, and W.V. Friesen, “Emotion in the Human Face,” Prentice Hall,
1975.
2. T. Kanade, J. Cohn, and Y. Tian, “Comprehensive Database for Facial Ex-
pression Analysis,” Proceedings of IEEE International Conference on Face and
Gesture Recognition, 2000.
3. B. Fasel, and J. Luettin, “Automatic Facial Expression Analysis: A Survey,”
Pattern Recognition, 2003.
4. S. Zafeiriou, A. Tefas, I. Buciu and I. Pitas, “Exploiting Discriminant Infor-
mation in Non-negative Matrix Factorization with application to Frontal Face
Verification,” IEEE Transactions on Neural Networks, accepted for publication,
2005.
5. D. D. Lee and H. S. Seung, “Algorithms for non-negative matrix factorization,”
Advances in Neural Information Processing Systems, vol. 13pp. 556−562, 2001.
6. I. Kotsia, and I. Pitas, “Real time facial expression recognition from image
sequences using Support Vector Machines,” IEEE International Conference on
Image Processing (ICIP 2005), 11-14 September, 2005.
7. V. Vapnik, “Statistical learning theory,” 1998.
8. A. G. Bors and I. Pitas, “Median Radial Basis Function Neural Network,” IEEE
Transactions on Neural Networks, vol. 7, pp. 1351-1364, November 1996.


Facial expression recognition using shape and texture information

http://sedici.unlp.edu.ar/bitstream/handle/10915/23944/Documento_completo.pdf?sequence=1

Facial expression recognition using shape and texture information

Abstract

Similar works

Full text

Available Versions

El Servicio de Difusión de la Creación Intelectual