Facial feature point tracking is a research area that can be used in human-computer interaction (HCI), facial expression analysis, fatigue detection, etc. In this paper, a statistical method for facial feature point tracking is proposed. Feature point tracking is a challenging topic in case of uncertain

data because of noise and/or occlusions. With this motivation, a graphical model that incorporates not only temporal information about feature point movements, but also information about the spatial relationships between such points is built. Based on this model, an algorithm that achieves feature point tracking through a video observation sequence is implemented. The proposed method is applied on 2D gray scale real video sequences taken in a vehicle environment and the superiority of this approach over existing techniques is demonstrated

Cetin, Mujdat

Cosar, Serhan

Coşar, Serhan

Ercil, Aytul

Erçil, Aytül

Çetin, Müjdat

English

Sabanci University Research Database

GRAPHICAL MODEL BASED FACIAL FEATURE POINT TRACKING IN A VEHICLEENVIRONMENTSerhan Cos¸ar, Mu¨jdat C¸etin, Aytu¨l Erc¸ilSabancı UniversityFaculty of Engineering and Natural SciencesOrhanlı- Tuzla, 34956 I˙stanbul, TURKEYserhancosar@su.sabanciuniv.edu,{mcetin,aytulercil}@sabanciuniv.eduABSTRACTFacial feature point tracking is a research area that can beused in human-computer interaction (HCI), facial expres-sion analysis, fatigue detection, etc. In this paper, a statis-tical method for facial feature point tracking is proposed.Feature point tracking is a challenging topic in case of un-certain data because of noise and/or occlusions. With thismotivation, a graphical model that incorporates not onlytemporal information about feature point movements, butalso information about the spatial relationships between suchpoints is built. Based on this model, an algorithm that achievesfeature point tracking through a video observation sequenceis implemented. The proposed method is applied on 2Dgray scale real video sequences taken in a vehicle environ-ment and the superiority of this approach over existing tech-niques is demonstrated.1. INTRODUCTIONFacial feature point tracking is an important step in prob-lems such as video based facial expression analysis and human-computer interaction. Generally, a facial expression analy-sis system consists of three components: feature detection,feature tracking, and expression recognition. Feature detec-tion involves detecting some distinguishable points that candefine the movement of facial components. This may in-volve, detection of eyes, eye brows, mouth or feature pointsof these components. Then comes the tracking part whichconsists of tracking the detected feature points. Finally, ac-cording to tracking results of these feature points, the ex-pression recognition component outputs results such as happy,sad, surprised, etc. This paper is an application of facialfeature point tracking to video sequences taken in a vehicleenvironment [1]. The output of the proposed work can beconsidered as an input to a facial expression analysis systemin a vehicle for driver fatigue detection and driver behaviormodeling.Such a fatigue detection system in vehicles is very importantbecause statistics show that transporting by vehicle is quiteunsafe compared to the other transporting options. Policereports say that driver faults are the main reason of morethan 80% of the accidents. Cooperating the emerging el-ements of digital life and technology with the recent au-tomobile technology will produce intelligent vehicles thatcan warn the driver automatically in possible accident situ-ations.For feature point tracking, roughly there are two classesof methods in literature: general purpose approaches, face-specific approaches. One of the general purpose approachesis moving-point-correspondence methods [2, 3]. The smoothmotion, limited speed, and no occlusion assumptions makethese methods inapplicable to facial-feature tracking. An-other approach is the patch correlation method [4, 5] whichis sensitive to illumination and object-pose variations. Thereare also some optical-flow based methods [6, 7] which oftenassume image-intensity constancy for corresponding pixels,which may not be the case for facial features.Compared with the general-purpose feature-tracking tech-niques, the face-specific methods are more effective. Thereare methods which use Gabor Filters [8, 9] to track facialfeature points. Also, active appearance (AAM) or activeshape models (ASM) [10, 11, 12] are used to track featurepoints based on a face model. Additionally, the work in [8]tracks feature points based on spatial and temporal connec-tions using non-parametric methods.Generally, feature point tracking is done by using a tempo-ral model that is based on pixel values. Consequently, thesemethods are sensitive to illumination and pose changes, andignore the spatial relationships between feature points. Thisaffects the tracking performance adversely, causes drifts andphysically unreasonable results when the data are noisy oruncertain due to occlusions. In [13], a method where thespatial relationships are taken into account is proposed forcontour tracking. However, since the method is based onnon-parametric estimation techniques, it is rather computa-tionally intensive.In this paper feature point tracking is performed in a frame-work that incorporates the temporal and spatial informa-tion between feature points. This framework is based ongraphical models that have recently been used in many com-puter vision problems. The model is based on a parametricmodel in which the probability densities involved are Gaus-sian. The parametric nature of the models makes the methodcomputationally efficient. The spatial connections betweenpoints allow the tracking to continue reasonably well by ex-ploiting the information from neighboring points, even if apoint disappears from the scene or cannot be observed. Thefeature values from video sequences are based on Gabor fil-ters. Filters are used in a way to detect the edge informationin the image, to be sensitive to different poses, orientationsand different feature sizes. Tests on videos recorded in avehicle environment showed that tracking of facial featurepoints is performed successfully.2. PROPOSED METHOD2.1. PreprocessingGabor filters are used as a preprocessing stage for the obser-vation section of the proposed work. The filters are selectedas in [14]. Then as in [8], consecutive frames are con-volved with 24 filters consisting of 6 different orientationsand 4 different wavelengths. The magnitude and phase ofthe complex outputs of the filtering is compared using thesimilarity metric in [14]. The location of the point in thenext frame is found by comparing the filter outputs from thecurrent and the next frame. This is used as the data compo-nent of the method.2.2. Graphical ModelsGraphical models can be defined as a marriage of graph the-ory and probability theory. The visualization property ofgraph theory makes even a complex model clear and under-standable. This provides a powerful, general framework fordeveloping statistical models of computer vision problems.Generally a graph G is defined by a set of nodes V , and acorresponding set of edges E. The neighborhood of a nodes ∈ V is defined as N(s) = {t|(s, t) ∈ E}. The models aredivided into two main categories: directed and undirectedgraphs. Directed graphs are graphs in which there is a causalrelation between random variables. In undirected graphs therelation is bidirectional. Some examples of graphical mod-els are illustrated in Figure 1.Fig. 1. Some examples of graphical modelsGraphical models usually associate each node s ∈ V withan unobserved, hidden random variable (xs), and a noisy lo-cal observation (ys). Let x = {xs|s ∈ V } and y = {ys|s ∈V } denote the sets of all hidden and observed variables, re-spectively. This simply makes the factorization of a jointprobability function p(x, y) as shown below.p(x, y) =1Z∏(s,t)∈Eψs,t(xs, xt)∏(s)∈Vψs(xs, ys) (1)Here, ψs,t(xs, xt) is the edge potential between hidden vari-ables. The other term, ψs(xs, ys) is the observation poten-tial.The graphical model that is used in the proposed method,when we track two feature points, is shown in Figure 2.Each hidden variable (xs) in the model is a vector with fourelements. Assuming the movement of the feature points arein 2D, these four elements are x-coordinates, y-coordinates,velocity at x-axis and velocity at y-axis of the points. Theobserved nodes (ys) are a vector with two elements; x -coordinates and y-coordinates of observation data.In this notation, (x1t ) means hidden variable of the first fea-ture point at time t and (y2t+1) is the observed variable of thesecond feature point at time t+1.Fig. 2. The graphical model used in this workThe selection of the edge potentials mentioned in thissection is explained in 2.3–2.5.2.3. Temporal ModelThe temporal model makes the connection of the featurepoints with the previous value of the points. This is basedon the translation model shown below.xt+1 = A · xt + w w v N(0, Q) (2)Here, A is the translation matrix. Q is the covariance ma-trix of the noise which is a normal distribution with zero-mean. Assuming the points move with a constant velocityand the point coordinates and velocities are independent ofeach other, these are selected as:A =1 0 1 00 1 0 10 0 1 00 0 0 1Q =σ2x 0 0 00 σ2y 0 00 0 σ2u 00 0 0 σ2v(3)Hence the temporal connection between two nodes involvesa Gaussian distribution and the edge potential can be definedas follows:ψt,t+1(xt, xt+1) =α exp{−12[xTt+1 xTt] [ Q−1 −Q−1A−ATQ−1 ATQ−1A] [xTt+1xTt]}(4)2.4. Spatial ModelThe spatial model defines the spatial connection betweenfeature points. This connection is selected to simply usethe expected spatial distance between feature points, for ex-ample; the distance between the eye corners. According tothese, the spatial connection is selected as below:ψ1,2(x1, x2) =α exp{[ x1 − (x2 −4x) ]T Σ−1 [ x1 − (x2 −4x) ]} (5)Here, 4x is a four elements vector containing only the dis-tance on x–axis as: 4x = [ 4 0 0 0 ]T . The covari-ance matrix is selected as below:Σ =σ′2x 0 0 00 σ′2y 0 00 0 σ′2u 00 0 0 σ′2v (6)2.5. Observation ModelThe extraction of the observations from video sequences isexplained in section 2.1.The observation model makes the connection between thehidden random variable xs and the noisy local observationvariable ys. The model is as follows:yt = C · xt + v v v N(0, R) (7)Here, C is the observation matrix and R is the covariancematrix of the noise which is a normal distribution with zero-mean. These are selected as follows:C =(1 0 0 00 1 0 0)R =(σ′′2x 00 σ′′2y)(8)As a result, the relation between xs and ys is a normal dis-tribution and it is defined as follows:ψs(xs, ys) = N(C · xs, R) = p(ys|xs) (9)2.6. Loopy Belief Propagation AlgorithmIn many computer vision and image processing applications,the main target is to find the conditional density functionp(xs|y). For graphs which are acyclic or tree–structured,the desired conditional distributions can be directly calcu-lated by a local message–passing algorithm known as be-lief propagation (BP). In chain-structured graphs, this al-gorithm is equal to Kalman or Particle filtering. For cyclicgraphs, Pearl [15] showed that belief propagation producesexcellent empirical results in many cases. The algorithm isas follows: Each node t ∈ V calculates a message mt,s(xs)to be sent to each neighboring node s ∈ N(t):mt,s(xs) = α∫xtψs,t(xs, xt)ψt(xt, yt)×∏u∈N(t)\smu,t(xt)dxt(10)Each node combines these messages and its own observa-tion and produces its own conditional density function :p(xs|y) = αψs(xs, ys)∏t∈N(s)mt,s(xs) (11)Since the relations in the model are selected as Gaussian, thetwo steps of the algorithm shown above simplify to updatingmeans and covariances. For this reason, it works faster thannon-parametric methods. Since the hidden random vectorconsists of the x,y axis coordinates and velocities, the meanvalues of the normal distributions are the estimation of thesevalues. Each update step is done using the current and theprevious data, as a result the algorithm used becomes a fil-tering algorithm. For the update equations please see [16].3. EXPERIMENTAL RESULTSThe performance of the proposed work is shown by usingthe data recorded in a laboratory environment and in a ve-hicle environment [1]. The data consist of translation ofhead and external occlusion of some points. These video se-quences consist of real–world head movements, facial move-ments, gestures etc. So this gives a sense of the practicalapplication of the proposed method.For a simplicity, only four eye corner points are tracked.The covariance matrices in the temporal, spatial and obser-vation model are selected suitably according to the videos.The results of the proposed work in videos taken in a labora-tory environment are shown in Figure 3–b and results fromthe vehicle data are shown in Figure 4–b. For comparison,the results of an algorithm that exploits only the temporalrelations [8] are shown in Figure 3–a and 4–a. Green dotsin the result images are the estimated point locations ob-tained using the observation data up to now.As shown in Figures 3–b and 4–b, tracking of feature pointsis successfully done in the case of head translation or ex-ternal occlusion. On the other hand, as seen in Figure 3–aand 4–a, the method that only use the temporal relation can-not track well and drifts occur and unreasonable trackingresults.The occlusion sequence is recorded by occluding a part ofthe face by hand. In this case it is assumed that there is anocclusion detector. The data term is closed when there is anocclusion. For a regular comparison, this is applied for bothproposed method and the method in [8].4. CONCLUSION AND FUTURE WORKIn this paper a robust feature point tracker for real time ap-plications, such as driver fatigue detection, driver behaviormodeling etc., is developed. The significant advantage ofthe algorithm is the incorporation of the temporal and spa-tial information. So if a point disappears from the scene dueto an occlusion, the information from the neighboring pointswill allow the tracking of the lost point to continue success-fully. Another advantage of the method is the computationalefficiency. The parametric assumptions make computationssimpler with respect to non-parametric techniques.For future work, the model will be improved to track all fa-cial feature points as an input to a facial expression analysissystem or a fatigue detection system in vehicles.5. ACKNOWLEDGEMENTSThis work was partially supported by the European Com-mission under Grants FP6-2004-ACC-SSA-2 (SPICE) andMIRG-CT-2006-041919, the Turkish State Planning Orga-nization under the DRIVE-SAFE project, the New Energyand Industrial Technology Development Organization (NEDO)of Japan under the project ”International Research Coor-dination of Driving Behavior Signal Processing based onLarge Scale Real World Database”.6. REFERENCES[1] H. Abut, H. Erdogan, A. Ercil, B. Curuklu, H.C.Koman, F. Tas, A.O. Argunsah, S. Cosar, B. Akan,H. Karabalkan, E. Cokelek, R. Ficici, V. Sezer, S. Da-nis, M. Karaca, M. Abbak. M.G. Uzunbas, K. Erti-men, C. Kalaycioglu, M. Imamoglu, C. Karabat, andM. Peyic, “Data collection with ’uyanik’: Too muchpain; but gains are coming,” in Proc. Biennial on DSPfor In-Vehicle and Mobile Systems, Istanbul, Turkey,June 2007.[2] Chetverikov D and Verestoy J, “Tracking featurepoints: a new algorithm,” in Proceedings of the in-ternational conference on pattern recognition, 1998,p. 14361438.[3] Rangarajan K and Shah M, “Establishing motion cor-respondence,” CVGIP: Image Understanding, vol. 54,pp. 56–73, 1991.[4] Bretzner L and Lindeberg T, “Feature tracking withautomatic selection of spatial scales,” Computer Vi-sion Image Understanding, vol. 71, no. 3, pp. 385–392, 1998.[5] Shapiro L, Wang H, and Brady J, “A matching andtracking strategy for independently-moving, non-rigidobjects,” in Proceedings of the 3rd British machinevision conference, 1992, pp. 306–315.[6] Meyer F and Bouthemy P, “Region-based tracking us-ing affine motion models in long image sequences,”Computer Vision Image Understanding, vol. 60, pp.119140, 1994.[7] Zheng Q and Chellappa R, “Automatic feature pointextraction and tracking in image sequences for arbi-trary camera motion,” International Journal of Com-puter Vision, vol. 15, pp. 3176, 1995.[8] Gu H and Ji Q, “Information extraction from image se-quences of real-world facial expressions,” Mach. Vis.Appl., vol. 16, no. 2, pp. 105–115, 2005.(a) (b)Fig. 3. Ideal environment tracking result of (a) the methodin [8] (b) the proposed method[9] Ji Q and Yang X, “Real-time eye, gaze, and face posetracking for monitoring driver vigilance,” Real-TimeImaging, vol. 8, no. 5, pp. 357–377, 2002.[10] Cootes T F, Edwards G J, and Taylor C J, “Active(a) (b)Fig. 4. Vehicle environment tracking result of (a) themethod in [8] (b) proposed methodappearance models,” in Proceedings of the Europeanconference on computer vision, 1998, p. 484498.[11] Xiao J, Baker S, Matthews I, and Kanade T, “Real-time combined 2d+3d active appearance models,”CVPR, vol. 2, pp. 535–542, 2004.[12] Wan K, Lam K, and Chong N, “An accurate activeshape model for facial feature extraction,” PatternRecognition Letters, vol. 26, no. 15, pp. 2409–2423,2005.[13] Su C and Huang L, “Spatio-temporal graphical-model-based multiple facial feature tracking,”EURASIP Journal on Applied Signal Processing, vol.13, pp. 2091–2100, 2005.[14] David S. Bolme, “Elastic bunch graph matching,”M.S. thesis, Colorado State University, 2003.[15] J. Pearl, Probabilistic Reasoning in Intelligent Sys-tems, Morgan Kaufman, San Mateo, 1988.[16] E. Sudderth, “Embedded trees: Estimation of gaussianprocesses on graphs with cycles,” M.S. thesis, MIT,2002.

Graphical model based facial feature point tracking in a vehicle environment

http://research.sabanciuniv.edu/6566/1/cosar_DSPinCars07.pdf

Graphical model based facial feature point tracking in a vehicle environment

Abstract

Similar works

Full text

Available Versions

Sabanci University Research Database