Presentado al 12th International Conference on Computer Analysis of Images and Patterns (CAIP-2007) celebrado en Viena (Austria) del 27 al 29 de agosto.Psychological evidence demonstrates how eye gaze analysis is requested for human computer interaction endowed with emotion recognition capabilities. The existing proposals analyse eyelid and iris motion by using colour information and edge detectors, but eye movements are quite fast and difficult for precise and robust tracking. Instead, we propose to reduce the dimensionality of the image-data by using multi-Gaussian modelling and transition estimations by applying partial differences. The tracking system can handle illumination changes, low-image resolution and occlusions while estimating eyelid and iris movements as continuous variables. Therefore, this is an accurate and robust tracking system for eyelids and irises in 3D for standard image quality.This work is supported by EC grants IST-027110 for the HERMES project, IST-045547 for the VIDI-Video project, by the Spanish MEC under
projects TIN2006-14606 and DPI-2004-5414. Jordi Gonzàlez also acknowledges the support of a Juan de la Cierva Postdoctoral fellowship from the Spanish MEC.Peer Reviewe

Gonzàlez, Jordi

Orozco, Javier

Roca, F. Xavier

Digital.CSIC

Deterministic and Stochastic Methodsfor Gaze Tracking in Real-TimeJ. Orozco†, F.X. Roca† and J. Gonza`lez‡† Computer Vision Center & Dept. de Cie`ncies de la Computacio´,Edifici O, Campus UAB, 08193 Bellaterra, Spain‡ Institut de Robo`tica i Informa`tica Industrial (UPC – CSIC),C. Llorens i Artigas 4-6, 08028, Barcelona, SpainAbstract. Psychological evidence demonstrates how eye gaze analysis isrequested for human computer interaction endowed with emotion recog-nition capabilities. The existing proposals analyse eyelid and iris mo-tion by using colour information and edge detectors, but eye movementsare quite fast and difficult for precise and robust tracking. Instead, wepropose to reduce the dimensionality of the image-data by using multi-Gaussian modelling and transition estimations by applying partial differ-ences. The tracking system can handle illumination changes, low-imageresolution and occlusions while estimating eyelid and iris movements ascontinuous variables. Therefore, this is an accurate and robust trackingsystem for eyelids and irises in 3D for standard image quality.1 IntroductionEyelid and iris motion description is demanded from human emotion, truthand deception evaluation by combining psychological and pattern recognitiontechniques. Ekman and Frisen [2] already established that there are percepti-ble human emotions, which can be early detected by analysing eyelid and irismovements.Applications on Human Computer Interaction (HCI) demand robustness andaccuracy in real-time, which determine the performance evaluation of alreadyproposed techniques. In the literature, there are approaches for gaze analysisdealing with contour detectors, colour segmentation, Hough transform and Op-tical Flow for eyelids and irises [10,5,9]. These methods are time-consuming anddepend on the image quality. On the other hand, restricted detailed textures andtemplates have been proposed to apply template matching, skin colour detectionand image energy minimization [8,6]; for example Moriyama et al [6] deal withthree eyelid states namely open, closed and fluttering by constructing detailedtemplates of skin textures for eyelid, iris and sclera. This approach requires train-ing and texture matching. So, these methods are difficult to be generalized fordifferent image and environment conditions.We propose in this paper a robust and accurate eyelid and iris tracking bycombining stochastic and deterministic approaches with reduced image-data. Toprovide robustness, we reduce the input image by applying Appearance-Models2 J. Orozco et al.(a) (b) (c)Fig. 1. The 3D mesh (a) is projected onto the input image (b) to construct thecorresponding appearance (c).[3], which learn the skin texture on-line based on multi-Gaussian assumptions.To provide accuracy, we construct two Appearance-Based Trackers (ABT) toestimate the transition by partial differences with respect to appearance para-meters. The first one excludes the sclera and iris information, while achievingfast and accurate eyelid adaptation for any kind of blinking and fluttering mo-tion. The second one is able to track the iris movements, while retrieving thecorrect adaptation after eyelid occlusions or iris saccade movements. Both track-ers agree on the best 3D mesh pose that depends on the head position. The headpose estimation enhances the system capabilities for tracking eyelids and irisesin different head position, instead of a frontal face as restriction.Compared to existing gaze tracking methods, our proposed approach hasseveral advantages. First, our system achieves an efficient eyelid and iris track-ing, whose movements are encoded as continuous values. Second, this approachis suitable for real-time applications while handling occlusions, illuminationchanges, faster saccade and blinking movements. Third, we deal with small im-ages and low resolution, which extends the capabilities of the system for differenttype of applications.The paper is organized as follows: section 2 describes the theoretical foun-dations for appearance-based trackers, the stochastic observation and determin-istic transition models. Section 3 presents experimental results and discussion.Finally, section 4 concludes the paper with the main conclusion and future av-enues of research.2 Appearance-Based Tracking2.1 Image-Data ReductionThe appearance model components are the deformable model and texture. Inorder to model the eye region, we construct a 3D model of both left and righteyes, which is composed of 36 vertices and 53 triangles, see Fig. 1.(a). This meshcovers the eyeballs, the upper and the lower eyelids, the sclera and the iris. Themesh deformation is determined by the matrix M, which is a n x i matrix:Mn,i = mn,i +Gn,i,k ∗ γk , (1)where n is the number of vertices and i corresponds to the Cartesian coordinatesin the image plane. The matrix mn,i is determined by the biometry of eachperson in neutral position. The matrix Gn,i,k deforms the mesh depending onEyelid and Iris Tracking with Row-Images. 3the eyelid and iris movements. The eyelid and iris movements are controlledby the vector γk for k = 0, 1, 2 for eyelid, iris yaw and iris pitch respectively.These variables are encoded according to the Facial Action Codifying System(FACS), which obey the MPEG-4 codification. Thereby, the 3D mesh Mn,i canbe adapted to the eye region, see Fig. 1.(b). The 3D pose of the mesh is taken into account ρ = [θx, θy, θz, x, y, s], while assuming a weak perspective projectionmodel. Therefore, each 3D point Pi = (Xi, Yi, Zi) ⊂ M will be projected ontothe image point pi = (ui, vi) by using a projection matrix B. That is, (ui, vi) =B(Xi, Yi, Zi, 1).Consequently, given an input frame F and the parameters to modify themesh, which are encoded as q = [ρ,γ], we construct an appearance model torepresent the eye region [1]. The shape is provided by the 3D mesh and thetexture is obtained by applying a warping function Ψ(F,q) which transfers eachpixel from the input frame F into a reference texture according to the vector q.In this way, the final appearance A(q) = [a0, ..., al] is obtained, which dependson the mesh configuration q, see Fig. 1.(c).2.2 Stochastic Appearance ObservationThe observation model aims to provide the expected appearances over time,which are based on previous estimations. Therefore, a likelihood distributionis an appropriate method to generate the expected appearances while learningprevious estimations. Given an appearance model, A(q) = [a0, ..., al], which de-pends on the mesh configuration q, we assume each pixel of the appearance aifollowing a Gaussian distribution over time. Thus, we can collect all the vari-ables in a multi-dimensional vector, which can be assumed following a Gaussiandistribution as well, N(µ,σ2). µ and σ are vectors of l values according tothe appearance variables ai, which means that the variance must be computedcomponent by component and not through inner product of the vector σ. There-fore, the probability for each observation is given by the conditional likelihoodfunction:P(At|qt−1) =l∏i=0N(ai;µi, σi). (2)The tracking goal is the estimation of the vector q at each frame t. We rep-resent as qˆt and Aˆt(qˆt) the tracked parameters and the estimated appearances[1]. For the sake of clarity, hence we assume that At(qt) and At are equivalentand used depending on the specification level. The estimated average appear-ance is obtained by applying a recursive filtering technique. Thus, µ and σ2 areupdated for the next frame with respect to previous adaptations and a learningcoefficient λ:µt+1 = λµt + (1− λ)Aˆt and σ2t+1 = λσ2t + (1− λ)(Aˆt − µt)2, (3)where µ and σ are initialized with the first appearance A0.4 J. Orozco et al.(a) (b) (c) (d)Fig. 2. The 3D mesh (a) and the appearance (b) for the eyelid tracker. The 3Dmesh (c) and the appearance (d) for the iris tracker.2.3 Deterministic Appearance TransitionIn order to estimate the vector qt for the next frame, we adopt an adaptivevelocity model, which is predicted by using a deterministic function to obtainthe transition state based on the previous prediction, qˆt−1:qˆt = qˆt−1 +∆qˆt (4)where ∆qˆt is the shift of the mesh configuration.Consequently, for each qˆt we construct the corresponding appearance, whichis compared with the likelihood average appearance by the Mahalanobis dis-tance. Therefore, given Eq. (4), the appearance becomes At ≈ µt, which canbe approximated via a first-order Taylor series expansion around qˆt using theVanilla gradient descent method [7]:At(qt) ≈ Ψ(Ft, qˆt−1) +∂(Aˆt, qˆt)∂qˆt(qˆt − qˆt−1). (5)As a result, the estimation of the vector qˆt depends on both the previousadapted and the current average appearance, as well as the minimization dis-tance. The gradient is computed by partial differences with specific descent steps,due to the saccade movements and spontaneous blinking. Thus, tracking is en-hanced by quickly retrieving the best adaptation while avoiding drifting prob-lems. Illumination changes, occlusions and faster movements are considered asoutliers by constraining with the Huber’s function [4] the gradient descent stepfor each component of the shift vector ∆q. The Huber’s function, ξˆ function isdefined as:η(y) =1ydξˆ(y)dy={1 if|y| ≤ cc|y| if|y| > c(6)where y is the value of a pixel in the appearanceAt normalized by the appearancestatistics µ¯ and σ¯ according to the Gaussian assumption for the Appearance.The constant c is set 3 ∗ σ¯. Thus, we constrain the appearance registration onthe probabilistic model.2.4 Eyelid and Iris TrackingEyelids and irises perform fast movements, which are difficult to predict dueto the spontaneous motion and the low resolution on images from monocularEyelid and Iris Tracking with Row-Images. 5cameras. Therefore, a probabilistic model for them would be quite uncertain.Instead, we propose two models for each tracker, which avoid those pixels thatadd uncertainty for the appearance predictions, see Fig. 2.Eyelid tracker uses an appearance model, which excludes the sclera and theiris regions by warping these pixels like eyelid skin. Subsequently, the iris trackeruses an appearance that includes the sclera and iris region. Thus, we constructtwo Appearance-Based Trackers (ABT), which are combined sequentially, as de-scribed next.Eyelid Tracker Tw: estimates the eyelid position independently from irises,see Fig. 2.(a). The tracking vector is w = [ρ, γ0], and the appearance modelA(w), which excludes sclera and iris, see Fig. 2.(b):1. To obtain At(wt−1) by applying the warping function, Ψ(Ft,wt−1).2. The Gaussian parameters1 are estimated, µt(w) and σ2t (w) by using Eq. (3).We try with five different learning coefficients λw, the high values allow tolearn fast movements while the low values keep more information to handleocclusions.3. To compute the eyelid gradient based on the previous adaptation wt−1 byusing Eq. (5), while testing the whole FACS range [-1,1].4. The best estimation is obtained using Eq. (4), by comparing the averageand likelihood appearances through a Mahalanobis distance in an iterativeGauss-Newton process. The search involves exploitation more than explo-ration to avoid local minima and to estimate the spontaneous movements.Iris Tracker Tq: estimates yaw and pitch orientations for irises, see Fig.2.(c). The tracking vector is q = [ρ, γ0, γ1, γ2] to obtain the appearance A(q):1. To obtain At(qt−1), by applying the warping function Ψ(Ft,qt−1).2. The Gaussian parameters µt(q) and σ2t (q) by applying Eq. (3). The learningcoefficient λq is lower than λw to keep more information from previousframes, since the iris motion is slower than the eyelid blinking.3. To compute the iris gradient At(qt−1), the iris gradient is estimated forq = [ρ, γ0, γ1, γ2] in the whole FACS range [-1,1], using Eq. (5).4. Finally, the best estimation is computed using Eq. (4), by minimizing theMahalanobis distance. The convergence is achieved by using a shorter explo-ration than the previous tracker.Both trackers are connected through the error estimation, which is standard-ized according to the number of pixels of each appearance. Subsequently, theeyelid tracker provides the vector w = [ρ, γ0] while the iris tracker estimates theiris while correcting the previous mesh orientation, q = [ρ, γ0, γ1, γ2], see Fig. 3.1 Let be µ(w) and σ2(w) the Gaussian parameters corresponding to the eyelid trackerTw. Similarly, µ(q) and σ2(q) are the Gaussian parameters for the iris tracker Tq.6 J. Orozco et al.Fig. 3. The eyelid tracking handle movements and blinks as continuous values.Iris tracking for spontaneous movements and saccades. The eyelid position iscorrected for the pitch variation effects.Fig. 4. Eyelid and Iris trackers are applied sequentially while learning each ap-pearance texture on-line, which enhances the robustness and accuracy.3 Experimental ResultsExperiments were run in a 3.2 GHz Pentium PC, in ANSI C code. Three imagesequences of 250 frames were used for testing both trackers, which correspondoriginally to facial image sequences with the cropped eye region. They wererecorded with monocular and photographic cameras without illumination con-ditions. We do not use high image resolution, because the reference texture sizeis 14x18 pixels.We computed the eyelid estimation by using Tw while dealing with any kindof blinking, e.g. open, closed and fluttering. In addition, the iris estimation isdone by using Tq, which estimates yaw and pitch motion, saccade movementsand eyelid occlusions.On one hand, the eyelid tracker adapts the eyelid position for slow movementsand blinks while estimating the mesh orientation, see Fig. 4. This tracker doesEyelid and Iris Tracking with Row-Images. 7Fig. 5. Due to tracking capability to handle illumination changes and occlusions,the tracker has a good performance with eyes wearing glasses.Fig. 6. The eyelid and iris tracking deals with images of small and low-resolution.These images are 64x20 pixels where each input eye is 18x12 pixels.not depend on iris estimations even when the pitch movement affects the eyelidposition. However, this tracker warps the sclera and iris pixels as eyelid skin,otherwise, those pixels are considered outliers when the eyelid is occluding theinner eye region. On the other hand, the iris tracker adapts well the iris positionwhile dealing with slow yaw and pitch motion while retrieving the correct adap-tation of the mesh after either saccade motion or iris movements while the eyelidoccludes the iris. Both trackers use the FACS codification, which is expressed ascontinuous values between -1.0 and 1.0, see Figs. 3. The eyelid and iris pitch havesimilar plots because they commonly perform a synchronized and spontaneousmovement. This is a correlation that is independently well estimated.For an appearance model size of 14x18 pixels, we obtained a performanceof 21 fps, and a 96% of correct adaptations. The error is principally due to thesaccade motion or eyelid occlusions. Each sequence was also tested using anappearance size of 5x11 pixels, thus obtaining 52 fps for the iris tracker and 67fps for the eyelid tracker, and average accuracy adaptation of 85%. Iris trackingis a challenge for small reference texture, where the iris size is 2x3 instead of 5x6for the big resolution.Learning the texture on-line and handling illumination changes are importantcapabilities to obtain good adaptations by analysing row-image resolution, andoccluded eye regions. The Fig. 5 shows a 400 image sequence, where the subjectis wearing sunglasses, the input eye region is 42x82 pixels. The tracking got a91% of correct adaptations with a reference texture of 14x18 pixels. We obtain82% of correct adaptation analysing small images with low resolution, where theinput eye region is 10x18, see Fig. 6. The whole input image is 112x160 that isappropriate for video conference software.8 J. Orozco et al.4 ConclusionsThree main contributions were proven in this work. First, the information anddimensionality reduction for the input image by constructing appearance models,which are appropriate for statistical modelling. Second, the stochastic observa-tion model provides an accurate likelihood function, which is conditioned byprevious estimations and accumulative appearance information. Third, the de-terministic transition model allows generating an appearance space by applyingfirst-order Taylor approximations around to the previous adaptation.The experimental results have proven that combining sequentially two ABT,the system is able to estimate accurately the eyelid and iris position in 3D im-ages. The system does not require high quality images or specific illuminationconditions, since we do not use colour information, edge detectors, or motionextraction algorithms. We have demonstrated in this framework that eyelid andiris motion is reliable for HCI applications and psychological systems, which de-mand real-time performance with accurate results. Our system provides a robustand accurate gaze description, able to handle occlusions, illumination changes,small and low-resolution images in 3D.Acknowledgement: This work is supported by EC grants IST-027110 for theHERMES project, IST-045547 for the VIDI-Video project, by the Spanish MEC underprojects TIN2006-14606 and DPI-2004-5414. Jordi Gonza`lez also acknowledges thesupport of a Juan de la Cierva Postdoctoral fellowship from the Spanish MEC.References1. T.F. Cootes and C.J.Taylor. Statistical Models of Appearance for Computer Vision.Imaging Science and Biomedical Engineering, University of Manchester, 2004.2. P. Ekman and V. Friesen. Facial action coding system: A technique for the mea-surement of facial movement. Consulting Psychologists Press, Palo Alto, 1978.3. T. F. Cootes G. J. Edwards and C. J. Taylor. Face recognition using active ap-pearance models. In Proceedings of the Fifth European Conference on ComputerVision (ECCV), 2:581–695, 1998.4. P.J. Huber. Robust estimation of a location parameter. The Annals of Mathemat-ical Statistics, 35:73 – 101, 1964.5. H. Liu, Y. Wu, and H. Zha. Eye states detection from color facial image sequence.Proc. 2nd Int. Conf. Image and Graphics (ICIG02), 4875:693–698, 2002.6. T. Moriyama, J. Xiao, J. Cohn, and T. Kanade. Meticulously detailed eye modeland its application to analysis of facial image. IEEE Transactions on PatternAnalysis and Machine Intelligence, 28(5):738 – 752, May 2006.7. J. Nocedal and S. Wright. Numerical optimization. In Springer, New York, 1999.8. A. Basu S. Bernogger, L. Yin and A. Pinz. Eye tracking and animation for mpeg-4coding. IEEE Pattern Recognition, 1998. Proceedings, Fourteenth InternationalConference, 2:1281–1284, 1998.9. H. Tan and Y. Zhang. Detecting eye blink states by tracking iris and eyelids.Pattern Recognition Letters, 2005, 2005.10. J.F. Cohn Y. Tian, T. Kanade. Dual-state parametric eye tracking. In: Interna-tional Conference on Automatic Face and Gesture Recognition, 2000.

Deterministic and stochastic methods for gaze tracking in real-time

Abstract

Similar works

Full text

Available Versions

Digital.CSIC