Good registration (alignment to a reference) is essential for accurate face recognition. We use the locations of facial features (eyes, nose, mouth, etc) as landmarks for registration. Two landmarking methods are explored and compared: (1) the Most Likely-Landmark Locator (MLLL), based on maximizing the likelihood ratio [1], and (2) Viola-Jones detection [2]. Further, a landmark-correction method based on projection into a subspace is introduced. Both landmarking methods have been trained on the landmarked images in the BioID database [3]. The MLLL has been trained for locating 17 landmarks and the Viola-Jones method for 5 landmarks. The localization error and effects on the equal-error rate (EER) have been measured. In these experiments ground- truth data has been used as a reference. The results are described as follows:\ud
1. The localization errors obtained on the FRGC database are 4.2, 8.6 and 4.6 pixels for the Viola-Jones, the MLLL, and the MLLL after landmark correction, respectively. The inter-eye distance of the reference face is 100 pixels. The MLLL with landmark correction scores best in the verification experiment.\ud
2. Using more landmarks decreases the average localization error and the EER

Bazen, A.M.

Beumer, G.M.

Tao, Q.

Veldhuis, R.N.J.

English

Good registration (alignment to a reference) is essential for accurate face recognition. We use the locations of facial features (eyes, nose, mouth, etc) as landmarks for registration. Two landmarking methods are explored and compared: (1) the Most Likely-Landmark Locator (MLLL), based on maximizing the likelihood ratio [1], and (2) Viola-Jones detection [2]. Further, a landmark-correction method based on projection into a subspace is introduced. Both landmarking methods have been trained on the landmarked images in the BioID database [3]. The MLLL has been trained for locating 17 landmarks and the Viola-Jones method for 5 landmarks. The localization error and effects on the equal-error rate (EER) have been measured. In these experiments ground- truth data has been used as a reference. The results are described as follows: 1. The localization errors obtained on the FRGC database are 4.2, 8.6 and 4.6 pixels for the Viola-Jones, the MLLL, and the MLLL after landmark correction, respectively. The inter-eye distance of the reference face is 100 pixels. The MLLL with landmark correction scores best in the verification experiment. 2. Using more landmarks decreases the average localization error and the EER

Veldhuis, Raymond N.J.

NARCIS 

Comparing Landmarking Methods for Face Recognition

University of Twente Research Information

1Comparing landmarking methods for face recognitionG.M. Beumer, Q. Tao, A.M. Bazen and R.N.J. Veldhuisg.m.beumer@utwente.nlUniversity of Twente, EEMSC, Signals & SystemsP.O. box 217, 7500 AE, Enschede, The NetherlandsAbstract—Good registration (alignment to a reference) isessential for accurate face recognition. We use the locationsof facial features (eyes, nose, mouth, etc) as landmarks forregistration. Two landmarking methods are explored andcompared: (1) the Most Likely-Landmark Locator (MLLL),based on maximizing the likelihood ratio [1], and (2) Viola-Jones detection [2]. Further, a landmark-correction methodbased on projection into a subspace is introduced.Both landmarking methods have been trained on thelandmarked images in the BioID database [3]. The MLLLhas been trained for locating 17 landmarks and the Viola-Jones method for 5 landmarks. The localization error andeffects on the equal-error rate (EER) have been measured.In these experiments ground- truth data has been used as areference. The results are described as follows:1. The localization errors obtained on the FRGC databaseare 4.2, 8.6 and 4.6 pixels for the Viola-Jones, the MLLL,and the MLLL after landmark correction, respectively. Theinter-eye distance of the reference face is 100 pixels. TheMLLL with landmark correction scores best in the verifica-tion experiment.2. Using more landmarks decreases the average localizationerror and the EER.Keywords: facial feature, face registration, face recogni-tion, landmarking, likelihood ratio, Viola-Jones, landmarkcorrectionI. I NTRODUCTIONResearch by Riopka et al. [4], Cristinacce et al. [5] andBeumer et al. [6] showed that precise landmarks are essen-tial for a good recognition performance. Cristinacce didresearch on landmark locators based on correlation, ori-entation maps and Viola-Jones based algorithm [7]. TheViola-Jones algorithm was proposed by Viola et al. [2] Inthis paper we propose an improvement on earlier work byBazen et al. [1] and a Viola-Jones based landmark finder.Both methods will be compared to each other and thegroundtruth data. The methods will be compared by cal-culating the RMS value of the error between the resultsand to the groundtruth data. Also the outcome of an verifi-cation experiment measured by the equal error rate (EER)will be given.II. L ANDMARK DETECTIONThe first step in face recognition is to locate the face inthe image. This is done with a Viola-Jones based algorithm[2] from the OpenCV library [8]. We assumed that there isonly one face per image. When the face is found a regionof interest (ROI) is selected for each landmark. In this ROIwe search for landmarks using one of the two algorithms.Both will be explained in this section.A. Most Likely Landmark LocationIn MLLL landmark finding is seen as a two-class classi-fication problem: a location in an image is either the land-mark or it is not. The texture values in a region surround-ing a landmark are used as features for the classification.For each location in the ROI the likelihood ratio -for thatlocation to be the landmark- is calculated. Themostlikelylocation, i.e. the one with the highest score, is assumedto be the landmark. The landmarks used can be seen inFigure 1. Outliers may occur due to errors by the MLLL.Sometimes these outliers can be corrected using a shapecorrection. Both the landmark detection and the shape cor-rection are discussed in more detail in the next sections.Fig. 1. Landmarks in the face usedA.1 Likelihood ratio based facial feature finderThe MLLL calculates the likelihood ratio for a landmarkat each position in the ROI. The gray level intensities of theneighbourhood of a candidate location(u, v) form a vectorxu,v. The likelihood thatxu,v is the neighbourhood of thelandmark is found by calculatingL(u, v) =p(xu,v|L)p(xu,v|L)(1)594The location of the landmark is chosen at the point(u, v)at whichL(u, v) is maximum. This is done for all 17 land-marks.A.2 Shape correctionSometimes landmarks are detected incorrectly. The aimof shape correction is to detect or correct the errors made.This can be done by defining a space in which correctshapes fit but deformed ones will not. A shape is the col-lection of the coordinates of a set of landmarks. Correctshapes are assumed to be in a subspace ofR2d with d thenumber of landmarks. A basis of the subspace is deter-mined by means of principal component analysis (PCA).The shape is projectedthere and back again(BILBO). Inthe subspace the number of features is reduced. Project-ing it back to the original space coordinates which not fitthe model will have shifted. The landmarks of which thelocation changed significantly during BILBO, are consid-ered to be wrong. The new location is taken as the correctlocation.To train BILBO we need a set of shapes. The shapesconsist of 17 coordinates of landmarks in the face. A shapevector consists of a column vector with all theu coordi-nates and all thev coordinates makingx, 34 × 1 in size.A model is trained on groundtruth data which are put ascolumns in a matrixX.Training BILBO consists of the following steps:1. Register all the shapes inX, to the average shape.2. Add -limited- rotation, translation and scaling to allshapes inX in order to model variance encountered inthe images. We added normal distributed variations. Thetranslation has a standard deviation of 5 pixels. The scalehas a standard deviation of 5%. Finally the rotation has astandard deviation of 3 degrees.3. Perform an SVD onX: X = USV T .4. Reduce features by taking only the firstn columns ofU .5. Calculate a transformation matrix;T = UUT .To correct a shape the next algorithm is used:1. Estimate the shape after transformation,x′ = Tx.2. Determine the Euclidian distanceDi per landmark be-tweenx andx′.3. Determine thresholdτ . The threshold is calculated asτ = RC11717∑i=1Di (2)whereC is a constant andR is the run number.4. Only for coordinates of whichDi > τ : update vectorx: xi = x′i.5. Repeat steps 1 to 4. Once for a landmarkDi < τ stopupdating it until noDi > τ for all i.6. Repeat step 1 to 5 changing all coordinates untilR = 5.7. Transform the coordinates back to the original scale.In Figure 2 an example of corrected landmarks is shown.The circles show the raw landmarks and the triangles showthe new location of the landmarks.Fig. 2. Original shape and corrected landmarks (triangles)B. Viola-Jones based facial landmark localizationThe second approach for landmark localization is theViola-Jones based method [2], which uses a combinationof Haar-like features to represent the texture informationin an image. A detailed description of this method and thetraining method -Adaboost- can be found in [2].We developed 5 landmark detectors, namely twoeyes (size28×14), one nose (size28×14), and two mouthcorners (size20×20). The locations of the templates (pos-itive samples) are obtained from the BioID ground truthdata, as shown in Figure 1. The negative samples are ran-domly chosen from the background which does not con-tain the landmarks. Compared to the MLLL method, only5 landmarks are chosen for Viola-Jones based method, be-cause the texture of other landmarks (like eyebrows andlips) are not as constant as these 5 landmarks for AdaBoosttraining. Experiments also showed that they could not re-sult in fast and compact cascades for detection.The BioID database contains 1,521 images which havebeen manually landmarked. From each image we obtaine 5facial landmark templates.The code of the Adaboost train-ing is taken from the Intel OpenCV library [2]. In our workeach detector is trained with 3,000 positive samples and6,000 negative samples. For simplicity the face region isfirstly detected as ROI to localize landmarks in face. Fig-ure 3 shows the results of applying the Viola-Jones methodfor localizing face and landmarks. This method does amulti-scale search and chooses the facial landmark can-didates through thresholding [2]. There is the possibilityof multiple candidates (multi-size and/or different posi-tion), or missing candidate for one facial landmark. For themulti-size case, we choose the candidate with the largestsize. The reason is that it is observed that smaller-size595Fig. 3. The landmarking result by Viola-Jones methodcandidates tend to be less reliable, like a small eye can-didate localized on the eyebrow. For candidates with thesame size but in difference positions, we calculate theirrelative locations in reference to the face ROI, and choosethe one nearest to its average position. In case of missinglandmarks, we use fewer points for registration. Besides,it is also possible to restore the missing landmarks accord-ing to the statistics of their geometrical distribution in faceregion.III. E XPERIMENTS AND RESULTSTwo methods for the localization of landmarks wereproposed. Both will be evaluated in two ways. First weevaluate the RMS distance between the found landmarklocation and the groundtruth. Then the landmarks are usedfor registration in a verification experiment. The EER willserve as a benchmark for the quality of the landmarks.For the experiments we used two databases. TheBioID [3] is used to train the feature detection and theshape correction. The BioID database consists of 1521images which vary in pose, scale and lightning conditions,but are mainly frontal. The FRGC database [9] was usedto test the alignment algorithms. The FRGC database wasalso used to train and test the verification experiment. TheFRGC database consists 5658 images. Two third, 3772, ofthe database are high quallity images with low variety inpose, lighting and scale with around 300 pixels inter eyedistance. For our tests we used only the high quality partof FRGC database images.A. Training the MLLLThe MLLL detector is trained on the 17 landmarks asshown in Figure 1. The positive templates were all selectedusing hand labeled groundtruth data. The negative sampleswere all taken around the landmark at a minimal distanceof half the size of the template. All templates are either40x40 pixels or 60x40 pixels in size. The templates areall normalized in energy. This means that they are zeromean and have standard deviation one. For training thefeature extraction we used the method proposed by Bazenet al. [10], which uses AMDA for feature reduction. Thefeature extraction matrixT is stored as are the covariancematrices and averages.B. Accuracy of the facial feature detectionTo evaluate the landmarking we need a well definedmeasure for error. Since there are images of various scalesin the image a simple root mean square (RMS) distance isnot sufficient. In order to calculate a meaning full measurea simple method was used:1. Translate, scale and rotate the groundtruth data so thatthe eyes ate at 100 pixels apart.2. Align the shape found to the groundtruth shape.3. Calculate the distance between each landmark and it’sgroundtruth equivalent.4. Remove bias which is caused by the differences in la-beling policy between the databases i.e. tip of the nose ver-sus a point between the nostrils in the different databases.5. Calculate the RMS value of the remaining differencebetween the found shape and the groundtruth shape.This was done on the FRGC database. The labels from theFRGC database the center of the mouth while our methodslabel the mouth corners. In this case the mouth cornerswere averaged to make an estimate of the center of themouth.B.1 ResultsThe results can be found in Table I. This table showsthat BILBO works. The RMS error on the landmark lo-cation improves a lot. After BILBO the MLLL performsapproximately the same as the Viola-Jones method. Onthe FRGC data again both perform equally well on theeyes. On the nose and the mouth the BILBO+MLLL out-performs the Viola-Jones method. The results on the nosesuggest that the Viola-Jones method needs more training.C. Impact on verification experimentIn this section, the verification experiment we used forface recognition is discussed. The images are aligned us-FRGC Right eye Left eye Nose MouthViola-Jones 3.2 3.3 6.3 4.1MLLL 6.7 7.2 13.0 7.3BILBO+MLLL 4.2 4.6 5.8 3.7TABLE IRELATIVE RMS ERROR RESULTS ON THEFRGCDATABASE.596ing the found coordinates. Half the images are used totrain the verification while the other half is used to test it.The experiments are repeated 10 times in order to get agood estimate of the expected EER of the data for this al-gorithm. More information on the algorithm can be foundin in an article by Beumer et al. [6]. To investigate theimpact of the number of landmarks on the registration wealso did experiments on only 5 out of 17 landmarks for theMLLL coordinates. The eyes, mouth corners and the nosewere used for this.It should be noted that the verification experiment isnot optimized and that parameter tuning might improveits overall performance. However, this is not necessaryin order to compare results. The experiment uses thecoordinates found by one of the two algorithms or thegroundtruth data to register the faces.C.1 ResultsThe results can be seen in Table II. It shown that us-ing groundtruth verification gives good results (EER =0.45%). Our methods results in higher EERs around 4%.The results for the MLLL and BILBO+MLLL show thatthe shape correction does improve the results. Also theyboth outperform the Viola-Jones method. The experimenton only 5 landmarks shows however that this difference iscaused by the fact that only 5 landmarks are available forthe Viola-Jones method and 17 for the MLLL.IV. CONCLUSIONSThe proposed methods for registration are not yet accu-rate enough to compete with groundtruth data. The pro-posed shape correction work but only when enough land-marks are available. The Viola-Jones method performsbest in terms of absolute distance. For the verification ex-periment it is lacking. This is due to the fact that it onlyis trained on 5 landmarks. It ourperforms the MLLL whenonly using 5 landmarks. The MLLL with shape correc-tion performs best. It is however to be expected that whenthe number of landmarks for the Viola-Jones method canFRGC EER[%] std(EER)[%]Ground truth data 0.45 0.03Viola-Jones 4.9 0.1MLLL 4.0 0.1BILBO+MLLL 17 landmarks 3.6 0.1BILBO+MLLL 5 landmarks 6.1 0.1TABLE IIRESULTS OF THE BIOMETRIC VERIFICATION EXPERIMENTbe extended this method will outperform the MLLL alsoon the verification experiment. This is also due to the factthat a shape correction method then can be used. The re-sults from experiments by Beumer et al. [6] which showthat more landmarks are good for noise reduction are con-firmed by this research.REFERENCES[1] A.M. Bazen, R.N.J. Veldhuis, and G.H. Croonen, “Likelihoodratio-based detection of facial features,” inProc. ProRISC 2003,14th Annual Workshop on Circuits, Systems and Signal Process-ing, Veldhoven, The Netherlands, nov 2003, pp. 323–329.[2] Paul Viola and Michael Jones, “Robust real-time object detec-tion,” International Journal of Computer Vision, 2002.[3] HumanScan, “Bioid face db,”http://www.humanscan.de/ .[4] T.P. Riopka and T. Boult, “The eyes have it,” inProceedings ofACM SIGMM Multimedia Biometrics Methods and ApplicationsWorkshop., Berkeley, CA, 2003, pp. 9–16.[5] D. Cristinacce, T. Cootes, and I. Scott, “A multi-stage approachto facial feature detection,” in15th British Machine Vision Con-ference, London, England, 2004, pp. 277–286.[6] G.M. Beumer, A.M.Bazen, and R.N.J. Veldhuis, “On the accu-racy of eers in face recognition and the importance of reliable reg-istration.,” inSPS 2005. IEEE Benelux/DSP Valley, April 2005.[7] D. Cristinacce and T. Cootes, “A comparison of shape constrainedfacial feature detectors,” in6th International Conference on Au-tomatic Face and Gesture Recognition 2004, Seoul, Korea, 2004,pp. 375–380.[8] Intel, “Open computer vision library,” http://sourceforge.net/projects/opencvlibrary/ .[9] NIST, “Frgc face db,”http://www.frvt.org/FRGC/ .[10] A.M. Bazen and R.N.J. Veldhuis, “Detection of cores in finger-prints with improved dimension reduction,” inProc. SPS 2004,Hilvarenbeek, The Netherlands, apr 2004, pp. 41–44.597

A comparison of shape constrained facial feature detectors,”

A multi-stage approach to facial feature detection,”

Detection of cores in ﬁngerprints with improved dimension reduction,” in

Likelihood ratio-based detection of facial features,” in

On the accuracy of eers in face recognition and the importance of reliable registration.,”

Comparing landmarking methods for face recognition

Abstract

Similar works

Full text

Available Versions

NARCIS

University of Twente Research Information