Lading, Brian

Larsen, Rasmus

Sjöstrand, Karl

This paper describes a method for building an appearance model from three-dimensional data of human faces. The data consists of 3-D vertices, polygons and a texture map. The method uses a set of nine manually placed landmarks to automatically form a dense correspondence of thousands of points. This makes sure the model is able to capture the subtle details of a face. The model can be used for face segmentation and fully automated face registration

Online Research Database In Technology

Building a 3-D Appearance Model of the Human FaceKarl Skoglund, Rasmus Larsen and Brian LadingInformatics and Mathematical Modelling, Technical University of DenmarkSeptember 8, 2003AbstractThis paper describes a method for building an ap-pearance model from three-dimensional data of hu-man faces. The data consists of 3-D vertices, poly-gons and a texture map. The method uses a setof nine manually placed landmarks to automaticallyform a dense correspondence of thousands of points.This makes sure the model is able to capture thesubtle details of a face. The model can be used forface segmentation and fully automated face regis-tration.1 IntroductionThe human face is one of the most popular andbest understood objects in research areas such asstatistical shape analysis, object recognition, fea-ture extraction, synthesis and tracking. The rea-son for this is the importance of the face in ourdaily life. The appearance of faces helps us deter-mine properties such as gender, age, race and moreintricate properties such as mood and health sta-tus. Because the face appearance is so important,the human has developed into an expert in face in-terpretation and recognition. This puts very highdemands on an area such as face synthesis.This article presents the design of a three-dimen-sional appearance model for face synthesis. An ap-pearance model is a popular type of deformablemodel built from a database of examples. It is asuitable choice for addressing the problems statedabove, as it generates suitable linear combinationsof the examples in the training set.Three-dimensional models are still fairly uncom-mon because of the large amount of overhead andcomputational difficulty involved in working with athree-dimensional data set. However, as computerpower increases and equipment for acquiring three-dimensional data becomes reasonably priced, thistype of model will be more widely used.2 BackgroundIn 1995 Cootes et al introduced the concept ofshape models [5] where shapes are defined by a setof landmarks. From a database of shapes a statisti-cal point distribution model (PDM) can be built de-scribing the main modes of shape variation. In 1998the shape model was extended to include texture,the intensity values contained by the shape bound-aries. Such models are called appearance models [4].The majority of work on appearance models havebeen done using two-dimensional data. The rea-son for this is the ease of gathering and annotatingdata, and the low computational power demands.Mitchell et al describe the building of a three-dimensional appearance model from volumetriccardiac magnetic resonance (MR) images [11]. Thispaper also describes the implementation of an ac-tive appearance model for image segmentation andrecognition. The volumetric data produced by theMR camera differs from the surface data of a hu-man face, and the methods used cannot be usedhere without modifications.Blanz and Vetter show how a three-dimensionalmorphable model of human faces can be built [2].Although the model is similar to the appearancemodel, separate models for shape and texture areused. The dense point-to-point correspondence be-tween shapes are formed using an optical flow algo-rithm. To fit the model to two-dimensional imagesa gradient descent optimization function is used.Hutton et al build a dense correspondence modelof the human face using a semi-automatic algo-rithm [7]. Each face is manually annotated witha sparse set of landmarks which are used to formthe dense correspondence. This is the method usedin this article, with a minor modification suggestedby Paulsen [9].3 MethodThe data was acquired using a Minolta Vivid 900laser scanner provided by the 3D-Laboratory atthe School of Dentistry, University of Copenhagen.15 faces were scanned, each consisting of roughly30000 3-D points and polygons and an 800×400, 24bit color image representing texture. Ages rangedfrom 20 to 40, most of the people were male andof Scandinavian origin. The age, gender and racedistribution is therefore limited. One scan takesapproximately 5 seconds, and three scans from dif-ferent angles are necessary to get a decent repre-sentation of the face area. The scanner is not ableto register hair, so a full head representation is notpossible to acquire. Eyes are also hard to registersince the laser beam gets too dispersed to record.Therefore, all scans are preformed with the eyesclosed. The scanning process takes approximately15 minutes per face including some post-processing.The resulting shape and texture data is partiallyof poor quality. The shape is well represented buthas a rough surface. This can be relaxed using asmoothing algorithm. The texture is projected ontoa cylinder with resulting artifacts. The mappingfrom the 3-D points to the texture map (called tex-ture coordinates) is poor, with many points lackingtexture coordinates. This is dealt with using linearinterpolation.3.1 Annotating ShapesWhen constructing a 2-D appearance model eachexample in the data set is annotated with corre-sponding landmarks. This is often done manually,although algorithms for automatic annotation ex-ist. [1]. Around 60 landmarks is a suitable amountfor a 2-D facial image. In three dimensions, thou-sands of landmarks are required to capture thecomplex surface of a human face. Obviously, it isnot feasible to annotate these by hand, some sort ofautomated process is necessary. A semi-automaticalgorithm is used here, which constructs a densedistribution of corresponding points from a sparseset of manually placed landmarks [7]. The algo-rithm uses one of the examples as a template shape.This example should be well represented and con-tain no statistical abnormalities. The goal of thealgorithm is to change the extent and point order-ing of all the other examples to that of the template.This brings all points defining the shape into cor-respondence. The template is first pruned so thatthe resulting extent of the template is present inall the other shapes. Each shape is then manuallylandmarked using a 3-D annotation software devel-oped by Rasmus Paulsen which has been alteredto be able to work with the type of data used here.Nine landmarks were used. These are (in this orderand from the observers point of view): the chin, theleft and right corner of the mouth, the tip of thenose, the left and right corner of the left eye, thenose curve minimum and the left and right cornerof the right eye.3.2 RegistrationThe shapes are brought into correspondence (calledregistration) as follows:The template face is deformed [9] to roughly fitthe shape to be registered using a thin-plate splinewarp [3]. The two sets of landmarks define thewarp transform. This makes the shapes coincideat the landmarks. The template is now treatedas a set of points, and the shape to be registeredis treated as a continuous surface. For each pointon the template, the closest point on the targetsurface is found and stored. When this is donefor all template points, the old target points arediscarded and replaced by the new stored points.This makes sure that the template and the newshape has the same number of points and the samepoint ordering. An effect of this is that the newshapes are also pruned according to the template.The template shape also defines the texture co-ordinates to be used in the final model. Therefore,the texture maps of all examples must be warpedand pruned according to the template. The three-dimensional shape landmarks can be transformedto two-dimensional texture map landmarks by useof the texture coordinates of each landmark. Theselandmarks are then used to thin-plate spline warpeach texture map to fit the template. The closed-loop boundary of the shape is used to form a tex-ture boundary. All texture maps are cropped sothat only textural information inside the boundaryis saved.Since the registration makes sure each pointroughly represent the same part of the face in allexamples, the polygon definition of the templateshape can be used for all shapes.3.3 Procrustes AnalysisThe faces now have an identical representation inboth shape and texture, but before any statisticalanalysis can be preformed, a Generalized PartialProcrustes Analysis [6] must be carried out for bothshapes and textures. For the shapes this meansthat differences in location and rotation are filteredout to leave only size and shape. Differences in sizeare not filtered out since the distance from the cam-era to the objects was held constant throughout thedata acquisition. Size is an interesting attribute ofa face, so it is desirable to maintain the size dif-ferences in the model. The textures are altered us-ing a one-dimensional Procrustes analysis, makingsure differences in intensity and color balance areremoved.3.4 Building the Appearance ModelA 3-D shape consisting of ns points can be seenas a single vector in R3ns where the vector can beconstructed ass = (x0, . . . , xns , y0, . . . , yns , z0, . . . , zns)The set of k shapes now consists of k points in 3ns-dimensional space. A similar vector can be con-structed for each texture map, consisting of nt pix-els. Each pixel is represented by three color values,red, green and blue, which yields the following vec-tor representation:t = (r0, . . . , rnt , g0, . . . , gnt , b0, . . . , bnt)The centroids of the point clouds defined by theshape and texture vectors represent the mean shapeand mean texture.s¯ =1kk∑i=1si, t¯ =1kk∑i=1tiThe total number of dimensions of the modelare 3ns + 3nt. To reduce the dimensionality tosomething more manageable, a principal compo-nent analysis (PCA) [10] is preformed. A PCAtransform rotates the coordinate system of high-dimensional data so that the axes point in direc-tions of maximum variance. The axes are the eigen-vectors of the covariance matrix of the data. Forthe shapes the covariance is constructed asΣs =1kk∑i=1(si − s¯)(si − s¯)TThe texture covariance matrix is constructed sim-ilarly. The PCA operations results in one shapemodel and one texture model.s = s¯+Φsbs, t = t¯+ΦtbtThe columns of the matrixΦ are the eigenvectors ofΣ, and b denotes the parameters of the deformablemodel. To remove any correlation between theshape and texture models, a third PCA is pre-formed as follows. For each example in the trainingset the corresponding parameters are found:bs = ΦTs (s− s¯), bt = ΦTt (t− t¯)For each example, the shape and texture parame-ters are concatenated into a single vector,b =[Wsbsbt]whereWs is a diagonal matrix of weights account-ing for the difference in magnitude between the dis-tance units of the shapes and the intensity units ofthe texture. The PCA is then applied to these vec-tors giving a third model,b = ΦccHere, c denotes the appearance parameters, con-trolling both shape and texture.3.5 Synthesis of FacesBy altering c, new faces can be synthesized. c = 0results in the mean shape with the mean texture.A suitable amount of alteration is ±3 standard de-viations. From the appearance parameters c, a newface can be synthesized bys = s¯+ΦsW−1s Φc,sct = t¯+ΦtΦc,tcAs an alternative, new faces can be synthesized us-ing all three models by first calculating b from cand then calculating s and t from bs and bt.4 ResultsFigure 1, 2 and 3 show the first three modes of vari-ation. All three modes has a face size component,since the model is of size-and-shape type. Further-more, the first mode seem to model gender, whilethe second and third mode model aspect ratio andamount of beard.The six first modes each describe 10% of the to-tal model variation. This is an effect of the lownumber of faces in the training set. No apparentclustering of the face vectors can occur for sucha small number of examples, instead they form aroughly gaussian distribution. With more faces inthe database, the PCA transform would constructa basis with clearly descending modes of variation.The semi-automatic registration algorithm workswell, but requires that the template and surfaceto be registered are similar. Any large variationresults in uneven and incorrect registration. Theproblem becomes apparent when the template sur-face has high curvature and the novel surface cur-vature is low. The resulting surface will have anuneven point distribution and the high curvatureparts will be cut off. For the human face, this prob-lem occurs mainly around the nose and eyebrows.A method for regularizing correspondences foundthrough methods such as this one is described in[8].To be able to examine the model easily, a graph-ical user interface has been built, allowing the userto rotate, zoom and change the modes of variationinteractively. Figure 4 shows a snapshot of the soft-ware.5 SummaryThis paper has described the building of a three-dimensional appearance model. A dense correspon-dence throughout all shapes was formed using ninemanually defined landmarks, a thin-plate splinewarp and a closest point operation. The results aresatisfying, but some artifacts occur which calls forFigure 1: Mode 1 (10%). Left to right represents-3, 0 and +3 standard deviations.Figure 2: Mode 2 (10%)Figure 3: Mode 3 (10%)Figure 4: The appearance model viewing software.a replacement of the point-to-surface closest pointoperation. To make the model more general, thedatabase of faces needs to be extended.6 Future WorkCurrently work is being done on using the appear-ance model for face segmentation and fully auto-mated face registration. The database will be ex-tended to include more face scans. Eight new facescans exist and will be added shortly.References[1] Ericsson A. and strm K. An affine invariantdeformable shape representation for generalcurves. In Proc. 9th Int. Conf. on ComputerVision, Nice, France, 2003 (to appear).[2] Volker Blanz and Thomas Vetter. A mor-phable model for the synthesis of 3D faces. InAlyn Rockwood, editor, Siggraph 1999, Com-puter Graphics Proceedings, pages 187–194,Los Angeles, 1999. Addison Wesley Longman.[3] F. L. Bookstein. Shape and the information inmedical images: A decade of the morphomet-ric synthesis. In Computer Vision and ImageUnderstanding, pages 97–118, 1997.[4] T.F Cootes and C.J. Taylor. Statistical Modelsof Appearance for Computer Vision. Univer-sity of Manchester, 2001.[5] T.F. Cootes, C.J. Taylor, Cooper D., and Gra-ham J. Active shape models-their training andapplication. Computer Vision and Image Un-derstanding, 61:38–59, 1995.[6] Ian L. Dryden and Kanti V. Mardia. StatisticalShape Analysis. John Wiley & Sons, 1999.[7] Tim J. Hutton, Bernard F. Buxton, and PeterHammond. Dense surface point distributionmodels of the human face, 2001.[8] R. R. Paulsen and K. Hilger. Shape modellingusing markov random field restoration of pointcorrespondences. In Information Processing inMedical Imaging, IPMI, 2003.[9] R. R. Paulsen, R. Larsen, S. Laugesen,C. Nielsen, and B. K. Ersbøll. Building andtesting a statistical shape model of the hu-man ear canal. In Medical Image Computingand Computer-Assisted Intervention - MIC-CAI 2002, 5th Int. Conference, Tokyo, Japan,.Springer, 2002.[10] S.C. Sharma. Applied Multivariate Tech-niques. John Wiley & Sons, 1996.[11] Boudewijn P.F. Lelieveldt Rob J. van derGeest Johan H.C. Reiber Milan Sonka StevenC. Mitchell, Johan G. Bosch. 3-d active ap-pearance models: Segmentation of cardiac mrand ultrasound images. 2002.

Building a 3-D Appearance Model of the Human Face

http://orbit.dtu.dk/ws/files/3064829/imm2541.pdf

Building a 3-D Appearance Model of the Human Face

Abstract

Similar works

Full text

Available Versions

Online Research Database In Technology