3D facial landmark localization using combinatorial search and shape regression

This paper presents a method for the automatic detection of facial landmarks. The algorithm receives a set of 3D candidate points for each landmark (e.g. from a feature detector) and performs combinatorial search constrained by a deformable shape model. A key assumption of our approach is that for some landmarks there might not be an accurate candidate in the input set. This is tackled by detecting partial subsets of landmarks and inferring those that are missing so that the probability of the deformable model is maximized. The ability of the model to work with incomplete information makes it possible to limit the number of candidates that need to be retained, substantially reducing the number of possible combinations to be tested with respect to the alternative of trying to always detect the complete set of landmarks. We demonstrate the accuracy of the proposed method in a set of 144 facial scans acquired by means of a hand-held laser scanner in the context of clinical craniofacial dysmorphology research. Using spin images to describe the geometry and targeting 11 facial landmarks, we obtain an average error below 3 mm, which compares favorably with other state of the art approaches based on geometric descriptors

Automatic landmark annotation and dense correspondence registration for 3D human facial images

Dense surface registration of three-dimensional (3D) human facial images holds great potential for studies of human trait diversity, disease genetics, and forensics. Non-rigid registration is particularly useful for establishing dense anatomical correspondences between faces. Here we describe a novel non-rigid registration method for fully automatic 3D facial image mapping. This method comprises two steps: first, seventeen facial landmarks are automatically annotated, mainly via PCA-based feature recognition following 3D-to-2D data transformation. Second, an efficient thin-plate spline (TPS) protocol is used to establish the dense anatomical correspondence between facial images, under the guidance of the predefined landmarks. We demonstrate that this method is robust and highly accurate, even for different ethnicities. The average face is calculated for individuals of Han Chinese and Uyghur origins. While fully automatic and computationally efficient, this method enables high-throughput analysis of human facial feature variation.Comment: 33 pages, 6 figures, 1 tabl

A quantitative assessment of 3D facial key point localization fitting 2D shape models to curvature information

This work addresses the localization of 11 prominent facial landmarks in 3D by fitting state of the art shape models to 2D data. Quantitative results are provided for 34 scans at high resolution (texture maps of 10 M-pixels) in terms of accuracy (with respect to manual measurements) and precision (repeatability on different images from the same individual). We obtain an average accuracy of approximately 3 mm, and median repeatability of inter-landmark distances typically below 2 mm, which are values comparable to current algorithms on automatic localization of facial landmarks. We also show that, in our experiments, the replacement of texture information by curvature features produced little change in performance, which is an important finding as it suggests the applicability of the method to any type of 3D data

Dense 3D Face Correspondence

We present an algorithm that automatically establishes dense correspondences between a large number of 3D faces. Starting from automatically detected sparse correspondences on the outer boundary of 3D faces, the algorithm triangulates existing correspondences and expands them iteratively by matching points of distinctive surface curvature along the triangle edges. After exhausting keypoint matches, further correspondences are established by generating evenly distributed points within triangles by evolving level set geodesic curves from the centroids of large triangles. A deformable model (K3DM) is constructed from the dense corresponded faces and an algorithm is proposed for morphing the K3DM to fit unseen faces. This algorithm iterates between rigid alignment of an unseen face followed by regularized morphing of the deformable model. We have extensively evaluated the proposed algorithms on synthetic data and real 3D faces from the FRGCv2, Bosphorus, BU3DFE and UND Ear databases using quantitative and qualitative benchmarks. Our algorithm achieved dense correspondences with a mean localisation error of 1.28mm on synthetic faces and detected $14$ anthropometric landmarks on unseen real faces from the FRGCv2 database with 3mm precision. Furthermore, our deformable model fitting algorithm achieved 98.5% face recognition accuracy on the FRGCv2 and 98.6% on Bosphorus database. Our dense model is also able to generalize to unseen datasets.Comment: 24 Pages, 12 Figures, 6 Tables and 3 Algorithm

The assessment of distorted facial muscles movements in facial palsy

Introduction The clinical evaluation of facial palsy remains the routine approach for the assessment of facial muscle movements. However, there is a lack of data to link the mathematical analysis of 3D dynamic facial morphology with the subjective clinical assessments. Quantifying the degree of distortion of facial expressions is a vital step in evaluating the clinical impact of facial palsy. 4D imaging is a reliable modality for recording the dynamics of facial expressions. This study aimed to assess distorted facial muscles movements in unilateral facial palsy and mathematically validate clinical grading indices. Material & Method The study recruited 50 patients who suffered from unilateral facial palsy and a control group of an equal number (50) of age- and sex-matched cases. The dynamics of facial expressions were captured using a stereophotogrammetric 4D imaging system. Six facial expressions were recorded (rest, maximum smile, cheek puff, lip purse, eyebrow-raising, eye closure), each one took 4 seconds and generated about 240 3D images for analysis. An advanced geometric morphometric approach using Dense Surface Models was applied for the mathematical quantification of the 3D facial dysmorphology over time. The asymmetries of 10 facial anatomical regions were calculated. For each participant, six mathematical values which quantify asymmetry were measured per expression (the minimal, mean, median, maximum, range, and standard deviation). The 4D image data of sixteen facial paralysis patients were assessed by 7 expert assessors using two clinical grading indices for the assessment of unilateral facial palsy, the modified Sunnybrook index, and the Glasgow Index. The reproducibility of the clinical gradings between two rating sessions was examined. The measured asymmetries of the 4D images were treated as the gold standard to evaluate the performance of the subjective grading indices. Cross-correlations between the mathematical measurements and the subjective grades were calculated. The Modified Sunnybrook index assessed 8 parameters (3 at rest and 5 at individual facial expression). The Glasgow index assessed 29 parameters for the assessment of dynamic facial abnormalities with considerations for the directionality and severity of asymmetry. The similarities and dissimilarities between the two clinical assessments and to the mathematical measurements were investigated. Results The modified Sunnybrook index was reproducible for grading the dysmorphology and dysfunction of unilateral facial paralysis. The Glasgow Index was reproducible after excluding three parameters of poor reproducibility. The modified Sunnybrook index and the Glasgow index correlated reasonably well with the mathematical measurements of facial asymmetry at rest and with facial expressions. • The minimal value of facial asymmetries of the rest expression had a stronger correlation coefficient than that of other values. • The mean and median values of facial asymmetries of the other five nonverbal expressions had a stronger correlation coefficient than that of other values. The following were the main regions affected by facial dysmorphology which showed a correlation above -0.6 between the subjective and objective assessments: • The full face at rest as well as the forehead, cheek, nose and nasolabial, upper lip, corner of the mouth, and chin regions. • The full face, cheek, nasolabial, upper lip, and lower lip of the smile. • The full face, upper and lower lips of the lip purse. • Most of the facial regions, except the cheek, showed moderate to weak correlations with cheek puff. • A strong correlation was detected between the subjective and objective assessments of the forehead and eye regions with eye closure. Based on the correlation results between the mathematical measurements and clinical evaluation of facial asymmetry in unilateral facial paralysis, the study highlighted the following points: • Smile expression: the assessors encountered difficulties to judge the direction of the asymmetry of the corner of the mouth. It is easier to observe the upper lip and the cheek instead of the corner of the mouth when assessing the smile. • Lip purse: the evaluation of the directionality of lip movement was more accurate and sensitive at the lower lip. • Cheek puff: grading the cheek may not grasp the severity of the asymmetry. We would suggest observing the corner of the mouth and lower lip in cheek puff expressions. • Eyebrow raising expression: grading the 4D movement of the upper margin of the eyebrow may be more sensitive than depending on the assessment of the wrinkles of the forehead. • Eye closure: the clinical assessment of the eyes based on 4D image data was not ideal due to the 4D imaging surface defects secondary to the reflective surface of the cornea. Conclusion The mathematical assessment of the dynamics of facial expressions in unilateral facial palsy using advanced geometric morphometrics provides a state-of-art approach for the quantification and visualization of facial dysmorphology. The Glasgow Index and the Modified Sunnybrook Index were reproducible. The clinical assessors were reasonably consistent in the grading of facial palsy. The significant correlations between the clinical grading of facial palsy and the mathematical calculation of the same facial muscle movements provided satisfactory evidence of objectivity to the clinical assessments. The Glasgow index provided more validated parameters for the assessment of facial palsy in comparison to the modified Sunnybrook index. The mathematical quantification of the 3D facial dysmorphology and the associated dynamic asymmetry provides invaluable information to complement the clinical assessments. This is particularly important for the assessment of regional asymmetries and the directionality of the asymmetry for the evaluation of facial contour (anteroposterior direction), face drooping (vertical direction), especially in cases where surgical rehabilitation is indicated

The correlation between static and dynamic facial dysmorphology in unilateral cleft lip and palate (UCLP)

Objective – The objective of this study was to assess the correlation between static (3D) facial dysmorphology at both rest and maximum smile and dynamic (4D) facial dysmorphology as represented by the frame of maximum asymmetry during a maximum smile in unilateral cleft lip and palate (UCLP). When maximum asymmetry occurs during a maximum smile was also analysed. Design and setting – This study was designed as a retrospective cross-sectional study utilising quantitative methodology to assess a cohort of patients with a diagnosis of non-syndromic unilateral cleft lip and palate (UCLP). Patients had 4D imaging carried out at Glasgow Dental Hospital and School for audit purposes and as part of their routine care by the local cleft team working under the Managed Clinical Network (MCN) Cleft Care Scotland. Materials and Methods – Thirty-one (31) patients between the ages of 13 to 17 years old with a diagnosis of non-syndromic unilateral cleft lip and palate (UCLP) had 4D images based on passive stereophotogrammetry captured at 60 frames per second (fps). Each patient was captured performing a maximum smile three times over a three second window. Data processing involved the conformation of a generic mesh containing over 7000 vertices or quasi-landmarks to the facial surface before tracking all the vertices during the facial expression. Partial ordinary Procrustes analysis was utilised to calculate an asymmetry score for each frame during the expression. The rest frame (3D), frame of maximum smile (3D), and frame of maximum asymmetry (4D) were used to determine the correlation between static and dynamic facial dysmorphology. Results – Asymmetry scores were higher at maximum smile than at rest and maximum asymmetry was most frequently observed during the relaxation phase following a maximum facial expression (N=27/31; 87.1%). Asymmetry scores for maximum smile were less than but comparable to maximum asymmetry scores. Static (3D) asymmetry at rest and maximum smile was strongly correlated to dynamic (4D) facial asymmetry represented by the frame of maximum asymmetry during a maximum smile. The strongest correlation was seen with analysis using the frame of maximum smile focussing on the nasolabial region. Conclusions – The use of 4D imaging combined with mesh conformation and dense correspondence analysis provides a valid objective measure of facial asymmetry. The fact that asymmetry scores for maximum smile were less than but comparable to maximum asymmetry scores highlights that assessment of facial asymmetry at maximum smile is still a valid assessment of facial dysmorphology despite not depicting the full extent of the asymmetry. Static (3D) asymmetry at rest and maximum smile is strongly correlated to; and likely highly predictive of, dynamic (4D) facial asymmetry represented by the frame of maximum asymmetry during a maximum smile. The strongest correlation is seen with analysis using the frame of maximum smile focussing on the nasolabial region. Future research could use linear regression modelling to predict dynamic (4D) asymmetry scores using static (3D) images

Image-based Detection of Neuro-facial Differences in Foetal Alcohol Spectrum Disorders

Prenatal exposure to alcohol remains as one of the leading, yet preventable, causes of birth defects and neurodevelopmental disorders in the Western world. Over the past 50 years, since the first documented report on the impact of in utero alcohol exposure, a broad spectrum of associated effects have been recognised. Foetal alcohol spectrum disorders is the collective term encompassing a range of diagnostic classifications that can be identified. At the most severe end of this spectrum are foetal alcohol syndrome (FAS), recognisable by a characteristic set of facial features, growth delay, neurocognitive deficit, and behavioural impairments. Criteria for either of these diagnostic categories typically requires at least two ‘cardinal’ facial features: short palpebral fissure length; thin upper lip-vermillion; and, a smooth philtrum. Methods for identifying these features typically rely on subjective observation. This subjectivity means that accuracy of diagnosis is reliant on the skill and experience of the clinician. However, the main clinical challenges arise when an individual presents with confirmed or suspected prenatal alcohol exposure, but without the facial criteria required for FAS diagnoses. These individuals make up the vast majority of the affected population, and clinical recognition can be extremely challenging. Identification and recognition of facial features associated with prenatal alcohol exposure remain a key area of study. This thesis establishes a novel perspective on the issue of subjective clinical assessment and recognition using 3D face and brain shape analysis. We utilise data from 3D facial imaging, MRI brain images and neurocognitive measures to assess subtle facial differences, face-brain associations and the relationships between face, brain and cognition. Development of innovative techniques and methodologies have allowed us to develop a set of analysis tools applicable to craniofacial assessment, and potentially contribute to the analysis of other facially affected conditions in both clinical and research environments

3D Face Reconstruction from Light Field Images: A Model-free Approach

Reconstructing 3D facial geometry from a single RGB image has recently instigated wide research interest. However, it is still an ill-posed problem and most methods rely on prior models hence undermining the accuracy of the recovered 3D faces. In this paper, we exploit the Epipolar Plane Images (EPI) obtained from light field cameras and learn CNN models that recover horizontal and vertical 3D facial curves from the respective horizontal and vertical EPIs. Our 3D face reconstruction network (FaceLFnet) comprises a densely connected architecture to learn accurate 3D facial curves from low resolution EPIs. To train the proposed FaceLFnets from scratch, we synthesize photo-realistic light field images from 3D facial scans. The curve by curve 3D face estimation approach allows the networks to learn from only 14K images of 80 identities, which still comprises over 11 Million EPIs/curves. The estimated facial curves are merged into a single pointcloud to which a surface is fitted to get the final 3D face. Our method is model-free, requires only a few training samples to learn FaceLFnet and can reconstruct 3D faces with high accuracy from single light field images under varying poses, expressions and lighting conditions. Comparison on the BU-3DFE and BU-4DFE datasets show that our method reduces reconstruction errors by over 20% compared to recent state of the art

Facial Curvature Detects and Explicates Ethnic Differences in Effects of Prenatal Alcohol Exposure

Background Our objective is to help clinicians detect the facial effects of prenatal alcohol exposure by developing computer-based tools for screening facial form. Methods All 415 individuals considered were evaluated by expert dysmorphologists and categorized as (i) healthy control (HC), (ii) fetal alcohol syndrome (FAS), or (iii) heavily prenatally alcohol exposed (HE) but not clinically diagnosable as FAS; 3D facial photographs were used to build models of facial form to support discrimination studies. Surface curvature-based delineations of facial form were introduced. Results (i) Facial growth in FAS, HE, and control subgroups is similar in both cohorts. (ii) Cohort consistency of agreement between clinical diagnosis and HC-FAS facial form classification is lower for midline facial regions and higher for nonmidline regions. (iii) Specific HC-FAS differences within and between the cohorts include: for HC, a smoother philtrum in Cape Coloured individuals; for FAS, a smoother philtrum in Caucasians; for control-FAS philtrum difference, greater homogeneity in Caucasians; for control-FAS face difference, greater homogeneity in Cape Coloured individuals. (iv) Curvature changes in facial profile induced by prenatal alcohol exposure are more homogeneous and greater in Cape Coloureds than in Caucasians. (v) The Caucasian HE subset divides into clusters with control-like and FAS-like facial dysmorphism. The Cape Coloured HE subset is similarly divided for nonmidline facial regions but not clearly for midline structures. (vi) The Cape Coloured HE subset with control-like facial dysmorphism shows orbital hypertelorism. Conclusions Facial curvature assists the recognition of the effects of prenatal alcohol exposure and helps explain why different facial regions result in inconsistent control-FAS discrimination rates in disparate ethnic groups. Heavy prenatal alcohol exposure can give rise to orbital hypertelorism, supporting a long-standing suggestion that prenatal alcohol exposure at a particular time causes increased separation of the brain hemispheres with a concomitant increase in orbital separation