Generation, Estimation and Tracking of Faces

This thesis describes techniques for the construction of face models for both computer graphics and computer vision applications. It also details model-based computer vision methods for extracting and combining data with the model. Our face models respect the measurements of populations described by face anthropometry studies. In computer graphics, the anthropometric measurements permit the automatic generation of varied geometric models of human faces. This is accomplished by producing a random set of face measurements generated according to anthropometric statistics. A face fitting these measurements is realized using variational modeling. In computer vision, anthropometric data biases face shape estimation towards more plausible individuals. Having such a detailed model encourages the use of model-based techniques—we use a physics-based deformable model framework. We derive and solve a dynamic system which accounts for edges in the image and incorporates optical flow as a motion constraint on the model. Our solution ensures this constraint remains satisfied when edge information is used, which helps prevent tracking drift. This method is extended using the residuals from the optical flow solution. The extracted structure of the model can be improved by determining small changes in the model that reduce this error residual. We present experiments in extracting the shape and motion of a face from image sequences which exhibit the generality of our technique, as well as provide validation

Optical Flow Constraints on Deformable Models With Applications to Face Tracking

Optical flow provides a constraint on the motion of a deformable model. We derive and solve a dynamic system incorporating flow as a hard constraint, producing a model-based least-squares optical flow solution. Our solution also ensures the constraint remains satisfied when combined with edge information, which helps combat tracking error accumulation. Constraint enforcement can be relaxed using a Kalman filter, which permits controlled constraint violations based on the noise present in the optical flow information, and enables optical flow and edge information to be combined more robustly and efficiently. We apply this framework to the estimation of face shape and motion using a 3D deformable face model. This model uses a small number of parameters to describe a rich variety of face shapes and facial expressions. We present experiments in extracting the shape and motion of a face from image sequences which validate the accuracy of the method. They also demonstrate that our treatment of optical flow as a hard constraint, as well as our use of a Kalman filter to reconcile these constraints with the uncertainty in the optical flow, are vital for improving the performance of our system

Three-dimensional assessment of dentofacial deformity in children with clefts

Background: Changes in clinical management; advances in non-invasive three-dimensional imaging; developments in methods of shape analysis. Aim: To assess three-dimensional dentofacial deformity with a view to early appraisal of primary surgical outcome. Results: Significant differences in upper lip morphology were found between the cleft children and their unaffected peers; nasal asymmetry that became more obvious in function was noted in cleft children; the maxillary dental arches of the children with repaired cleft palate were shallow, short and narrow; and the dental arch, deformity and the facial soft tissue deformity were unrelated. Contributions to the field: It has been shown that deviation from normal could be detected as young as 3 years of age using computerised stereophotogrammetry; preliminary, objective, three-dimensional analysis of facial function has been completed in young children; the accuracy of three-dimensional CT scanning of dentate study models and the time cost of data collection were quantified; and this study has produced a body of three-dimensional data that can test and support analytical advances

Timing is everything: A spatio-temporal approach to the analysis of facial actions

This thesis presents a fully automatic facial expression analysis system based on the Facial Action Coding System (FACS). FACS is the best known and the most commonly used system to describe facial activity in terms of facial muscle actions (i.e., action units, AUs). We will present our research on the analysis of the morphological, spatio-temporal and behavioural aspects of facial expressions. In contrast with most other researchers in the field who use appearance based techniques, we use a geometric feature based approach. We will argue that that approach is more suitable for analysing facial expression temporal dynamics. Our system is capable of explicitly exploring the temporal aspects of facial expressions from an input colour video in terms of their onset (start), apex (peak) and offset (end). The fully automatic system presented here detects 20 facial points in the first frame and tracks them throughout the video. From the tracked points we compute geometry-based features which serve as the input to the remainder of our systems. The AU activation detection system uses GentleBoost feature selection and a Support Vector Machine (SVM) classifier to find which AUs were present in an expression. Temporal dynamics of active AUs are recognised by a hybrid GentleBoost-SVM-Hidden Markov model classifier. The system is capable of analysing 23 out of 27 existing AUs with high accuracy. The main contributions of the work presented in this thesis are the following: we have created a method for fully automatic AU analysis with state-of-the-art recognition results. We have proposed for the first time a method for recognition of the four temporal phases of an AU. We have build the largest comprehensive database of facial expressions to date. We also present for the first time in the literature two studies for automatic distinction between posed and spontaneous expressions

Over time RF fitting for Jitter Free 3D Vertebra Reconstruction from Video Fluoroscopy

Over the past decades, there has been an increasing interest in spine kinematics. Various approaches have been proposed on how to observe and analyse spine kinematics from a computer vision perspective. Amongst all, emphasis has been given to both the shape of the individual vertebrae as well as the overall spine curvature as a means of providing accurate and valid spinal condition diagnosis. Traditional invasive methods cannot accurately delineate the intersegmental motion of the spine vertebrae. On the contrary, capturing and measuring spinal motion via the non-invasive fluoroscopy has been a popular technique choice because of its low incurred patient radiation exposure nature. In general, image-based and other reconstruction methods target individual frames and focus on static spine instances. However, even the ones analysing sequences yield in unstable and jittery animations of the reconstructed spine. In this report, we address this issue using a novel approach to robustly reconstruct and rigidly derive a shape with no inter-frame variations. This is to produce animations that are jitter free across our sequence based on fluoroscopy video. Our main contributions are 1) retaining the shape of the solid vertebrae across the frame range, 2) helping towards a more accurate image segmentation even when there's a limited training set. We show our pipeline's success by reconstructing and comparing 3D animations of the lumbar spine from a corresponding fluoroscopic video

The assessment of distorted facial muscles movements in facial palsy

Introduction The clinical evaluation of facial palsy remains the routine approach for the assessment of facial muscle movements. However, there is a lack of data to link the mathematical analysis of 3D dynamic facial morphology with the subjective clinical assessments. Quantifying the degree of distortion of facial expressions is a vital step in evaluating the clinical impact of facial palsy. 4D imaging is a reliable modality for recording the dynamics of facial expressions. This study aimed to assess distorted facial muscles movements in unilateral facial palsy and mathematically validate clinical grading indices. Material & Method The study recruited 50 patients who suffered from unilateral facial palsy and a control group of an equal number (50) of age- and sex-matched cases. The dynamics of facial expressions were captured using a stereophotogrammetric 4D imaging system. Six facial expressions were recorded (rest, maximum smile, cheek puff, lip purse, eyebrow-raising, eye closure), each one took 4 seconds and generated about 240 3D images for analysis. An advanced geometric morphometric approach using Dense Surface Models was applied for the mathematical quantification of the 3D facial dysmorphology over time. The asymmetries of 10 facial anatomical regions were calculated. For each participant, six mathematical values which quantify asymmetry were measured per expression (the minimal, mean, median, maximum, range, and standard deviation). The 4D image data of sixteen facial paralysis patients were assessed by 7 expert assessors using two clinical grading indices for the assessment of unilateral facial palsy, the modified Sunnybrook index, and the Glasgow Index. The reproducibility of the clinical gradings between two rating sessions was examined. The measured asymmetries of the 4D images were treated as the gold standard to evaluate the performance of the subjective grading indices. Cross-correlations between the mathematical measurements and the subjective grades were calculated. The Modified Sunnybrook index assessed 8 parameters (3 at rest and 5 at individual facial expression). The Glasgow index assessed 29 parameters for the assessment of dynamic facial abnormalities with considerations for the directionality and severity of asymmetry. The similarities and dissimilarities between the two clinical assessments and to the mathematical measurements were investigated. Results The modified Sunnybrook index was reproducible for grading the dysmorphology and dysfunction of unilateral facial paralysis. The Glasgow Index was reproducible after excluding three parameters of poor reproducibility. The modified Sunnybrook index and the Glasgow index correlated reasonably well with the mathematical measurements of facial asymmetry at rest and with facial expressions. • The minimal value of facial asymmetries of the rest expression had a stronger correlation coefficient than that of other values. • The mean and median values of facial asymmetries of the other five nonverbal expressions had a stronger correlation coefficient than that of other values. The following were the main regions affected by facial dysmorphology which showed a correlation above -0.6 between the subjective and objective assessments: • The full face at rest as well as the forehead, cheek, nose and nasolabial, upper lip, corner of the mouth, and chin regions. • The full face, cheek, nasolabial, upper lip, and lower lip of the smile. • The full face, upper and lower lips of the lip purse. • Most of the facial regions, except the cheek, showed moderate to weak correlations with cheek puff. • A strong correlation was detected between the subjective and objective assessments of the forehead and eye regions with eye closure. Based on the correlation results between the mathematical measurements and clinical evaluation of facial asymmetry in unilateral facial paralysis, the study highlighted the following points: • Smile expression: the assessors encountered difficulties to judge the direction of the asymmetry of the corner of the mouth. It is easier to observe the upper lip and the cheek instead of the corner of the mouth when assessing the smile. • Lip purse: the evaluation of the directionality of lip movement was more accurate and sensitive at the lower lip. • Cheek puff: grading the cheek may not grasp the severity of the asymmetry. We would suggest observing the corner of the mouth and lower lip in cheek puff expressions. • Eyebrow raising expression: grading the 4D movement of the upper margin of the eyebrow may be more sensitive than depending on the assessment of the wrinkles of the forehead. • Eye closure: the clinical assessment of the eyes based on 4D image data was not ideal due to the 4D imaging surface defects secondary to the reflective surface of the cornea. Conclusion The mathematical assessment of the dynamics of facial expressions in unilateral facial palsy using advanced geometric morphometrics provides a state-of-art approach for the quantification and visualization of facial dysmorphology. The Glasgow Index and the Modified Sunnybrook Index were reproducible. The clinical assessors were reasonably consistent in the grading of facial palsy. The significant correlations between the clinical grading of facial palsy and the mathematical calculation of the same facial muscle movements provided satisfactory evidence of objectivity to the clinical assessments. The Glasgow index provided more validated parameters for the assessment of facial palsy in comparison to the modified Sunnybrook index. The mathematical quantification of the 3D facial dysmorphology and the associated dynamic asymmetry provides invaluable information to complement the clinical assessments. This is particularly important for the assessment of regional asymmetries and the directionality of the asymmetry for the evaluation of facial contour (anteroposterior direction), face drooping (vertical direction), especially in cases where surgical rehabilitation is indicated

Aerospace medicine and biology: A continuing bibliography with indexes (supplement 341)

This bibliography lists 133 reports, articles and other documents introduced into the NASA Scientific and Technical Information System during September 1990. Subject coverage includes: aerospace medicine and psychology, life support systems and controlled environments, safety equipment, exobiology and extraterrestrial life, and flight crew behavior and performance

Development of the components of a low cost, distributed facial virtual conferencing system

This thesis investigates the development of a low cost, component based facial virtual conferencing system. The design is decomposed into an encoding phase and a decoding phase, which communicate with each other via a network connection. The encoding phase is composed of three components: model acquisition (which handles avatar generation), pose estimation and expression analysis. Audio is not considered part of the encoding and decoding process, and as such is not evaluated. The model acquisition component is implemented using a visual hull reconstruction algorithm that is able to reconstruct real-world objects using only sets of images of the object as input. The object to be reconstructed is assumed to lie in a bounding volume of voxels. The reconstruction process involves the following stages: - Space carving for basic shape extraction; - Isosurface extraction to remove voxels not part of the surface of the reconstruction; - Mesh connection to generate a closed, connected polyhedral mesh; - Texture generation. Texturing is achieved by Gouraud shading the reconstruction with a vertex colour map; - Mesh decimation to simplify the object. The original algorithm has complexity O(n), but suffers from an inability to reconstruct concave surfaces that do not form part of the visual hull of the object. A novel extension to this algorithm based on Normalised Cross Correlation (NCC) is proposed to overcome this problem. An extension to speed up traditional NCC evaluations is proposed which reduces the NCC search space from a 2D search problem down to a single evaluation. Pose estimation and expression analysis are performed by tracking six fiducial points on the face of a subject. A tracking algorithm is developed that uses Normalised Cross Correlation to facilitate robust tracking that is invariant to changing lighting conditions, rotations and scaling. Pose estimation involves the recovery of the head position and orientation through the tracking of the triangle formed by the subject's eyebrows and nose tip. A rule-based evaluation of points that are tracked around the subject's mouth forms the basis of the expression analysis. A user assisted feedback loop and caching mechanism is used to overcome tracking errors due to fast motion or occlusions. The NCC tracker is shown to achieve a tracking performance of 10 fps when tracking the six fiducial points. The decoding phase is divided into 3 tasks, namely: avatar movement, expression generation and expression management. Avatar movement is implemented using the base VR system. Expression generation is facilitated using a Vertex Interpolation Deformation method. A weighting system is proposed for expression management. Its function is to gradually transform from one expression to the next. The use of the vertex interpolation method allows real-time deformations of the avatar representation, achieving 16 fps when applied to a model consisting of 7500 vertices. An Expression Parameter Lookup Table (EPLT) facilitates an independent mapping between the two phases. It defines a list of generic expressions that are known to the system and associates an Expression ID with each one. For each generic expression, it relates the expression analysis rules for any subject with the expression generation parameters for any avatar model. The result is that facial expression replication between any subject and avatar combination can be performed by transferring only the Expression ID from the encoder application to the decoder application. The ideas developed in the thesis are demonstrated in an implementation using the CoRgi Virtual Reality system. It is shown that the virtual-conferencing application based on this design requires only a bandwidth of 2 Kbps.Adobe Acrobat Pro 9.4.6Adobe Acrobat 9.46 Paper Capture Plug-i