Facial Paralysis Grading Based on Dynamic and Static Features

Peripheral facial nerve palsy, also known as facial paralysis (FP), is a common clinical disease, which requires subjective judgment and scoring based on the FP scale. There exists some automatic facial paralysis grading methods, but the current methods mostly only consider either static or dynamic features, resulting in a low accuracy rate of FP grading. This thesis proposes an automatic facial paralysis assessment method including both static and dynamic characteristics. The first step of the method performs preprocessing on the collected facial expression videos of the subjects, including rough video interception, video stabilization, keyframe extraction, image geometric normalization and gray-scale normalization. Next, the method selects as keyframes no facial expression state and maximum facial expression state in the image data to build the the research data set. Data preprocessing reduces errors, noise, redundancy and even errors in the original data. The basis for extracting static and dynamic features of an image is to use Ensemble of Regression Trees algorithm to determine 68 facial landmarks. Based on landmark points, image regions of image are formed. According to the Horn-Schunck optical flow method, the optical flow information of parts of the face are extracted, and the dynamic characteristics of the optical flow difference between the left and right parts are calculated. Finally, the results of dynamic and static feature classification are weighted and analyzed to obtain FP ratings of subjects. A 32-dimensional static feature is fed into the support vector machine for classification. A 60-dimensional feature vector of dynamical aspects is fed into a long and short-term memory network for classification. Videos of 30 subjects are used to extract 1419 keyframes to test the algorithm. The accuracy, precision, recall and f1 of the best classifier reach 93.33%, 94.29%, 91.33% and 91.87%, respectively.Perifeerinen kasvojen hermohalvaus, joka tunnetaan myös nimellä kasvojen halvaus (FP), on yleinen kliininen sairaus, joka vaatii subjektiivista arviointia ja FP -asteikon pisteytystä. Joitakin automaattisia kasvohalvauksen luokittelumenetelmiä on olemassa, mutta yleensä ottaen ne punnitsevat vain joko staattisia tai dynaamisia piirteitä. Tässä tutkielmassa ehdotetaan automaattista kasvojen halvaantumisen arviointimenetelmää, joka kattaa sekä staattiset että dynaamiset ominaisuudet. Menetelmän ensimmäinen vaihe suorittaa ensin esikäsittelyn kohteiden kerätyille kasvojen ilmevideoille, mukaan lukien karkea videon sieppaus, videon vakautus, avainruudun poiminta, kuvan geometrinen normalisointi ja harmaasävyjen normalisointi. Seuraavaksi menetelmä valitsee avainruuduiksi ilmeettömän tilan ja kasvojen ilmeiden maksimitilan kuvadatasta kerryttäen tutkimuksen data-aineiston. Tietojen esikäsittely vähentää virheitä, kohinaa, redundanssia ja jopa virheitä alkuperäisestä datasta. Kuvan staattisten ja dynaamisten piirteiden poimimisen perusta on käyttää Ensemble of Regression Trees -algoritmia 68 kasvojen merkkipisteiden määrittämiseen. Merkkipisteiden perusteella määritellään kuvan kiinnostavat alueet. Horn-Schunckin optisen virtausmenetelmän mukaisesti poimitaan optisen virtauksen tiedot joistakin kasvojen osista, ja dynaaminen luonnehdinta lasketaan vasempien ja oikeiden osien välille. Lopuksi dynaamisen ja staattisen piirteiden luokittelun tulokset painotetaan ja analysoidaan kattavasti koehenkilöiden FP-luokitusten saamiseksi. 32- ulotteinen staattisten piirteiden vektori syötetään tukivektorikoneeseen luokittelua varten. 60-ulotteinen dynaamisten piirteiden ominaisuusvektori syötetään pitkän ja lyhyen aikavälin muistiverkkoon luokittelua varten. Parhaan luokittelijan tarkkuus, täsmällisyys, palautustaso ja f1 saavuttavat arvot 93,33%, 94,29%, 91,33% ja 91,87%

The assessment of distorted facial muscles movements in facial palsy

Introduction The clinical evaluation of facial palsy remains the routine approach for the assessment of facial muscle movements. However, there is a lack of data to link the mathematical analysis of 3D dynamic facial morphology with the subjective clinical assessments. Quantifying the degree of distortion of facial expressions is a vital step in evaluating the clinical impact of facial palsy. 4D imaging is a reliable modality for recording the dynamics of facial expressions. This study aimed to assess distorted facial muscles movements in unilateral facial palsy and mathematically validate clinical grading indices. Material & Method The study recruited 50 patients who suffered from unilateral facial palsy and a control group of an equal number (50) of age- and sex-matched cases. The dynamics of facial expressions were captured using a stereophotogrammetric 4D imaging system. Six facial expressions were recorded (rest, maximum smile, cheek puff, lip purse, eyebrow-raising, eye closure), each one took 4 seconds and generated about 240 3D images for analysis. An advanced geometric morphometric approach using Dense Surface Models was applied for the mathematical quantification of the 3D facial dysmorphology over time. The asymmetries of 10 facial anatomical regions were calculated. For each participant, six mathematical values which quantify asymmetry were measured per expression (the minimal, mean, median, maximum, range, and standard deviation). The 4D image data of sixteen facial paralysis patients were assessed by 7 expert assessors using two clinical grading indices for the assessment of unilateral facial palsy, the modified Sunnybrook index, and the Glasgow Index. The reproducibility of the clinical gradings between two rating sessions was examined. The measured asymmetries of the 4D images were treated as the gold standard to evaluate the performance of the subjective grading indices. Cross-correlations between the mathematical measurements and the subjective grades were calculated. The Modified Sunnybrook index assessed 8 parameters (3 at rest and 5 at individual facial expression). The Glasgow index assessed 29 parameters for the assessment of dynamic facial abnormalities with considerations for the directionality and severity of asymmetry. The similarities and dissimilarities between the two clinical assessments and to the mathematical measurements were investigated. Results The modified Sunnybrook index was reproducible for grading the dysmorphology and dysfunction of unilateral facial paralysis. The Glasgow Index was reproducible after excluding three parameters of poor reproducibility. The modified Sunnybrook index and the Glasgow index correlated reasonably well with the mathematical measurements of facial asymmetry at rest and with facial expressions. • The minimal value of facial asymmetries of the rest expression had a stronger correlation coefficient than that of other values. • The mean and median values of facial asymmetries of the other five nonverbal expressions had a stronger correlation coefficient than that of other values. The following were the main regions affected by facial dysmorphology which showed a correlation above -0.6 between the subjective and objective assessments: • The full face at rest as well as the forehead, cheek, nose and nasolabial, upper lip, corner of the mouth, and chin regions. • The full face, cheek, nasolabial, upper lip, and lower lip of the smile. • The full face, upper and lower lips of the lip purse. • Most of the facial regions, except the cheek, showed moderate to weak correlations with cheek puff. • A strong correlation was detected between the subjective and objective assessments of the forehead and eye regions with eye closure. Based on the correlation results between the mathematical measurements and clinical evaluation of facial asymmetry in unilateral facial paralysis, the study highlighted the following points: • Smile expression: the assessors encountered difficulties to judge the direction of the asymmetry of the corner of the mouth. It is easier to observe the upper lip and the cheek instead of the corner of the mouth when assessing the smile. • Lip purse: the evaluation of the directionality of lip movement was more accurate and sensitive at the lower lip. • Cheek puff: grading the cheek may not grasp the severity of the asymmetry. We would suggest observing the corner of the mouth and lower lip in cheek puff expressions. • Eyebrow raising expression: grading the 4D movement of the upper margin of the eyebrow may be more sensitive than depending on the assessment of the wrinkles of the forehead. • Eye closure: the clinical assessment of the eyes based on 4D image data was not ideal due to the 4D imaging surface defects secondary to the reflective surface of the cornea. Conclusion The mathematical assessment of the dynamics of facial expressions in unilateral facial palsy using advanced geometric morphometrics provides a state-of-art approach for the quantification and visualization of facial dysmorphology. The Glasgow Index and the Modified Sunnybrook Index were reproducible. The clinical assessors were reasonably consistent in the grading of facial palsy. The significant correlations between the clinical grading of facial palsy and the mathematical calculation of the same facial muscle movements provided satisfactory evidence of objectivity to the clinical assessments. The Glasgow index provided more validated parameters for the assessment of facial palsy in comparison to the modified Sunnybrook index. The mathematical quantification of the 3D facial dysmorphology and the associated dynamic asymmetry provides invaluable information to complement the clinical assessments. This is particularly important for the assessment of regional asymmetries and the directionality of the asymmetry for the evaluation of facial contour (anteroposterior direction), face drooping (vertical direction), especially in cases where surgical rehabilitation is indicated

3D facial model analysis for clinical medicine

Ph.DDOCTOR OF PHILOSOPH

Validation of the automatic tracking for facial landmarks in 3D motion captured images

Aim: The aim of this study was to validate the automatic tracking of facial landmarks in 3D image sequences captured using the Di4D system (Dimensional Imaging Ltd., Glasgow, UK). MATERIALS AND METHODS: 32 subjects (16 males; 16 females) range 18-35 years were recruited. 23 facial landmarks were marked on the face of each subject with a 0.5 mm non-permanent ink. The subjects were asked to perform three facial animations from the rest position (maximal smile, lip purse and cheek puff). Each animation was captured by a 3D stereophotogrammetry video system (Di4D). A single operator digitized landmarks on captured 3D models and the manual digitised landmarks were compared with the automatic tracked landmarks. To investigate the accuracy of manual digitisation, the same operator re-digitized 2 subjects (1 male and 1 female). RESULTS & CONCLUSION: The discrepancies in x, y and z coordinates between the manual digitised landmarks and the automatic tracked facial landmarks were within 0.5 mm and the mean distance between the manual digitisation and the automatic tracking of corresponding landmarks using tracking software was within 0.7 mm which reflects the accuracy of the method( p value was very small). The majority of these distances were within 1 mm. The correlation coefficient between the manual and the automatic tracking of facial landmarks was 0.999 in all x, y, and z coordinates. In conclusion, Automatic tracking of facial landmarks with satisfactory accuracy, would facilitate the analysis of the dynamic motion during facial animations

Deep human face analysis and modelling

Human face appearance and motion play a significant role in creating the complex social environments of human civilisation. Humans possess the capacity to perform facial analysis and come to conclusion such as the identity of individuals, understanding emotional state and diagnosing diseases. The capacity though is not universal for the entire population, where there are medical conditions such prosopagnosia and autism which can directly affect facial analysis capabilities of individuals, while other facial analysis tasks require specific traits and training to perform well. This has lead to the research of facial analysis systems within the computer vision and machine learning fields over the previous decades, where the aim is to automate many facial analysis tasks to a level similar or surpassing humans. While breakthroughs have been made in certain tasks with the emergence of deep learning methods in the recent years, new state-of-the-art results have been achieved in many computer vision and machine learning tasks. Within this thesis an investigation into the use of deep learning based methods for facial analysis systems takes place, following a review of the literature specific facial analysis tasks, methods and challenges are found which form the basis for the research findings presented. The research presented within this thesis focuses on the tasks of face detection and facial symmetry analysis specifically for the medical condition facial palsy. Firstly an initial approach to face detection and symmetry analysis is proposed using a unified multi-task Faster R-CNN framework, this method presents good accuracy on the test data sets for both tasks but also demonstrates limitations from which the remaining chapters take their inspiration. Next the Integrated Deep Model is proposed for the tasks of face detection and landmark localisation, with specific focus on false positive face detection reduction which is crucial for accurate facial feature extraction in the medical applications studied within this thesis. Evaluation of the method on the Face Detection Dataset and Benchmark and Annotated Faces in-the-Wild benchmark data sets shows a significant increase of over 50% in precision against other state-of-the-art face detection methods, while retaining a high level of recall. The task of facial symmetry and facial palsy grading are the focus of the finals chapters where both geometry-based symmetry features and 3D CNNs are applied. It is found through evaluation that both methods have validity in the grading of facial palsy. The 3D CNNs are the most accurate with an F1 score of 0.88. 3D CNNs are also capable of recognising mouth motion for both those with and without facial palsy with an F1 score of 0.82