Automatic 3D Facial Performance Acquisition and Animation using Monocular Videos

Facial performance capture and animation is an essential component of many applications such as movies, video games, and virtual environments. Video-based facial performance capture is particularly appealing as it offers the lowest cost and the potential use of legacy sources and uncontrolled videos. However, it is also challenging because of complex facial movements at different scales, ambiguity caused by the loss of depth information, and a lack of discernible features on most facial regions. Unknown lighting conditions and camera parameters further complicate the problem. This dissertation explores the video-based 3D facial performance capture systems that use a single video camera, overcome the challenges aforementioned, and produce accurate and robust reconstruction results. We first develop a novel automatic facial feature detection/tracking algorithm that accurately locates important facial features across the entire video sequence, which are then used for 3D pose and facial shape reconstruction. The key idea is to combine the respective powers of local detection, spatial priors for facial feature locations, Active Appearance Models (AAMs), and temporal coherence for facial feature detection. The algorithm runs in realtime and is robust to large pose and expression variations and occlusions. We then present an automatic high-fidelity facial performance capture system that works on monocular videos. It uses the detected facial features along with multilinear facial models to reconstruct 3D head poses and large-scale facial deformation, and uses per-pixel shading cues to add fine-scale surface details such as emerging or disappearing wrinkles and folds. We iterate the reconstruction procedure on large-scale facial geometry and fine-scale facial details to improve the accuracy of facial reconstruction. We further improve the accuracy and efficiency of the large-scale facial performance capture by introducing a local binary feature based 2D feature regression and a convolutional neural network based pose and expression regression, and complement it with an efficient 3D eye gaze tracker to achieve realtime 3D eye gaze animation. We have tested our systems on various monocular videos, demonstrating the accuracy and robustness under a variety of uncontrolled lighting conditions and overcoming significant shape differences across individuals

Statistical Modeling of Craniofacial Shape and Texture

We present a fully-automatic statistical 3D shape modeling approach and apply it to a large dataset of 3D images, the Headspace dataset, thus generating the first public shape-and-texture 3D Morphable Model (3DMM) of the full human head. Our approach is the first to employ a template that adapts to the dataset subject before dense morphing. This is fully automatic and achieved using 2D facial landmarking, projection to 3D shape, and mesh editing. In dense template morphing, we improve on the well-known Coherent Point Drift algorithm, by incorporating iterative data-sampling and alignment. Our evaluations demonstrate that our method has better performance in correspondence accuracy and modeling ability when compared with other competing algorithms. We propose a texture map refinement scheme to build high quality texture maps and texture model. We present several applications that include the first clinical use of craniofacial 3DMMs in the assessment of different types of surgical intervention applied to a craniosynostosis patient group

Groupwise non-rigid registration for automatic construction of appearance models of the human craniofacial complex for analysis, synthesis and simulation

Finally, a novel application of 3D appearance modelling is proposed: a faster than real-time algorithm for statistically constrained quasi-mechanical simulation. Experiments demonstrate superior realism, achieved in the proposed method by employing statistical appearance models to drive the simulation, in comparison with the comparable state-of-the-art quasi-mechanical approaches.EThOS - Electronic Theses Online ServiceGBUnited Kingdo

Automatic analysis of facial actions: a survey

As one of the most comprehensive and objective ways to describe facial expressions, the Facial Action Coding System (FACS) has recently received significant attention. Over the past 30 years, extensive research has been conducted by psychologists and neuroscientists on various aspects of facial expression analysis using FACS. Automating FACS coding would make this research faster and more widely applicable, opening up new avenues to understanding how we communicate through facial expressions. Such an automated process can also potentially increase the reliability, precision and temporal resolution of coding. This paper provides a comprehensive survey of research into machine analysis of facial actions. We systematically review all components of such systems: pre-processing, feature extraction and machine coding of facial actions. In addition, the existing FACS-coded facial expression databases are summarised. Finally, challenges that have to be addressed to make automatic facial action analysis applicable in real-life situations are extensively discussed. There are two underlying motivations for us to write this survey paper: the first is to provide an up-to-date review of the existing literature, and the second is to offer some insights into the future of machine recognition of facial actions: what are the challenges and opportunities that researchers in the field face

Towards 3D facial morphometry:facial image analysis applications in anesthesiology and 3D spectral nonrigid registration

In anesthesiology, the detection and anticipation of difficult tracheal intubation is crucial for patient safety. When undergoing general anesthesia, a patient who is unexpectedly difficult to intubate risks potential life-threatening complications with poor clinical outcomes, ranging from severe harm to brain damage or death. Conversely, in cases of suspected difficulty, specific equipment and personnel will be called upon to increase safety and the chances of successful intubation. Research in anesthesiology has associated a certain number of morphological features of the face and neck with higher risk of difficult intubation. Detecting and analyzing these and other potential features, thus allowing the prediction of difficulty of tracheal intubation in a robust, objective, and automatic way, may therefore improve the patients' safety. In this thesis, we first present a method to automatically classify images of the mouth cavity according to the visibility of certain oropharyngeal structures. This method is then integrated into a novel and completely automatic method, based on frontal and profile images of the patient's face, to predict the difficulty of intubation. We also provide a new database of three dimensional (3D) facial scans and present the initial steps towards a complete 3D model of the face suitable for facial morphometry applications, which include difficult tracheal intubation prediction. In order to develop and test our proposed method, we collected a large database of multimodal recordings of over 2700 patients undergoing general anesthesia. In the first part of this thesis, using two dimensional (2D) facial image analysis methods, we automatically extract morphological and appearance-based features from these images. These are used to train a classifier, which learns to discriminate between patients as being easy or difficult to intubate. We validate our approach on two different scenarios, one of them being close to a real-world clinical scenario, using 966 patients, and demonstrate that the proposed method achieves performance comparable to medical diagnosis-based predictions by experienced anesthesiologists. In the second part of this thesis, we focus on the development of a new 3D statistical model of the face to overcome some of the limitations of 2D methods. We first present EPFL3DFace, a new database of 3D facial expression scans, containing 120 subjects, performing 35 different facial expressions. Then, we develop a nonrigid alignment method to register the scans and allow for statistical analysis. Our proposed method is based on spectral geometry processing and makes use of an implicit representation of the scans in order to be robust to noise or holes in the surfaces. It presents the significant advantage of reducing the number of free parameters to optimize for in the alignment process by two orders of magnitude. We apply our proposed method on the data collected and discuss qualitative results. At its current level of performance, our fully automatic method to predict difficult intubation already has the potential to reduce the cost, and increase the availability of such predictions, by not relying on qualified anesthesiologists with years of medical training. Further data collection, in order to increase the number of patients who are difficult to intubate, as well as extracting morphological features from a 3D representation of the face are key elements to further improve the performance

Groupwise non-rigid registration for automatic construction of appearance models of the human craniofacial complex for analysis, synthesis and simulation

Finally, a novel application of 3D appearance modelling is proposed: a faster than real-time algorithm for statistically constrained quasi-mechanical simulation. Experiments demonstrate superior realism, achieved in the proposed method by employing statistical appearance models to drive the simulation, in comparison with the comparable state-of-the-art quasi-mechanical approaches

Assessing the existence of visual clues of human ovulation

Is the concealed human ovulation a myth? The author of this work tries to answer the above question by using a medium-size database of facial images specially created and tagged. Analyzing possible facial modifications during the mensal period is a formal tool to assess the veracity about the concealed ovulation. In normal view, the human ovulation remains concealed. In other words, there is no visible external sign of the mensal period in humans. These external signs are very much visible in many animals such as baboons, dogs or elephants. Some are visual (baboons) and others are biochemical (dogs). Insects use pheromones and other animals can use sounds to inform the partners of their fertility period. The objective is not just to study the visual female ovulation signs but also to understand and explain automatic image processing methods which could be used to extract precise landmarks from the facial pictures. This could later be applied to the studies about the fluctuant asymmetry. The field of fluctuant asymmetry is a growing field in evolutionary biology but cannot be easily developed because of the necessary time to manually extract the landmarks. In this work we have tried to see if any perceptible sign is present in human face during the ovulation and how we can detect formal changes, if any, in face appearance during the mensal period. We have taken photography from 50 girls for 32 days. Each day we took many photos of each girl. At the end we chose a set of 30 photos per girl representing the whole mensal cycle. From these photos 600 were chosen to be manually tagged for verification issues. The photos were organized in a rating software to allow human raters to watch and choose the two best looking pictures for each girl. These results were then checked to highlight the relation between chosen photos and ovulation period in the cycle. Results were indicating that in fact there are some clues in the face of human which could eventually give a hint about their ovulation. Later, different automatic landmark detection methods were applied to the pictures to highlight possible modifications in the face during the period. Although the precision of the tested methods, are far from being perfect, the comparison of these measurements to the state of art indexes of beauty shows a slight modification of the face towards a prettier face during the ovulation. The automatic methods tested were Active Appearance Model (AAM), the neural deep learning and the regression trees. It was observed that for this kind of applications the best method was the regression trees. Future work has to be conducted to firmly confirm these data, number of human raters should be augmented, and a proper learning data base should be developed to allow a learning process specific to this problematic. We also think that low level image processing will be necessary to achieve the final precision which could reveal more details of possible changes in human faces.A ovulação no ser humano é, em geral, considerada “oculta”, ou seja, sem sinais exteriores. Mas a ovulação ou o período mensal é uma mudança hormonal extremamente importante que se repete em cada ciclo. Acreditar que esta mudança hormonal não tem nenhum sinal visível parece simplista. Estes sinais externos são muito visíveis em animais, como babuínos, cães ou elefantes. Alguns são visuais (babuínos) e outros são bioquímicos (cães). Insetos usam feromonas e outros animais podem usar sons para informar os parceiros do seu período de fertilidade. O ser humano tem vindo a esconder ou pelo menos camuflar sinais desses durante a evolução. As razoes para esconder ou camuflar a ovulação no ser humano não são claros e não serão discutidos nesta dissertação. Na primeira parte deste trabalho, a autora deste trabalho, depois de criar um base de dados de tamanho médio de imagens faciais e anotar as fotografias vai verificar se sinais de ovulação podem ser detetados por outros pessoas. Ou seja, se modificações que ‘as priori’ são invisíveis podem ser percebidas de maneira inconsciente pelo observador. Na segunda parte, a autora vai analisar as eventuais modificações faciais durante o período, de uma maneira formal, utilizando medidas faciais. Métodos automáticos de analise de imagem aplicados permitem obter os dados necessários. Uma base de dados de imagens para efetuar este trabalho foi criado de raiz, uma vez que nenhuma base de dados existia na literatura. 50 raparigas aceitaram de participar na criação do base de dados. Durante 32 dias e diariamente, cada rapariga foi fotografada. Em cada sessão foi tirada várias fotos. As fotos foram depois apuradas para deixar só 30 fotos ao máximo, para cada rapariga. 600 fotos foram depois escolhidas para serem manualmente anotadas. Essas 600 fotos anotadas, definam a base de dados de verificação. Assim as medidas obtidas automaticamente podem ser verificadas comparando com a base de 600 fotos anotadas. O objetivo deste trabalho não é apenas estudar os sinais visuais da ovulação feminina, mas também testar e explicar métodos de processamento automático de imagens que poderiam ser usados para extrair pontos de interesse, das imagens faciais. A automatização de extração dos pontos de interesse poderia mais tarde ser aplicado aos estudos sobre a assimetria flutuante. O campo da assimetria flutuante é um campo crescente na biologia evolucionária, mas não pode ser desenvolvido facilmente. O tempo necessário para extrair referencias e pontos de interesse é proibitivo. Por além disso, estudos de assimetria flutuante, muitas vezes, baseado numa só fotografia pode vier a ser invalido, se modificações faciais temporárias existirem. Modificações temporárias, tipo durante o período mensal, revela que estudos fenotípicos baseados numa só fotografia não pode constituir uma base viável para estabelecer ligas genótipo-fenótipo. Para tentar ver se algum sinal percetível está presente no rosto humano durante a ovulação, as fotos foram organizadas num software de presentação para permitir o observador humano escolher duas fotos (as mais atraentes) de cada rapariga. Estes resultados foram então analisados para destacar a relação entre as fotos escolhidas e o período de ovulação no ciclo mensal. Os resultados sugeriam que, de facto, existem algumas indicações no rosto que poderiam eventualmente dar informações sobre o período de ovulação. Os observadores escolheram como mais atraente de cada rapariga, aquelas que tinham sido tiradas nos dias imediatos antes ou depois da ovulação. Ou seja, foi claramente estabelecido que a mesma rapariga parecia mais atraente durante os dias próximos da data da ovulação. O software também permite recolher dados sobre o observador para analise posterior de comportamento dos observadores perante as fotografias. Os dados dos observadores podem dar indicações sobre as razoes da ovulação escondida que foi desenvolvida durante a evolução. A seguir, diferentes métodos automáticos de deteção de pontos de interesse foram aplicados às imagens para detetar o tipo de modificações no rosto durante o período. A precisão dos métodos testados, apesar de não ser perfeita, permite observar algumas relações entre as modificações e os índices de atratividade. Os métodos automáticos testados foram Active Appearance Model (AAM), Convolutional Neural Networks (CNN) e árvores de regressão (Dlib-Rt). AAM e CNN foram implementados em Python utilizando o modulo Keras library. Dlib-Rt foi implementado em C++ utilizando OpenCv. Os métodos utilizados, estão todos baseados em aprendizagem e sacrificam a precisão. Comparando os resultados dos métodos automáticos com os resultados manualmente obtidos, indicaram que os métodos baseados em aprendizagem podem não ter a precisão necessária para estudos em simetria flutuante ou para estudos de modificação faciais finas. Apesar de falta de precisão, observou-se que, para este tipo de aplicação, o melhor método (entre os testados) foi as árvores de regressão. Os dados e medidas obtidas, constituíram uma base de dados com a data de período, medidas faciais, dados sociais e dados de atratividade que poderem ser utilizados para trabalhos posteriores. O trabalho futuro tem de ser conduzido para confirmar firmemente estes dados, o número de avaliadores humanos deve ser aumentado, e uma base de dados de aprendizagem adequada deve ser desenvolvida para permitir a definição de um processo de aprendizagem específico para esta problemática. Também foi observado que o processamento de imagens de baixo nível será necessário para alcançar a precisão final que poderia revelar detalhes finos de mudanças em rostos humanos. Transcrever os dados e medidas para o índice de atratividade e aplicar métodos de data-mining pode revelar exatamente quais são as modificações implicadas durante o período mensal. A autora também prevê a utilização de uma câmara fotográfica tipo true-depth permite obter os dados de profundidade e volumo que podem afinar os estudos. Os dados de pigmentação da pele e textura da mesma também devem ser considerados para obter e observar todos tipos de modificação facial durante o período mensal. Os dados também devem separar raparigas com métodos químicos de contraceção, uma vez que estes métodos podem interferir com os níveis hormonais e introduzir erros de apreciação. Por fim o mesmo estudo poderia ser efetuado nos homens, uma vez que homens não sofrem de mudanças hormonais, a aparição de qualquer modificação facial repetível pode indicar existência de fatos camuflados

Face modeling for face recognition in the wild.

Face understanding is considered one of the most important topics in computer vision field since the face is a rich source of information in social interaction. Not only does the face provide information about the identity of people, but also of their membership in broad demographic categories (including sex, race, and age), and about their current emotional state. Facial landmarks extraction is the corner stone in the success of different facial analyses and understanding applications. In this dissertation, a novel facial modeling is designed for facial landmarks detection in unconstrained real life environment from different image modalities including infra-red and visible images. In the proposed facial landmarks detector, a part based model is incorporated with holistic face information. In the part based model, the face is modeled by the appearance of different face part(e.g., right eye, left eye, left eyebrow, nose, mouth) and their geometric relation. The appearance is described by a novel feature referred to as pixel difference feature. This representation is three times faster than the state-of-art in feature representation. On the other hand, to model the geometric relation between the face parts, the complex Bingham distribution is adapted from the statistical community into computer vision for modeling the geometric relationship between the facial elements. The global information is incorporated with the local part model using a regression model. The model results outperform the state-of-art in detecting facial landmarks. The proposed facial landmark detector is tested in two computer vision problems: boosting the performance of face detectors by rejecting pseudo faces and camera steering in multi-camera network. To highlight the applicability of the proposed model for different image modalities, it has been studied in two face understanding applications which are face recognition from visible images and physiological measurements for autistic individuals from thermal images. Recognizing identities from faces under different poses, expressions and lighting conditions from a complex background is an still unsolved problem even with accurate detection of landmark. Therefore, a learning similarity measure is proposed. The proposed measure responds only to the difference in identities and filter illuminations and pose variations. similarity measure makes use of statistical inference in the image plane. Additionally, the pose challenge is tackled by two new approaches: assigning different weights for different face part based on their visibility in image plane at different pose angles and synthesizing virtual facial images for each subject at different poses from single frontal image. The proposed framework is demonstrated to be competitive with top performing state-of-art methods which is evaluated on standard benchmarks in face recognition in the wild. The other framework for the face understanding application, which is a physiological measures for autistic individual from infra-red images. In this framework, accurate detecting and tracking Superficial Temporal Arteria (STA) while the subject is moving, playing, and interacting in social communication is a must. It is very challenging to track and detect STA since the appearance of the STA region changes over time and it is not discriminative enough from other areas in face region. A novel concept in detection, called supporter collaboration, is introduced. In support collaboration, the STA is detected and tracked with the help of face landmarks and geometric constraint. This research advanced the field of the emotion recognition

Automatic age progression and estimation from faces

Recently, automatic age progression has gained popularity due to its numerous applications. Among these is the frequent search for missing people, in the UK alone up to 300,000 people are reported missing every year. Although many algorithms have been proposed, most of the methods are affected by image noise, illumination variations, and facial expressions. Furthermore, most of the algorithms use a pattern caricaturing approach which infers ages by manipulating the target image and a template face formed by averaging faces at the intended age. To this end, this thesis investigates the problem with a view to tackling the most prominent issues associated with the existing algorithms. Initially using active appearance models (AAM), facial features are extracted and mapped to people’s ages, afterward a formula is derived which allows the convenient generation of age progressed images irrespective of whether the intended age exists in the training database or not. In order to handle image noise as well as varying facial expressions, a nonlinear appearance model called kernel appearance model (KAM) is derived. To illustrate the real application of automatic age progression, both AAM and KAM based algorithms are then used to synthesise faces of two popular long missing British and Irish kids; Ben Needham and Mary Boyle. However, both statistical techniques exhibit image rendering artefacts such as low-resolution output and the generation of inconsistent skin tone. To circumvent this problem, a hybrid texture enhancement pipeline is developed. To further ensure that the progressed images preserve people’s identities while at the same time attaining the intended age, rigorous human and machine based tests are conducted; part of this tests resulted to the development of a robust age estimation algorithm. Eventually, the results of the rigorous assessment reveal that the hybrid technique is able to handle all existing problems of age progression with minimal error.National Information Technology Development Agency of Nigeria (NITDA