Automatic tailoring and cloth modelling for animation characters.

The construction of realistic characters has become increasingly important to the production of blockbuster films, TV series and computer games. The outfit of character plays an important role in the application of virtual characters. It is one of the key elements reflects the personality of character. Virtual clothing refers to the process that constructs outfits for virtual characters, and currently, it is widely used in mainly two areas, fashion industry and computer animation. In fashion industry, virtual clothing technology is an effective tool which creates, edits and pre-visualises cloth design patterns efficiently. However, using this method requires lots of tailoring expertises. In computer animation, geometric modelling methods are widely used for cloth modelling due to their simplicity and intuitiveness. However, because of the shortage of tailoring knowledge among animation artists, current existing cloth design patterns can not be used directly by animation artists, and the appearance of cloth depends heavily on the skill of artists. Moreover, geometric modelling methods requires lots of manual operations. This tediousness is worsen by modelling same style cloth for different characters with different body shapes and proportions. This thesis addresses this problem and presents a new virtual clothing method which includes automatic character measuring, automatic cloth pattern adjustment, and cloth patterns assembling. There are two main contributions in this research. Firstly, a geodesic curvature flow based geodesic computation scheme is presented for acquiring length measurements from character. Due to the fast growing demand on usage of high resolution character model in animation production, the increasing number of characters need to be handled simultaneously as well as improving the reusability of 3D model in film production, the efficiency of modelling cloth for multiple high resolution character is very important. In order to improve the efficiency of measuring character for cloth fitting, a fast geodesic algorithm that has linear time complexity with a small bounded error is also presented. Secondly, a cloth pattern adjusting genetic algorithm is developed for automatic cloth fitting and retargeting. For the reason that that body shapes and proportions vary largely in character design, fitting and transferring cloth to a different character is a challenging task. This thesis considers the cloth fitting process as an optimization procedure. It optimizes both the shape and size of each cloth pattern automatically, the integrity, design and size of each cloth pattern are evaluated in order to create 3D cloth for any character with different body shapes and proportions while preserve the original cloth design. By automating the cloth modelling process, it empowers the creativity of animation artists and improves their productivity by allowing them to use a large amount of existing cloth design patterns in fashion industry to create various clothes and to transfer same design cloth to characters with different body shapes and proportions with ease

A physically-based muscle and skin model for facial animation

Facial animation is a popular area of research which has been around for over thirty years, but even with this long time scale, automatically creating realistic facial expressions is still an unsolved goal. This work furthers the state of the art in computer facial animation by introducing a new muscle and skin model and a method of easily transferring a full muscle and bone animation setup from one head mesh to another with very little user input. The developed muscle model allows muscles of any shape to be accurately simulated, preserving volume during contraction and interacting with surrounding muscles and skin in a lifelike manner. The muscles can drive a rigid body model of a jaw, giving realistic physically-based movement to all areas of the face. The skin model has multiple layers, mimicking the natural structure of skin and it connects onto the muscle model and is deformed realistically by the movements of the muscles and underlying bones. The skin smoothly transfers underlying movements into skin surface movements and propagates forces smoothly across the face. Once a head model has been set up with muscles and bones, moving this muscle and bone set to another head is a simple matter using the developed techniques. The developed software employs principles from forensic reconstruction, using specific landmarks on the head to map the bone and muscles to the new head model and once the muscles and skull have been quickly transferred, they provide animation capabilities on the new mesh within minutes

Developing a Computer System for the Generation of Unique Wrinkle Maps for Human Faces. Generating 2D Wrinkle Maps using Various Image Processing Techniques and the Design of 3D Facial Ageing System using 3D Modelling Tools.

Facial Ageing (FA) is a very fundamental issue, as ageing in general, is part of our daily life process. FA is used in security, finding missing children and other applications. It is also a form of Facial Recognition (FR) that helps identifying suspects. FA affects several parts of the human face under the influence of different biological and environmental factors. One of the major facial feature changes that occur as a result of ageing is the appearance and development of wrinkles. Facial wrinkles are skin folds; their shapes and numbers differ from one person to another, therefore, an advantage can be taken over these characteristics if a system is implemented to extract the facial wrinkles in a form of maps. This thesis is presenting a new technique for three-dimensional facial wrinkle pattern information that can also be utilised for biometric applications, which will back up the system for further increase of security. The procedural approaches adopted for investigating this new technique are the extraction of two-dimensional wrinkle maps of frontal human faces for digital images and the design of three-dimensional wrinkle pattern formation system that utilises the generated wrinkle maps. The first approach is carried out using image processing tools so that for any given individual, two wrinkle maps are produced; the first map is in a binary form that shows the positions of the wrinkles on the face while the other map is a coloured version that indicates the different intensities of the wrinkles. The second approach of the 3D system development involves the alignment of the binary wrinkle maps on the corresponding 3D face models, followed by the projection of 3D curves in order to acquire 3D representations of the wrinkles. With the aid of the coloured wrinkle maps as well as some ageing parameters, simulations and predictions for the 3D wrinkles are performed

High-quality face capture, animation and editing from monocular video

Digitization of virtual faces in movies requires complex capture setups and extensive manual work to produce superb animations and video-realistic editing. This thesis pushes the boundaries of the digitization pipeline by proposing automatic algorithms for high-quality 3D face capture and animation, as well as photo-realistic face editing. These algorithms reconstruct and modify faces in 2D videos recorded in uncontrolled scenarios and illumination. In particular, advances in three main areas offer solutions for the lack of depth and overall uncertainty in video recordings. First, contributions in capture include model-based reconstruction of detailed, dynamic 3D geometry that exploits optical and shading cues, multilayer parametric reconstruction of accurate 3D models in unconstrained setups based on inverse rendering, and regression-based 3D lip shape enhancement from high-quality data. Second, advances in animation are video-based face reenactment based on robust appearance metrics and temporal clustering, performance-driven retargeting of detailed facial models in sync with audio, and the automatic creation of personalized controllable 3D rigs. Finally, advances in plausible photo-realistic editing are dense face albedo capture and mouth interior synthesis using image warping and 3D teeth proxies. High-quality results attained on challenging application scenarios confirm the contributions and show great potential for the automatic creation of photo-realistic 3D faces.Die Digitalisierung von Gesichtern zum Einsatz in der Filmindustrie erfordert komplizierte Aufnahmevorrichtungen und die manuelle Nachbearbeitung von Rekonstruktionen, um perfekte Animationen und realistische Videobearbeitung zu erzielen. Diese Dissertation erweitert vorhandene Digitalisierungsverfahren durch die Erforschung von automatischen Verfahren zur qualitativ hochwertigen 3D Rekonstruktion, Animation und Modifikation von Gesichtern. Diese Algorithmen erlauben es, Gesichter in 2D Videos, die unter allgemeinen Bedingungen und unbekannten Beleuchtungsverhältnissen aufgenommen wurden, zu rekonstruieren und zu modifizieren. Vor allem Fortschritte in den folgenden drei Hauptbereichen tragen zur Kompensation von fehlender Tiefeninformation und der allgemeinen Mehrdeutigkeit von 2D Videoaufnahmen bei. Erstens, Beiträge zur modellbasierten Rekonstruktion von detaillierter und dynamischer 3D Geometrie durch optische Merkmale und die Shading-Eigenschaften des Gesichts, mehrschichtige parametrische Rekonstruktion von exakten 3D Modellen mittels inversen Renderings in allgemeinen Szenen und regressionsbasierter 3D Lippenformverfeinerung mittels qualitativ hochwertigen Daten. Zweitens, Fortschritte im Bereich der Computeranimation durch videobasierte Gesichtsausdrucksübertragung und temporaler Clusterbildung, Übertragung von detaillierten Gesichtsmodellen, deren Mundbewegung mit Ton synchronisiert ist, und die automatische Erstellung von personalisierten "3D Face Rigs". Schließlich werden Fortschritte im Bereich der realistischen Videobearbeitung vorgestellt, welche auf der dichten Rekonstruktion von Hautreflektionseigenschaften und der Mundinnenraumsynthese mittels bildbasierten und geometriebasierten Verfahren aufbauen. Qualitativ hochwertige Ergebnisse in anspruchsvollen Anwendungen untermauern die Wichtigkeit der geleisteten Beiträgen und zeigen das große Potential der automatischen Erstellung von realistischen digitalen 3D Gesichtern auf

Beyond PCA: Deep Learning Approaches for Face Modeling and Aging

Modeling faces with large variations has been a challenging task in computer vision. These variations such as expressions, poses and occlusions are usually complex and non-linear. Moreover, new facial images also come with their own characteristic artifacts greatly diverse. Therefore, a good face modeling approach needs to be carefully designed for flexibly adapting to these challenging issues. Recently, Deep Learning approach has gained significant attention as one of the emerging research topics in both higher-level representation of data and the distribution of observations. Thanks to the nonlinear structure of deep learning models and the strength of latent variables organized in hidden layers, it can efficiently capture variations and structures in complex data. Inspired by this motivation, we present two novel approaches, i.e. Deep Appearance Models (DAM) and Robust Deep Appearance Models (RDAM), to accurately capture both shape and texture of face images under large variations. In DAM, three crucial components represented in hierarchical layers are modeled using Deep Boltzmann Machines (DBM) to robustly capture the variations of facial shapes and appearances. DAM has shown its potential in inferencing a representation for new face images under various challenging conditions. An improved version of DAM, named Robust DAM (RDAM), is also introduced to better handle the occluded face areas and, therefore, produces more plausible reconstruction results. These proposed approaches are evaluated in various applications to demonstrate their robustness and capabilities, e.g. facial super-resolution reconstruction, facial off-angle reconstruction, facial occlusion removal and age estimation using challenging face databases: Labeled Face Parts in the Wild (LFPW), Helen and FG-NET. Comparing to classical and other deep learning based approaches, the proposed DAM and RDAM achieve competitive results in those applications, thus this showed their advantages in handling occlusions, facial representation, and reconstruction. In addition to DAM and RDAM that are mainly used for modeling single facial image, the second part of the thesis focuses on novel deep models, i.e. Temporal Restricted Boltzmann Machines (TRBM) and tractable Temporal Non-volume Preserving (TNVP) approaches, to further model face sequences. By exploiting the additional temporal relationships presented in sequence data, the proposed models have their advantages in predicting the future of a sequence from its past. In the application of face age progression, age regression, and age-invariant face recognition, these models have shown their potential not only in efficiently capturing the non-linear age related variance but also producing a smooth synthesis in age progression across faces. Moreover, the structure of TNVP can be transformed into a deep convolutional network while keeping the advantages of probabilistic models with tractable log-likelihood density estimation. The proposed approach is evaluated in terms of synthesizing age-progressed faces and cross-age face verification. It consistently shows the state-of-the-art results in various face aging databases, i.e. FG-NET, MORPH, our collected large-scale aging database named AginG Faces in the Wild (AGFW), and Cross-Age Celebrity Dataset (CACD). A large-scale face verification on Megaface challenge 1 is also performed to further show the advantages of our proposed approach

Highly automated method for facial expression synthesis

This thesis was submitted for the degree of Doctor of Philosophy and awarded by Brunel University.The synthesis of realistic facial expressions has been an unexplored area for computer graphics scientists. Over the last three decades, several different construction methods have been formulated in order to obtain natural graphic results. Despite these advancements, though, current techniques still require costly resources, heavy user intervention and specific training and outcomes are still not completely realistic. This thesis, therefore, aims to achieve an automated synthesis that will produce realistic facial expressions at a low cost. This thesis, proposes a highly automated approach for achieving a realistic facial expression synthesis, which allows for enhanced performance in speed (3 minutes processing time maximum) and quality with a minimum of user intervention. It will also demonstrate a highly technical and automated method of facial feature detection, by allowing users to obtain their desired facial expression synthesis with minimal physical input. Moreover, it will describe a novel approach to the normalization of the illumination settings values between source and target images, thereby allowing the algorithm to work accurately, even in different lighting conditions. Finally, we will present the results obtained from the proposed techniques, together with our conclusions, at the end of the paper

Modelling facial action units using partial differential equations.

This thesis discusses a novel method for modelling facial action units. It presents facial action units model based on boundary value problems for accurate representation of human facial expression in three-dimensions. In particular, a solution to a fourth order elliptic Partial Differential Equation (PDE) subject to suitable boundary conditions is utilized, where the chosen boundary curves are based on muscles movement defined by Facial Action Coding System (FACS). This study involved three stages: modelling faces, manipulating faces and application to simple facial animation. In the first stage, PDE method is used in modelling and generating a smooth 3D face. The PDE formulation using small sets of parameters contributes to the efficiency of human face representation. In the manipulation stage, a generic PDE face of neutral expression is manipulated to a face with expression using PDE descriptors that uniquely represents an action unit. A combination of the PDE descriptor results in a generic PDE face having an expression, which successfully modelled four basic expressions: happy, sad, fear and disgust. An example of application is given using simple animation technique called blendshapes. This technique uses generic PDE face in animating basic expressions.Ministry of Higher Education, Malaysia and Universiti Malaysia Terenggan

Physics-based modelling, simulation, placement and learning for musculo-skeletal animations.

In character production for Visual Effects, the realism of deformations and flesh dynamics is a vital ingredient of the final rendered moving images shown on screen. This work is a collection of projects completed at the hosting company MPC London focused on the main components needed for the animation of musculo-skeletal systems: primitives modeling, physically accurate simulation, interactive placement. Complementary projects are also presented, including the procedural modeling of wrinkles and a machine learning approach for deformable objects based on Deep Neural Networks. Primitives modeling aims at proposing an approach to generating muscle geometry complete with tendons and fibers from superficial patches sketched on the character skin mesh. The method utilizes the physics of inflatable surfaces and produces meshes ready to be tetrahedralized, that is without compenetrations. A framework for the simulation of muscles, fascia and fat tissues based on the Finite Elements Method (FEM) is presented, together with the theoretical foundations of fiber-based materials with activations and their fitting in the Implicit Euler integration. The FEM solver is then simplified in or- der to achieve interactive rates to show the potential of interactive muscle placement on the skeleton to facilitate the creation of intersection-free primitives using collision detection and resolution. Alongside physics simulation for biological tissues, the thesis explores an approach that extends the Implicit Skinning technique with wrinkles based on convolution surfaces by exploiting the gradients of the combination of bones fields. Finally, this work discusses a possible approach to the learning of physics-based deformable objects based on deep neural networks which makes use of geodesic disks convolutional layers

Physically Based Forehead Modelling and Animation including Wrinkles

There has been a vast amount of research on the production of realistic facial models and animations, which is one of the most challenging areas of computer graphics. Recently, there has been an increased interest in the use of physically based approaches for facial animation, whereby the effects of muscle contractions are propagated through facial soft-tissue models to automatically deform them in a more realistic and anatomically accurate manner. Presented in this thesis is a fully physically based approach for efficiently producing realistic-looking animations of facial movement, including animation of expressive wrinkles, focussing on the forehead. This is done by modelling more physics-based behaviour than current computer graphics approaches. The presented research has two major components. The first is a novel model creation process to automatically create animatable non-conforming hexahedral finite element (FE) simulation models of facial soft tissue from any surface mesh that contains hole-free volumes. The generated multi-layered voxel-based models are immediately ready for simulation, with skin layers and element material properties, muscle properties, and boundary conditions being automatically computed. The second major component is an advanced optimised GPU-based process to simulate and visualise these models over time using the total Lagrangian explicit dynamic (TLED) formulation of the FE method. An anatomical muscle contraction model computes active and transversely isotropic passive muscle stresses, while advanced boundary conditions enable the sliding effect between the superficial and deep soft-tissue layers to be simulated. Soft-tissue models and animations with varying complexity are presented, from a simple soft-tissue-block model with uniform layers of skin and muscle, to a complex forehead model. These demonstrate the flexibility of the animation approach to produce detailed animations of realistic gross- and fine-scale soft-tissue movement, including wrinkles, with different muscle structures and material parameters, for example, to animate different-aged skin. Owing to the detail and accuracy of the models and simulations, the animation approach could also be used for applications outside of computer graphics, such as surgical applications. Furthermore, the animation approach can be used to animate any multi-layered soft body (not just soft tissue)