FDLS: A Deep Learning Approach to Production Quality, Controllable, and Retargetable Facial Performances

Visual effects commonly requires both the creation of realistic synthetic humans as well as retargeting actors' performances to humanoid characters such as aliens and monsters. Achieving the expressive performances demanded in entertainment requires manipulating complex models with hundreds of parameters. Full creative control requires the freedom to make edits at any stage of the production, which prohibits the use of a fully automatic ``black box'' solution with uninterpretable parameters. On the other hand, producing realistic animation with these sophisticated models is difficult and laborious. This paper describes FDLS (Facial Deep Learning Solver), which is Weta Digital's solution to these challenges. FDLS adopts a coarse-to-fine and human-in-the-loop strategy, allowing a solved performance to be verified and edited at several stages in the solving process. To train FDLS, we first transform the raw motion-captured data into robust graph features. Secondly, based on the observation that the artists typically finalize the jaw pass animation before proceeding to finer detail, we solve for the jaw motion first and predict fine expressions with region-based networks conditioned on the jaw position. Finally, artists can optionally invoke a non-linear finetuning process on top of the FDLS solution to follow the motion-captured virtual markers as closely as possible. FDLS supports editing if needed to improve the results of the deep learning solution and it can handle small daily changes in the actor's face shape. FDLS permits reliable and production-quality performance solving with minimal training and little or no manual effort in many cases, while also allowing the solve to be guided and edited in unusual and difficult cases. The system has been under development for several years and has been used in major movies.Comment: DigiPro '22: The Digital Production Symposiu

The biomechanical role of the chondrocranium and sutures in a lizard cranium

The role of soft tissues in skull biomechanics remains poorly understood. Not least, the chondrocranium, the portion of the braincase which persists as cartilage with varying degrees of mineralization. It also remains commonplace to overlook the biomechanical role of sutures despite evidence that they alter strain distribution. Here, we examine the role of both the sutures and the chondrocranium in the South American tegu lizard Salvator merianae. We use multi-body dynamics analysis (MDA) to provide realistic loading conditions for anterior and posterior unilateral biting and a detailed finite element model to examine strain magnitude and distribution. We find that strains within the chondrocraniumare greatest during anterior biting and are primarily tensile; also that strain within the cranium is not greatly reduced by the presence of the chondrocraniumunless it is given the same material properties as bone. This result contradicts previous suggestions that the anterior portion (the nasal septum) acts as a supporting structure. Inclusion of sutures to the cranium model not only increases overall strain magnitudes but also leads to a more complex distribution of tension and compression rather than that of a beam under sagittal bending

Neural Volumetric Blendshapes: Computationally Efficient Physics-Based Facial Blendshapes

Computationally weak systems and demanding graphical applications are still mostly dependent on linear blendshapes for facial animations. The accompanying artifacts such as self-intersections, loss of volume, or missing soft tissue elasticity can be avoided by using physics-based animation models. However, these are cumbersome to implement and require immense computational effort. We propose neural volumetric blendshapes, an approach that combines the advantages of physics-based simulations with realtime runtimes even on consumer-grade CPUs. To this end, we present a neural network that efficiently approximates the involved volumetric simulations and generalizes across human identities as well as facial expressions. Our approach can be used on top of any linear blendshape system and, hence, can be deployed straightforwardly. Furthermore, it only requires a single neutral face mesh as input in the minimal setting. Along with the design of the network, we introduce a pipeline for the challenging creation of anatomically and physically plausible training data. Part of the pipeline is a novel hybrid regressor that densely positions a skull within a skin surface while avoiding intersections. The fidelity of all parts of the data generation pipeline as well as the accuracy and efficiency of the network are evaluated in this work. Upon publication, the trained models and associated code will be released

Physics-based Reconstruction and Animation of Humans

Creating digital representations of humans is of utmost importance for applications ranging from entertainment (video games, movies) to human-computer interaction and even psychiatrical treatments. What makes building credible digital doubles difficult is the fact that the human vision system is very sensitive to perceiving the complex expressivity and potential anomalies in body structures and motion. This thesis will present several projects that tackle these problems from two different perspectives: lightweight acquisition and physics-based simulation. It starts by describing a complete pipeline that allows users to reconstruct fully rigged 3D facial avatars using video data coming from a handheld device (e.g., smartphone). The avatars use a novel two-scale representation composed of blendshapes and dynamic detail maps. They are constructed through an optimization that integrates feature tracking, optical flow, and shape from shading. Continuing along the lines of accessible acquisition systems, we discuss a framework for simultaneous tracking and modeling of articulated human bodies from RGB-D data. We show how semantic information can be extracted from the scanned body shapes. In the second half of the thesis, we will deviate from using standard linear reconstruction and animation models, and rather focus on exploiting physics-based techniques that are able to incorporate complex phenomena such as dynamics, collision response and incompressibility of the materials. The first approach we propose assumes that each 3D scan of an actor records his body in a physical steady state and uses a process called inverse physics to extract a volumetric physics-ready anatomical model of him. By using biologically-inspired growth models for the bones, muscles and fat, our method can obtain realistic anatomical reconstructions that can be later on animated using external tracking data such as the one resulting from tracking motion capture markers. This is then extended to a novel physics-based approach for facial reconstruction and animation. We propose a facial animation model which simulates biomechanical muscle contractions in a volumetric head model in order to create the facial expressions seen in the input scans. We then show how this approach allows for new avenues of dynamic artistic control, simulation of corrective facial surgery, and interaction with external forces and objects

SIMBIO-M 2014, SIMulation technologies in the fields of BIO-Sciences and Multiphysics: BioMechanics, BioMaterials and BioMedicine, Marseille, France, june 2014

Proceedings de la 3ème édition de la conférence internationale Simbio-M (2014). Organisée conjointement par l'IFSTTAR, Aix-Marseille Université, l'université de Coventry et CADLM, cette conférence se concentre sur les progrès des technologies de simulation dans les domaines des sciences du vivant et multiphysiques: Biomécanique, Biomatériaux et Biomédical. L'objectif de cette conférence est de partager et d'explorer les résultats dans les techniques d'analyse numérique et les outils de modélisation mathématique. Cette approche numérique permet des études prévisionnelles ou exploratoires dans les différents domaines des biosciences

Ground Robotic Hand Applications for the Space Program study (GRASP)

This document reports on a NASA-STDP effort to address research interests of the NASA Kennedy Space Center (KSC) through a study entitled, Ground Robotic-Hand Applications for the Space Program (GRASP). The primary objective of the GRASP study was to identify beneficial applications of specialized end-effectors and robotic hand devices for automating any ground operations which are performed at the Kennedy Space Center. Thus, operations for expendable vehicles, the Space Shuttle and its components, and all payloads were included in the study. Typical benefits of automating operations, or augmenting human operators performing physical tasks, include: reduced costs; enhanced safety and reliability; and reduced processing turnaround time

Muscle moment arm analyses applied to vertebrate paleontology: a case study using Stegosaurus stenops Marsh, 1887

The moment arm of a muscle defines its leverage around a given joint. In a clinical setting, the quantification of muscle moment arms is an important means of establishing the ‘healthy’ functioning of a muscle and in identifying and treating musculoskeletal abnormalities. Elsewhere in modern animal taxa, moment arm studies aim to illuminate adaptions of the musculoskeletal system towards particular locomotor or feeding behaviors. In the absence of kinematic data, paleontologists have likewise relied upon estimated muscle moment arms as a means of reconstructing musculoskeletal function and biomechanical performance in fossil species. With the application of ‘virtual paleontological’ techniques, it is possible to generate increasingly detailed musculoskeletal models of extinct taxa. However, the steps taken to derive such models of complex systems are seldom reported in detail. Here we present a case study for calculating three-dimensional muscle moment arms using Stegosaurus stenops Marsh, 1887 to highlight both the potential and the limitations of this approach in vertebrate paleontology. We find the technique to be mostly insensitive to choices in muscle modeling parameters (particularly relative to other sources of uncertainty in paleontological studies), although exceptions do exist. Of more concern is the current lack of consensus on what functional signals, if any, are contained within moment arm data derived from extant species. Until a correlation between muscle moment arm and function can be broadly identified across a range of modern taxa, the interpretation of moment arms calculated for extinct taxa should be approached with caution

Muscle moment arm analyses applied to vertebrate paleontology: a case study using Stegosaurus stenops Marsh, 1887

The moment arm of a muscle defines its leverage around a given joint. In a clinical setting, the quantification of muscle moment arms is an important means of establishing the ‘healthy’ functioning of a muscle and in identifying and treating musculoskeletal abnormalities. Elsewhere in modern animal taxa, moment arm studies aim to illuminate adaptions of the musculoskeletal system towards particular locomotor or feeding behaviors. In the absence of kinematic data, paleontologists have likewise relied upon estimated muscle moment arms as a means of reconstructing musculoskeletal function and biomechanical performance in fossil species. With the application of ‘virtual paleontological’ techniques, it is possible to generate increasingly detailed musculoskeletal models of extinct taxa. However, the steps taken to derive such models of complex systems are seldom reported in detail. Here we present a case study for calculating three-dimensional muscle moment arms using Stegosaurus stenops Marsh, 1887 to highlight both the potential and the limitations of this approach in vertebrate paleontology. We find the technique to be mostly insensitive to choices in muscle modeling parameters (particularly relative to other sources of uncertainty in paleontological studies), although exceptions do exist. Of more concern is the current lack of consensus on what functional signals, if any, are contained within moment arm data derived from extant species. Until a correlation between muscle moment arm and function can be broadly identified across a range of modern taxa, the interpretation of moment arms calculated for extinct taxa should be approached with caution