99 research outputs found
Anatomically Constrained Implicit Face Models
Coordinate based implicit neural representations have gained rapid popularity
in recent years as they have been successfully used in image, geometry and
scene modeling tasks. In this work, we present a novel use case for such
implicit representations in the context of learning anatomically constrained
face models. Actor specific anatomically constrained face models are the state
of the art in both facial performance capture and performance retargeting.
Despite their practical success, these anatomical models are slow to evaluate
and often require extensive data capture to be built. We propose the anatomical
implicit face model; an ensemble of implicit neural networks that jointly learn
to model the facial anatomy and the skin surface with high-fidelity, and can
readily be used as a drop in replacement to conventional blendshape models.
Given an arbitrary set of skin surface meshes of an actor and only a neutral
shape with estimated skull and jaw bones, our method can recover a dense
anatomical substructure which constrains every point on the facial surface. We
demonstrate the usefulness of our approach in several tasks ranging from shape
fitting, shape editing, and performance retargeting
Functionality-Driven Musculature Retargeting
We present a novel retargeting algorithm that transfers the musculature of a
reference anatomical model to new bodies with different sizes, body
proportions, muscle capability, and joint range of motion while preserving the
functionality of the original musculature as closely as possible. The geometric
configuration and physiological parameters of musculotendon units are estimated
and optimized to adapt to new bodies. The range of motion around joints is
estimated from a motion capture dataset and edited further for individual
models. The retargeted model is simulation-ready, so we can physically simulate
muscle-actuated motor skills with the model. Our system is capable of
generating a wide variety of anatomical bodies that can be simulated to walk,
run, jump and dance while maintaining balance under gravity. We will also
demonstrate the construction of individualized musculoskeletal models from
bi-planar X-ray images and medical examinations.Comment: 15 pages, 20 figure
Neural Volumetric Blendshapes: Computationally Efficient Physics-Based Facial Blendshapes
Computationally weak systems and demanding graphical applications are still
mostly dependent on linear blendshapes for facial animations. The accompanying
artifacts such as self-intersections, loss of volume, or missing soft tissue
elasticity can be avoided by using physics-based animation models. However,
these are cumbersome to implement and require immense computational effort. We
propose neural volumetric blendshapes, an approach that combines the advantages
of physics-based simulations with realtime runtimes even on consumer-grade
CPUs. To this end, we present a neural network that efficiently approximates
the involved volumetric simulations and generalizes across human identities as
well as facial expressions. Our approach can be used on top of any linear
blendshape system and, hence, can be deployed straightforwardly. Furthermore,
it only requires a single neutral face mesh as input in the minimal setting.
Along with the design of the network, we introduce a pipeline for the
challenging creation of anatomically and physically plausible training data.
Part of the pipeline is a novel hybrid regressor that densely positions a skull
within a skin surface while avoiding intersections. The fidelity of all parts
of the data generation pipeline as well as the accuracy and efficiency of the
network are evaluated in this work. Upon publication, the trained models and
associated code will be released
What a Feeling: Learning Facial Expressions and Emotions.
People with Autism Spectrum Disorders (ASD) find it difficult to understand facial expressions. We present a new approach that targets one of the core symptomatic deficits in ASD: the ability to recognize the feeling states of others. What a Feeling is a videogame that aims to improve the ability of socially and emotionally impaired individuals to recognize and respond to emotions conveyed by the face in a playful way. It enables people from all ages to interact with 3D avatars and learn facial expressions through a set of exercises. The game engine is based on real-time facial synthesis. This paper describes the core mechanics of our learning methodology and discusses future evaluation directions
A framework for automatic and perceptually valid facial expression generation
Facial expressions are facial movements reflecting the internal emotional states of a character or in response to social communications. Realistic facial animation should consider at least two factors: believable visual effect and valid facial movements. However, most research tends to separate these two issues. In this paper, we present a framework for generating 3D facial expressions considering both the visual the dynamics effect. A facial expression mapping approach based on local geometry encoding is proposed, which encodes deformation in the 1-ring vector. This method is capable of mapping subtle facial movements without considering those shape and topological constraints. Facial expression mapping is achieved through three steps: correspondence establishment, deviation transfer and movement mapping. Deviation is transferred to the conformal face space through minimizing the error function. This function is formed by the source neutral and the deformed face model related by those transformation matrices in 1-ring neighborhood. The transformation matrix in 1-ring neighborhood is independent of the face shape and the mesh topology. After the facial expression mapping, dynamic parameters are then integrated with facial expressions for generating valid facial expressions. The dynamic parameters were generated based on psychophysical methods. The efficiency and effectiveness of the proposed methods have been tested using various face models with different shapes and topological representations
Accurate and Interpretable Solution of the Inverse Rig for Realistic Blendshape Models with Quadratic Corrective Terms
We propose a new model-based algorithm solving the inverse rig problem in
facial animation retargeting, exhibiting higher accuracy of the fit and
sparser, more interpretable weight vector compared to SOTA. The proposed method
targets a specific subdomain of human face animation - highly-realistic
blendshape models used in the production of movies and video games. In this
paper, we formulate an optimization problem that takes into account all the
requirements of targeted models. Our objective goes beyond a linear blendshape
model and employs the quadratic corrective terms necessary for correctly
fitting fine details of the mesh. We show that the solution to the proposed
problem yields highly accurate mesh reconstruction even when general-purpose
solvers, like SQP, are used. The results obtained using SQP are highly accurate
in the mesh space but do not exhibit favorable qualities in terms of weight
sparsity and smoothness, and for this reason, we further propose a novel
algorithm relying on a MM technique. The algorithm is specifically suited for
solving the proposed objective, yielding a high-accuracy mesh fit while
respecting the constraints and producing a sparse and smooth set of weights
easy to manipulate and interpret by artists. Our algorithm is benchmarked with
SOTA approaches, and shows an overall superiority of the results, yielding a
smooth animation reconstruction with a relative improvement up to 45 percent in
root mean squared mesh error while keeping the cardinality comparable with
benchmark methods. This paper gives a comprehensive set of evaluation metrics
that cover different aspects of the solution, including mesh accuracy, sparsity
of the weights, and smoothness of the animation curves, as well as the
appearance of the produced animation, which human experts evaluated
- …