472 research outputs found
Neural Face Editing with Intrinsic Image Disentangling
Traditional face editing methods often require a number of sophisticated and
task specific algorithms to be applied one after the other --- a process that
is tedious, fragile, and computationally intensive. In this paper, we propose
an end-to-end generative adversarial network that infers a face-specific
disentangled representation of intrinsic face properties, including shape (i.e.
normals), albedo, and lighting, and an alpha matte. We show that this network
can be trained on "in-the-wild" images by incorporating an in-network
physically-based image formation module and appropriate loss functions. Our
disentangling latent representation allows for semantically relevant edits,
where one aspect of facial appearance can be manipulated while keeping
orthogonal properties fixed, and we demonstrate its use for a number of facial
editing applications.Comment: CVPR 2017 ora
Hyperparameter-free losses for model-based monocular reconstruction
This work proposes novel hyperparameter-free losses for single view 3D reconstruction with morphable models (3DMM). We dispense with the hyperparameters used in other works by exploiting geometry, so that the shape of the object and the camera pose are jointly optimized in a sole term expression. This simplification reduces the optimization time and its complexity. Moreover, we propose a novel implicit regularization technique based on random virtual projections that does not require additional 2D or 3D annotations. Our experiments suggest that minimizing a shape reprojection error together with the proposed implicit regularization is especially suitable for applications that require precise alignment between geometry and image spaces, such as augmented reality. We evaluate our losses on a large scale dataset with 3D ground truth and publish our implementations to facilitate reproducibility and public benchmarking in this field.Peer ReviewedPostprint (author's final draft
i3DMM: Deep Implicit 3D Morphable Model of Human Heads
We present the first deep implicit 3D morphable model (i3DMM) of full heads. Unlike earlier morphable face models it not only captures identity-specific geometry, texture, and expressions of the frontal face, but also models the entire head, including hair. We collect a new dataset consisting of 64 people with different expressions and hairstyles to train i3DMM. Our approach has the following favorable properties: (i) It is the first full head morphable model that includes hair. (ii) In contrast to mesh-based models it can be trained on merely rigidly aligned scans, without requiring difficult non-rigid registration. (iii) We design a novel architecture to decouple the shape model into an implicit reference shape and a deformation of this reference shape. With that, dense correspondences between shapes can be learned implicitly. (iv) This architecture allows us to semantically disentangle the geometry and color components, as color is learned in the reference space. Geometry is further disentangled as identity, expressions, and hairstyle, while color is disentangled as identity and hairstyle components. We show the merits of i3DMM using ablation studies, comparisons to state-of-the-art models, and applications such as semantic head editing and texture transfer. We will make our model publicly available
Morphable Face Models - An Open Framework
In this paper, we present a novel open-source pipeline for face registration
based on Gaussian processes as well as an application to face image analysis.
Non-rigid registration of faces is significant for many applications in
computer vision, such as the construction of 3D Morphable face models (3DMMs).
Gaussian Process Morphable Models (GPMMs) unify a variety of non-rigid
deformation models with B-splines and PCA models as examples. GPMM separate
problem specific requirements from the registration algorithm by incorporating
domain-specific adaptions as a prior model. The novelties of this paper are the
following: (i) We present a strategy and modeling technique for face
registration that considers symmetry, multi-scale and spatially-varying
details. The registration is applied to neutral faces and facial expressions.
(ii) We release an open-source software framework for registration and
model-building, demonstrated on the publicly available BU3D-FE database. The
released pipeline also contains an implementation of an Analysis-by-Synthesis
model adaption of 2D face images, tested on the Multi-PIE and LFW database.
This enables the community to reproduce, evaluate and compare the individual
steps of registration to model-building and 3D/2D model fitting. (iii) Along
with the framework release, we publish a new version of the Basel Face Model
(BFM-2017) with an improved age distribution and an additional facial
expression model
PhoMoH: Implicit Photorealistic 3D Models of Human Heads
We present PhoMoH, a neural network methodology to construct generative
models of photo-realistic 3D geometry and appearance of human heads including
hair, beards, an oral cavity, and clothing. In contrast to prior work, PhoMoH
models the human head using neural fields, thus supporting complex topology.
Instead of learning a head model from scratch, we propose to augment an
existing expressive head model with new features. Concretely, we learn a highly
detailed geometry network layered on top of a mid-resolution head model
together with a detailed, local geometry-aware, and disentangled color field.
Our proposed architecture allows us to learn photo-realistic human head models
from relatively little data. The learned generative geometry and appearance
networks can be sampled individually and enable the creation of diverse and
realistic human heads. Extensive experiments validate our method qualitatively
and across different metrics.Comment: To be published at the International Conference on 3D Vision 202
GANHead: Towards Generative Animatable Neural Head Avatars
To bring digital avatars into people's lives, it is highly demanded to
efficiently generate complete, realistic, and animatable head avatars. This
task is challenging, and it is difficult for existing methods to satisfy all
the requirements at once. To achieve these goals, we propose GANHead
(Generative Animatable Neural Head Avatar), a novel generative head model that
takes advantages of both the fine-grained control over the explicit expression
parameters and the realistic rendering results of implicit representations.
Specifically, GANHead represents coarse geometry, fine-gained details and
texture via three networks in canonical space to obtain the ability to generate
complete and realistic head avatars. To achieve flexible animation, we define
the deformation filed by standard linear blend skinning (LBS), with the learned
continuous pose and expression bases and LBS weights. This allows the avatars
to be directly animated by FLAME parameters and generalize well to unseen poses
and expressions. Compared to state-of-the-art (SOTA) methods, GANHead achieves
superior performance on head avatar generation and raw scan fitting.Comment: Camera-ready for CVPR 2023. Project page:
https://wsj-sjtu.github.io/GANHead
SIDER: Single-Image Neural Optimization for Facial Geometric Detail Recovery
© 2011 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.We present SIDER (Single-Image neural optimization for facial geometric DEtail Recovery), a novel photometric optimization method that recovers detailed facial geometry from a single image in an unsupervised manner. Inspired by classical techniques of coarse-to-fine optimization and recent advances in implicit neural representations of 3D shape, SIDER combines a geometry prior based on statistical models and Signed Distance Functions (SDFs) to recover facial details from single images. First, it estimates a coarse geometry using a morphable model represented as an SDF. Next, it reconstructs facial geometry details by optimizing a photometric loss with respect to the ground truth image. In contrast to prior work, SIDER does not rely on any dataset priors and does not require additional supervision from multiple views, lighting changes or ground truth 3D shape. Extensive qualitative and quantitative evaluation demonstrates that our method achieves state-of-the-art on facial geometric detail recovery, using only a single in the-wild image.Peer ReviewedPostprint (author's final draft
Learning Neural Parametric Head Models
We propose a novel 3D morphable model for complete human heads based on hybrid neural fields. At the core of our model lies a neural parametric representation that disentangles identity and expressions in disjoint latent spaces. To this end, we capture a person's identity in a canonical space as a signed distance field (SDF), and model facial expressions with a neural deformation field. In addition, our representation achieves high-fidelity local detail by introducing an ensemble of local fields centered around facial anchor points. To facilitate generalization, we train our model on a newly-captured dataset of over 3700 head scans from 203 different identities using a custom high-end 3D scanning setup. Our dataset significantly exceeds comparable existing datasets, both with respect to quality and completeness of geometry, averaging around 3.5M mesh faces per scan 1 1 We will publicly release our dataset along with a public benchmark for both neural head avatar construction as well as an evaluation on a hidden test-set for inference-time fitting.. Finally, we demonstrate that our approach outperforms state-of-the-art methods in terms of fitting error and reconstruction quality
An Implicit Parametric Morphable Dental Model
3D Morphable models of the human body capture variations among subjects and
are useful in reconstruction and editing applications. Current dental models
use an explicit mesh scene representation and model only the teeth, ignoring
the gum. In this work, we present the first parametric 3D morphable dental
model for both teeth and gum. Our model uses an implicit scene representation
and is learned from rigidly aligned scans. It is based on a component-wise
representation for each tooth and the gum, together with a learnable latent
code for each of such components. It also learns a template shape thus enabling
several applications such as segmentation, interpolation, and tooth
replacement. Our reconstruction quality is on par with the most advanced global
implicit representations while enabling novel applications. Project page:
https://vcai.mpi-inf.mpg.de/projects/DMM
- …