110 research outputs found
Warp-Guided GANs for Single-Photo Facial Animation
This paper introduces a novel method for realtime portrait animation in a single photo. Our method requires only a single portrait photo and a set of facial landmarks derived from a driving source (e.g., a photo or a video sequence), and generates an animated image with rich facial details. The core of our method is a warp-guided generative model that instantly fuses various fine facial details (e.g., creases and wrinkles), which are necessary to generate a high-fidelity facial expression, onto a pre-warped image. Our method factorizes out the nonlinear geometric transformations exhibited in facial expressions by lightweight 2D warps and leaves the appearance detail synthesis to conditional generative neural networks for high-fidelity facial animation generation. We show such a factorization of geometric transformation and appearance synthesis largely helps the network better learn the high nonlinearity of the facial expression functions and also facilitates the design of the network architecture. Through extensive experiments on various portrait photos from the Internet, we show the significant efficacy of our method compared with prior arts
FLNet: Landmark Driven Fetching and Learning Network for Faithful Talking Facial Animation Synthesis
Talking face synthesis has been widely studied in either appearance-based or
warping-based methods. Previous works mostly utilize single face image as a
source, and generate novel facial animations by merging other person's facial
features. However, some facial regions like eyes or teeth, which may be hidden
in the source image, can not be synthesized faithfully and stably. In this
paper, We present a landmark driven two-stream network to generate faithful
talking facial animation, in which more facial details are created, preserved
and transferred from multiple source images instead of a single one.
Specifically, we propose a network consisting of a learning and fetching
stream. The fetching sub-net directly learns to attentively warp and merge
facial regions from five source images of distinctive landmarks, while the
learning pipeline renders facial organs from the training face space to
compensate. Compared to baseline algorithms, extensive experiments demonstrate
that the proposed method achieves a higher performance both quantitatively and
qualitatively. Codes are at https://github.com/kgu3/FLNet_AAAI2020.Comment: Accepted by AAAI 202
Free-HeadGAN: Neural Talking Head Synthesis with Explicit Gaze Control
We present Free-HeadGAN, a person-generic neural talking head synthesis
system. We show that modeling faces with sparse 3D facial landmarks are
sufficient for achieving state-of-the-art generative performance, without
relying on strong statistical priors of the face, such as 3D Morphable Models.
Apart from 3D pose and facial expressions, our method is capable of fully
transferring the eye gaze, from a driving actor to a source identity. Our
complete pipeline consists of three components: a canonical 3D key-point
estimator that regresses 3D pose and expression-related deformations, a gaze
estimation network and a generator that is built upon the architecture of
HeadGAN. We further experiment with an extension of our generator to
accommodate few-shot learning using an attention mechanism, in case more than
one source images are available. Compared to the latest models for reenactment
and motion transfer, our system achieves higher photo-realism combined with
superior identity preservation, while offering explicit gaze control
ICface: Interpretable and Controllable Face Reenactment Using GANs
This paper presents a generic face animator that is able to control the pose
and expressions of a given face image. The animation is driven by human
interpretable control signals consisting of head pose angles and the Action
Unit (AU) values. The control information can be obtained from multiple sources
including external driving videos and manual controls. Due to the interpretable
nature of the driving signal, one can easily mix the information between
multiple sources (e.g. pose from one image and expression from another) and
apply selective post-production editing. The proposed face animator is
implemented as a two-stage neural network model that is learned in a
self-supervised manner using a large video collection. The proposed
Interpretable and Controllable face reenactment network (ICface) is compared to
the state-of-the-art neural network-based face animation techniques in multiple
tasks. The results indicate that ICface produces better visual quality while
being more versatile than most of the comparison methods. The introduced model
could provide a lightweight and easy to use tool for a multitude of advanced
image and video editing tasks.Comment: Accepted in WACV-202
- …