One-shot talking head generation produces lip-sync talking heads based on
arbitrary audio and one source face. To guarantee the naturalness and realness,
recent methods propose to achieve free pose control instead of simply editing
mouth areas. However, existing methods do not preserve accurate identity of
source face when generating head motions. To solve the identity mismatch
problem and achieve high-quality free pose control, we present One-shot
Pose-controllable Talking head generation network (OPT). Specifically, the
Audio Feature Disentanglement Module separates content features from audios,
eliminating the influence of speaker-specific information contained in
arbitrary driving audios. Later, the mouth expression feature is extracted from
the content feature and source face, during which the landmark loss is designed
to enhance the accuracy of facial structure and identity preserving quality.
Finally, to achieve free pose control, controllable head pose features from
reference videos are fed into the Video Generator along with the expression
feature and source face to generate new talking heads. Extensive quantitative
and qualitative experimental results verify that OPT generates high-quality
pose-controllable talking heads with no identity mismatch problem,
outperforming previous SOTA methods.Comment: Accepted by ICASSP202