92 research outputs found
Audio-Driven Talking Face Generation with Diverse yet Realistic Facial Animations
Audio-driven talking face generation, which aims to synthesize talking faces
with realistic facial animations (including accurate lip movements, vivid
facial expression details and natural head poses) corresponding to the audio,
has achieved rapid progress in recent years. However, most existing work
focuses on generating lip movements only without handling the closely
correlated facial expressions, which degrades the realism of the generated
faces greatly. This paper presents DIRFA, a novel method that can generate
talking faces with diverse yet realistic facial animations from the same
driving audio. To accommodate fair variation of plausible facial animations for
the same audio, we design a transformer-based probabilistic mapping network
that can model the variational facial animation distribution conditioned upon
the input audio and autoregressively convert the audio signals into a facial
animation sequence. In addition, we introduce a temporally-biased mask into the
mapping network, which allows to model the temporal dependency of facial
animations and produce temporally smooth facial animation sequence. With the
generated facial animation sequence and a source image, photo-realistic talking
faces can be synthesized with a generic generation network. Extensive
experiments show that DIRFA can generate talking faces with realistic facial
animations effectively
POCE: Pose-Controllable Expression Editing
Facial expression editing has attracted increasing attention with the advance
of deep neural networks in recent years. However, most existing methods suffer
from compromised editing fidelity and limited usability as they either ignore
pose variations (unrealistic editing) or require paired training data (not easy
to collect) for pose controls. This paper presents POCE, an innovative
pose-controllable expression editing network that can generate realistic facial
expressions and head poses simultaneously with just unpaired training images.
POCE achieves the more accessible and realistic pose-controllable expression
editing by mapping face images into UV space, where facial expressions and head
poses can be disentangled and edited separately. POCE has two novel designs.
The first is self-supervised UV completion that allows to complete UV maps
sampled under different head poses, which often suffer from self-occlusions and
missing facial texture. The second is weakly-supervised UV editing that allows
to generate new facial expressions with minimal modification of facial
identity, where the synthesized expression could be controlled by either an
expression label or directly transplanted from a reference UV map via feature
transfer. Extensive experiments show that POCE can learn from unpaired face
images effectively, and the learned model can generate realistic and
high-fidelity facial expressions under various new poses
Auto-regressive Image Synthesis with Integrated Quantization
Deep generative models have achieved conspicuous progress in realistic image
synthesis with multifarious conditional inputs, while generating diverse yet
high-fidelity images remains a grand challenge in conditional image generation.
This paper presents a versatile framework for conditional image generation
which incorporates the inductive bias of CNNs and powerful sequence modeling of
auto-regression that naturally leads to diverse image generation. Instead of
independently quantizing the features of multiple domains as in prior research,
we design an integrated quantization scheme with a variational regularizer that
mingles the feature discretization in multiple domains, and markedly boosts the
auto-regressive modeling performance. Notably, the variational regularizer
enables to regularize feature distributions in incomparable latent spaces by
penalizing the intra-domain variations of distributions. In addition, we design
a Gumbel sampling strategy that allows to incorporate distribution uncertainty
into the auto-regressive training procedure. The Gumbel sampling substantially
mitigates the exposure bias that often incurs misalignment between the training
and inference stages and severely impairs the inference performance. Extensive
experiments over multiple conditional image generation tasks show that our
method achieves superior diverse image generation performance qualitatively and
quantitatively as compared with the state-of-the-art.Comment: Accepted to ECCV 2022 as Oral Presentatio
Pose-Free Neural Radiance Fields via Implicit Pose Regularization
Pose-free neural radiance fields (NeRF) aim to train NeRF with unposed
multi-view images and it has achieved very impressive success in recent years.
Most existing works share the pipeline of training a coarse pose estimator with
rendered images at first, followed by a joint optimization of estimated poses
and neural radiance field. However, as the pose estimator is trained with only
rendered images, the pose estimation is usually biased or inaccurate for real
images due to the domain gap between real images and rendered images, leading
to poor robustness for the pose estimation of real images and further local
minima in joint optimization. We design IR-NeRF, an innovative pose-free NeRF
that introduces implicit pose regularization to refine pose estimator with
unposed real images and improve the robustness of the pose estimation for real
images. With a collection of 2D images of a specific scene, IR-NeRF constructs
a scene codebook that stores scene features and captures the scene-specific
pose distribution implicitly as priors. Thus, the robustness of pose estimation
can be promoted with the scene priors according to the rationale that a 2D real
image can be well reconstructed from the scene codebook only when its estimated
pose lies within the pose distribution. Extensive experiments show that IR-NeRF
achieves superior novel view synthesis and outperforms the state-of-the-art
consistently across multiple synthetic and real datasets.Comment: Accepted by ICCV202
GMLight: Lighting Estimation via Geometric Distribution Approximation
Lighting estimation from a single image is an essential yet challenging task
in computer vision and computer graphics. Existing works estimate lighting by
regressing representative illumination parameters or generating illumination
maps directly. However, these methods often suffer from poor accuracy and
generalization. This paper presents Geometric Mover's Light (GMLight), a
lighting estimation framework that employs a regression network and a
generative projector for effective illumination estimation. We parameterize
illumination scenes in terms of the geometric light distribution, light
intensity, ambient term, and auxiliary depth, and estimate them as a pure
regression task. Inspired by the earth mover's distance, we design a novel
geometric mover's loss to guide the accurate regression of light distribution
parameters. With the estimated lighting parameters, the generative projector
synthesizes panoramic illumination maps with realistic appearance and
frequency. Extensive experiments show that GMLight achieves accurate
illumination estimation and superior fidelity in relighting for 3D object
insertion.Comment: 12 pages, 11 figures. arXiv admin note: text overlap with
arXiv:2012.1111
Reemerging Rabies and Lack of Systemic Surveillance in People’s Republic of China
Standardized protocols and diagnostic-based surveillance are imperative for detection and elimination
Ferret badger rabies origin and its revisited importance as potential source of rabies transmission in Southeast China
<p>Abstract</p> <p>Background</p> <p>The frequent occurrence of ferret badger-associated human rabies cases in southeast China highlights the lack of laboratory-based surveillance and urges revisiting the potential importance of this animal in rabies transmission. To determine if the ferret badgers actually contribute to human and dog rabies cases, and the possible origin of the ferret badger-associated rabies in the region, an active rabies survey was conducted to determine the frequency of rabies infection and seroprevalence in dogs and ferret badgers.</p> <p>Methods</p> <p>A retrospective survey on rabies epidemics was performed in Zhejiang, Jiangxi and Anhui provinces in southeast China. The brain tissues from ferret badgers and dogs were assayed by fluorescent antibody test. Rabies virus was isolated and sequenced for phylogenetic analysis. The sera from ferret badgers and dogs were titrated using rabies virus neutralizing antibodies (VNA) test.</p> <p>Results</p> <p>The ferret badgers presented a higher percentage of rabies seroconversion than dogs did in the endemic region, reaching a maximum of 95% in the collected samples. Nine ferret badger-associated rabies viruses were isolated, sequenced, and were phylogenetically clustered as a separate group. Nucleotide sequence revealed 99.4-99.8% homology within the ferret badger isolates, and 83-89% homology to the dog isolates in the nucleoprotein and glycoprotein genes in the same rabies endemic regions.</p> <p>Conclusions</p> <p>Our data suggest ferret badger-associated rabies has likely formed as an independent enzootic originating from dogs during the long-term rabies infestation in southeast China. The eventual role of FB rabies in public health remains unclear. However, management of ferret badger bites, rabies awareness and control in the related regions should be an immediate need.</p
Deep learning for facial expression editing
In this day and age of digital media, facial expression editing, which aims to transform the facial expression of a source facial image to a desired one without changing the face identity, has attracted increasing interest from both academia and industrial communities due to its wide applications in many tasks. Automatic facial expression editing has been explored extensively with the prevalence of generative adversarial networks in recent years. Although some research works have been reported and achieved very promising progress, the task of facial expression editing is still facing four major challenges, including the unsatisfactory editing quality issue, the constrained data annotation issue, the limited controllability issue and the multi-modality issue. This thesis focuses on the above-mentioned challenges in facial expression editing task and introduces several novel deep-learning-based techniques to alleviate the corresponding challenges. Extensive experiments show that the proposed approaches achieve superior performance in facial expression editing.Doctor of Philosoph
- …