59 research outputs found
Generative Face Completion
In this paper, we propose an effective face completion algorithm using a deep
generative model. Different from well-studied background completion, the face
completion task is more challenging as it often requires to generate
semantically new pixels for the missing key components (e.g., eyes and mouths)
that contain large appearance variations. Unlike existing nonparametric
algorithms that search for patches to synthesize, our algorithm directly
generates contents for missing regions based on a neural network. The model is
trained with a combination of a reconstruction loss, two adversarial losses and
a semantic parsing loss, which ensures pixel faithfulness and local-global
contents consistency. With extensive experimental results, we demonstrate
qualitatively and quantitatively that our model is able to deal with a large
area of missing pixels in arbitrary shapes and generate realistic face
completion results.Comment: Accepted by CVPR 201
Unsupervised Domain Adaptation for Face Recognition in Unlabeled Videos
Despite rapid advances in face recognition, there remains a clear gap between
the performance of still image-based face recognition and video-based face
recognition, due to the vast difference in visual quality between the domains
and the difficulty of curating diverse large-scale video datasets. This paper
addresses both of those challenges, through an image to video feature-level
domain adaptation approach, to learn discriminative video frame
representations. The framework utilizes large-scale unlabeled video data to
reduce the gap between different domains while transferring discriminative
knowledge from large-scale labeled still images. Given a face recognition
network that is pretrained in the image domain, the adaptation is achieved by
(i) distilling knowledge from the network to a video adaptation network through
feature matching, (ii) performing feature restoration through synthetic data
augmentation and (iii) learning a domain-invariant feature through a domain
adversarial discriminator. We further improve performance through a
discriminator-guided feature fusion that boosts high-quality frames while
eliminating those degraded by video domain-specific factors. Experiments on the
YouTube Faces and IJB-A datasets demonstrate that each module contributes to
our feature-level domain adaptation framework and substantially improves video
face recognition performance to achieve state-of-the-art accuracy. We
demonstrate qualitatively that the network learns to suppress diverse artifacts
in videos such as pose, illumination or occlusion without being explicitly
trained for them.Comment: accepted for publication at International Conference on Computer
Vision (ICCV) 201
DIP: Differentiable Interreflection-aware Physics-based Inverse Rendering
We present a physics-based inverse rendering method that learns the
illumination, geometry, and materials of a scene from posed multi-view RGB
images. To model the illumination of a scene, existing inverse rendering works
either completely ignore the indirect illumination or model it by coarse
approximations, leading to sub-optimal illumination, geometry, and material
prediction of the scene. In this work, we propose a physics-based illumination
model that explicitly traces the incoming indirect lights at each surface point
based on interreflection, followed by estimating each identified indirect light
through an efficient neural network. Furthermore, we utilize the Leibniz's
integral rule to resolve non-differentiability in the proposed illumination
model caused by one type of environment light -- the tangent lights. As a
result, the proposed interreflection-aware illumination model can be learned
end-to-end together with geometry and materials estimation. As a side product,
our physics-based inverse rendering model also facilitates flexible and
realistic material editing as well as relighting. Extensive experiments on both
synthetic and real-world datasets demonstrate that the proposed method performs
favorably against existing inverse rendering methods on novel view synthesis
and inverse rendering
- …