3 research outputs found
VIVE3D: Viewpoint-Independent Video Editing using 3D-Aware GANs
We introduce VIVE3D, a novel approach that extends the capabilities of
image-based 3D GANs to video editing and is able to represent the input video
in an identity-preserving and temporally consistent way. We propose two new
building blocks. First, we introduce a novel GAN inversion technique
specifically tailored to 3D GANs by jointly embedding multiple frames and
optimizing for the camera parameters. Second, besides traditional semantic face
edits (e.g. for age and expression), we are the first to demonstrate edits that
show novel views of the head enabled by the inherent properties of 3D GANs and
our optical flow-guided compositing technique to combine the head with the
background video. Our experiments demonstrate that VIVE3D generates
high-fidelity face edits at consistent quality from a range of camera
viewpoints which are composited with the original video in a temporally and
spatially consistent manner.Comment: CVPR 2023. Project webpage and video available at
http://afruehstueck.github.io/vive3
On the Robustness of Quality Measures for GANs
This work evaluates the robustness of quality measures of generative models
such as Inception Score (IS) and Fr\'echet Inception Distance (FID). Analogous
to the vulnerability of deep models against a variety of adversarial attacks,
we show that such metrics can also be manipulated by additive pixel
perturbations. Our experiments indicate that one can generate a distribution of
images with very high scores but low perceptual quality. Conversely, one can
optimize for small imperceptible perturbations that, when added to real world
images, deteriorate their scores. We further extend our evaluation to
generative models themselves, including the state of the art network
StyleGANv2. We show the vulnerability of both the generative model and the FID
against additive perturbations in the latent space. Finally, we show that the
FID can be robustified by simply replacing the standard Inception with a robust
Inception. We validate the effectiveness of the robustified metric through
extensive experiments, showing it is more robust against manipulation.Comment: Accepted at the European Conference in Computer Vision (ECCV 2022