19 research outputs found
SEAN: Image Synthesis with Semantic Region-Adaptive Normalization
We propose semantic region-adaptive normalization (SEAN), a simple but
effective building block for Generative Adversarial Networks conditioned on
segmentation masks that describe the semantic regions in the desired output
image. Using SEAN normalization, we can build a network architecture that can
control the style of each semantic region individually, e.g., we can specify
one style reference image per region. SEAN is better suited to encode,
transfer, and synthesize style than the best previous method in terms of
reconstruction quality, variability, and visual quality. We evaluate SEAN on
multiple datasets and report better quantitative metrics (e.g. FID, PSNR) than
the current state of the art. SEAN also pushes the frontier of interactive
image editing. We can interactively edit images by changing segmentation masks
or the style for any given region. We can also interpolate styles from two
reference images per region.Comment: Accepted as a CVPR 2020 oral paper. The interactive demo is available
at https://youtu.be/0Vbj9xFgoU
Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?
We propose an efficient algorithm to embed a given image into the latent
space of StyleGAN. This embedding enables semantic image editing operations
that can be applied to existing photographs. Taking the StyleGAN trained on the
FFHQ dataset as an example, we show results for image morphing, style transfer,
and expression transfer. Studying the results of the embedding algorithm
provides valuable insights into the structure of the StyleGAN latent space. We
propose a set of experiments to test what class of images can be embedded, how
they are embedded, what latent space is suitable for embedding, and if the
embedding is semantically meaningful.Comment: Accepted for oral presentation at ICCV 2019, "For videos visit
https://youtu.be/RnTXLXw9o_I , https://youtu.be/zJoYY2eHAF0 and
https://youtu.be/bA893L-PjbI
Labels4Free: Unsupervised Segmentation using StyleGAN
We propose an unsupervised segmentation framework for StyleGAN generated objects. We build on two main observations. First, the features generated by StyleGAN hold valuable information that can be utilized towards training segmentation networks. Second, the foreground and background can often be treated to be largely independent and be swapped across images to produce plausible composited images. For our solution, we propose to augment the StyleGAN2 generator architecture with a segmentation branch and to split the generator into a foreground and background network. This enables us to generate soft segmentation masks for the foreground object in an unsupervised fashion. On multiple object classes, we report comparable results against state-of-the-art supervised segmentation networks, while against the best unsupervised segmentation approach we demonstrate a clear improvement, both in qualitative and quantitative metrics. Project Page: https:/rameenabdal.github.io/Labels4Free
Image2StyleGAN: How to embed images into the styleGAN latent space?
We propose an efficient algorithm to embed a given image into the latent space of StyleGAN. This embedding enables semantic image editing operations that can be applied to existing photographs. Taking the StyleGAN trained on the FFHD dataset as an example, we show results for image morphing, style transfer, and expression transfer. Studying the results of the embedding algorithm provides valuable insights into the structure of the StyleGAN latent space. We propose a set of experiments to test what class of images can be embedded, how they are embedded, what latent space is suitable for embedding, and if the embedding is semantically meaningful
3DAvatarGAN: Bridging Domains for Personalized Editable Avatars
Modern 3D-GANs synthesize geometry and texture by training on large-scale
datasets with a consistent structure. Training such models on stylized,
artistic data, with often unknown, highly variable geometry, and camera
information has not yet been shown possible. Can we train a 3D GAN on such
artistic data, while maintaining multi-view consistency and texture quality? To
this end, we propose an adaptation framework, where the source domain is a
pre-trained 3D-GAN, while the target domain is a 2D-GAN trained on artistic
datasets. We then distill the knowledge from a 2D generator to the source 3D
generator. To do that, we first propose an optimization-based method to align
the distributions of camera parameters across domains. Second, we propose
regularizations necessary to learn high-quality texture, while avoiding
degenerate geometric solutions, such as flat shapes. Third, we show a
deformation-based technique for modeling exaggerated geometry of artistic
domains, enabling -- as a byproduct -- personalized geometric editing. Finally,
we propose a novel inversion method for 3D-GANs linking the latent spaces of
the source and the target domains. Our contributions -- for the first time --
allow for the generation, editing, and animation of personalized artistic 3D
avatars on artistic datasets.Comment: Project Page: https://rameenabdal.github.io/3DAvatarGAN