56 research outputs found
SEAN: Image Synthesis with Semantic Region-Adaptive Normalization
We propose semantic region-adaptive normalization (SEAN), a simple but
effective building block for Generative Adversarial Networks conditioned on
segmentation masks that describe the semantic regions in the desired output
image. Using SEAN normalization, we can build a network architecture that can
control the style of each semantic region individually, e.g., we can specify
one style reference image per region. SEAN is better suited to encode,
transfer, and synthesize style than the best previous method in terms of
reconstruction quality, variability, and visual quality. We evaluate SEAN on
multiple datasets and report better quantitative metrics (e.g. FID, PSNR) than
the current state of the art. SEAN also pushes the frontier of interactive
image editing. We can interactively edit images by changing segmentation masks
or the style for any given region. We can also interpolate styles from two
reference images per region.Comment: Accepted as a CVPR 2020 oral paper. The interactive demo is available
at https://youtu.be/0Vbj9xFgoU
Road layout understanding by generative adversarial inpainting
Autonomous driving is becoming a reality, yet vehicles still need to rely on complex sensor fusion to understand the scene they act in. The ability to discern static environment and dynamic entities provides a comprehension of the road layout that poses constraints to the reasoning process about moving objects. We pursue this through a GAN-based semantic segmentation inpainting model to remove all dynamic objects from the scene and focus on understanding its static components such as streets, sidewalks and buildings. We evaluate this task on the Cityscapes dataset and on a novel synthetically generated dataset obtained with the CARLA simulator and specifically designed to quantitatively evaluate semantic segmentation inpaintings. We compare our methods with a variety of baselines working both in the RGB and segmentation domains
Blending Generative Adversarial Image Synthesis with Rendering for Computer Graphics
Conventional computer graphics pipelines require detailed 3D models, meshes,
textures, and rendering engines to generate 2D images from 3D scenes. These
processes are labor-intensive. We introduce Hybrid Neural Computer Graphics
(HNCG) as an alternative. The contribution is a novel image formation strategy
to reduce the 3D model and texture complexity of computer graphics pipelines.
Our main idea is straightforward: Given a 3D scene, render only important
objects of interest and use generative adversarial processes for synthesizing
the rest of the image. To this end, we propose a novel image formation strategy
to form 2D semantic images from 3D scenery consisting of simple object models
without textures. These semantic images are then converted into photo-realistic
RGB images with a state-of-the-art conditional Generative Adversarial Network
(cGAN) based image synthesizer trained on real-world data. Meanwhile, objects
of interest are rendered using a physics-based graphics engine. This is
necessary as we want to have full control over the appearance of objects of
interest. Finally, the partially-rendered and cGAN synthesized images are
blended with a blending GAN. We show that the proposed framework outperforms
conventional rendering with ablation and comparison studies. Semantic retention
and Fr\'echet Inception Distance (FID) measurements were used as the main
performance metrics
- …