11 research outputs found
Where and Who? Automatic Semantic-Aware Person Composition
Image compositing is a method used to generate realistic yet fake imagery by
inserting contents from one image to another. Previous work in compositing has
focused on improving appearance compatibility of a user selected foreground
segment and a background image (i.e. color and illumination consistency). In
this work, we instead develop a fully automated compositing model that
additionally learns to select and transform compatible foreground segments from
a large collection given only an input image background. To simplify the task,
we restrict our problem by focusing on human instance composition, because
human segments exhibit strong correlations with their background and because of
the availability of large annotated data. We develop a novel branching
Convolutional Neural Network (CNN) that jointly predicts candidate person
locations given a background image. We then use pre-trained deep feature
representations to retrieve person instances from a large segment database.
Experimental results show that our model can generate composite images that
look visually convincing. We also develop a user interface to demonstrate the
potential application of our method.Comment: 10 pages, 9 figure
SG-VAE: Scene Grammar Variational Autoencoder to generate new indoor scenes
Deep generative models have been used in recent years to learn coherent
latent representations in order to synthesize high-quality images. In this
work, we propose a neural network to learn a generative model for sampling
consistent indoor scene layouts. Our method learns the co-occurrences, and
appearance parameters such as shape and pose, for different objects categories
through a grammar-based auto-encoder, resulting in a compact and accurate
representation for scene layouts. In contrast to existing grammar-based methods
with a user-specified grammar, we construct the grammar automatically by
extracting a set of production rules on reasoning about object co-occurrences
in training data. The extracted grammar is able to represent a scene by an
augmented parse tree. The proposed auto-encoder encodes these parse trees to a
latent code, and decodes the latent code to a parse tree, thereby ensuring the
generated scene is always valid. We experimentally demonstrate that the
proposed auto-encoder learns not only to generate valid scenes (i.e. the
arrangements and appearances of objects), but it also learns coherent latent
representations where nearby latent samples decode to similar scene outputs.
The obtained generative model is applicable to several computer vision tasks
such as 3D pose and layout estimation from RGB-D data