774 research outputs found
SS-CPGAN: Self-Supervised Cut-and-Pasting Generative Adversarial Network for Object Segmentation
This paper proposes a novel self-supervised based Cut-and-Paste GAN to
perform foreground object segmentation and generate realistic composite images
without manual annotations. We accomplish this goal by a simple yet effective
self-supervised approach coupled with the U-Net based discriminator. The
proposed method extends the ability of the standard discriminators to learn not
only the global data representations via classification (real/fake) but also
learn semantic and structural information through pseudo labels created using
the self-supervised task. The proposed method empowers the generator to create
meaningful masks by forcing it to learn informative per-pixel as well as global
image feedback from the discriminator. Our experiments demonstrate that our
proposed method significantly outperforms the state-of-the-art methods on the
standard benchmark datasets
Distilling Localization for Self-Supervised Representation Learning
Recent progress in contrastive learning has revolutionized unsupervised
representation learning. Concretely, multiple views (augmentations) from the
same image are encouraged to map to the similar embeddings, while views from
different images are pulled apart. In this paper, through visualizing and
diagnosing classification errors, we observe that current contrastive models
are ineffective at localizing the foreground object, limiting their ability to
extract discriminative high-level features. This is due to the fact that view
generation process considers pixels in an image uniformly. To address this
problem, we propose a data-driven approach for learning invariance to
backgrounds. It first estimates foreground saliency in images and then creates
augmentations by copy-and-pasting the foreground onto a variety of backgrounds.
The learning still follows the instance discrimination pretext task, so that
the representation is trained to disregard background content and focus on the
foreground. We study a variety of saliency estimation methods, and find that
most methods lead to improvements for contrastive learning. With this approach
(DiLo), significant performance is achieved for self-supervised learning on
ImageNet classification, and also for object detection on PASCAL VOC and
MSCOCO.Comment: Accepted by AAAI202
Semantic Counting from Self-Collages
While recent supervised methods for reference-based object counting continue
to improve the performance on benchmark datasets, they have to rely on small
datasets due to the cost associated with manually annotating dozens of objects
in images. We propose Unsupervised Counter (UnCo), a model that can learn this
task without requiring any manual annotations. To this end, we construct
"SelfCollages", images with various pasted objects as training samples, that
provide a rich learning signal covering arbitrary object types and counts. Our
method builds on existing unsupervised representations and segmentation
techniques to successfully demonstrate the ability to count objects without
manual supervision. Our experiments show that our method not only outperforms
simple baselines and generic models such as FasterRCNN, but also matches the
performance of supervised counting models in some domains.Comment: 24 pages. Code available at
https://github.com/lukasknobel/SelfCollage
Staging E-Commerce Products for Online Advertising using Retrieval Assisted Image Generation
Online ads showing e-commerce products typically rely on the product images
in a catalog sent to the advertising platform by an e-commerce platform. In the
broader ads industry such ads are called dynamic product ads (DPA). It is
common for DPA catalogs to be in the scale of millions (corresponding to the
scale of products which can be bought from the e-commerce platform). However,
not all product images in the catalog may be appealing when directly
re-purposed as an ad image, and this may lead to lower click-through rates
(CTRs). In particular, products just placed against a solid background may not
be as enticing and realistic as a product staged in a natural environment. To
address such shortcomings of DPA images at scale, we propose a generative
adversarial network (GAN) based approach to generate staged backgrounds for
un-staged product images. Generating the entire staged background is a
challenging task susceptible to hallucinations. To get around this, we
introduce a simpler approach called copy-paste staging using retrieval assisted
GANs. In copy paste staging, we first retrieve (from the catalog) staged
products similar to the un-staged input product, and then copy-paste the
background of the retrieved product in the input image. A GAN based in-painting
model is used to fill the holes left after this copy-paste operation. We show
the efficacy of our copy-paste staging method via offline metrics, and human
evaluation. In addition, we show how our staging approach can enable animations
of moving products leading to a video ad from a product image.Comment: Accepted for publication in AdKDD 202
- …