6 research outputs found
Masked Discriminators for Content-Consistent Unpaired Image-to-Image Translation
A common goal of unpaired image-to-image translation is to preserve content
consistency between source images and translated images while mimicking the
style of the target domain. Due to biases between the datasets of both domains,
many methods suffer from inconsistencies caused by the translation process.
Most approaches introduced to mitigate these inconsistencies do not constrain
the discriminator, leading to an even more ill-posed training setup. Moreover,
none of these approaches is designed for larger crop sizes. In this work, we
show that masking the inputs of a global discriminator for both domains with a
content-based mask is sufficient to reduce content inconsistencies
significantly. However, this strategy leads to artifacts that can be traced
back to the masking process. To reduce these artifacts, we introduce a local
discriminator that operates on pairs of small crops selected with a similarity
sampling strategy. Furthermore, we apply this sampling strategy to sample
global input crops from the source and target dataset. In addition, we propose
feature-attentive denormalization to selectively incorporate content-based
statistics into the generator stream. In our experiments, we show that our
method achieves state-of-the-art performance in photorealistic sim-to-real
translation and weather translation and also performs well in day-to-night
translation. Additionally, we propose the cKVD metric, which builds on the sKVD
metric and enables the examination of translation quality at the class or
category level.Comment: 24 pages, 22 figures, under revie
Multimodal Structure-Consistent Image-to-Image Translation
Unpaired image-to-image translation is proven quite effective in boosting a CNN-based object detector for a different domain by means of data augmentation that can well preserve the image-objects in the translated images. Recently, multimodal GAN (Generative Adversarial Network) models have been proposed and were expected to further boost the detector accuracy by generating a diverse collection of images in the target domain, given only a single/labelled image in the source domain. However, images generated by multimodal GANs would achieve even worse detection accuracy than the ones by a unimodal GAN with better object preservation. In this work, we introduce cycle-structure consistency for generating diverse and structure-preserved translated images across complex domains, such as between day and night, for object detector training. Qualitative results show that our model, Multimodal AugGAN, can generate diverse and realistic images for the target domain. For quantitative comparisons, we evaluate other competing methods and ours by using the generated images to train YOLO, Faster R-CNN and FCN models and prove that our model achieves significant improvement and outperforms other methods on the detection accuracies and the FCN scores. Also, we demonstrate that our model could provide more diverse object appearances in the target domain through comparison on the perceptual distance metric