120 research outputs found
Demystifying Neural Style Transfer
Neural Style Transfer has recently demonstrated very exciting results which
catches eyes in both academia and industry. Despite the amazing results, the
principle of neural style transfer, especially why the Gram matrices could
represent style remains unclear. In this paper, we propose a novel
interpretation of neural style transfer by treating it as a domain adaptation
problem. Specifically, we theoretically show that matching the Gram matrices of
feature maps is equivalent to minimize the Maximum Mean Discrepancy (MMD) with
the second order polynomial kernel. Thus, we argue that the essence of neural
style transfer is to match the feature distributions between the style images
and the generated images. To further support our standpoint, we experiment with
several other distribution alignment methods, and achieve appealing results. We
believe this novel interpretation connects these two important research fields,
and could enlighten future researches.Comment: Accepted by IJCAI 201
Bringing Impressionism to Life with Neural Style Transfer in Come Swim
Neural Style Transfer is a striking, recently-developed technique that uses
neural networks to artistically redraw an image in the style of a source style
image. This paper explores the use of this technique in a production setting,
applying Neural Style Transfer to redraw key scenes in 'Come Swim' in the style
of the impressionistic painting that inspired the film. We document how the
technique can be driven within the framework of an iterative creative process
to achieve a desired look, and propose a mapping of the broad parameter space
to a key set of creative controls. We hope that this mapping can provide
insights into priorities for future research.Comment: 3 pages, 6 figures, paper is a case study of how Neural Style
Transfer can be used in a movie production contex
Weakly-supervised Caricature Face Parsing through Domain Adaptation
A caricature is an artistic form of a person's picture in which certain
striking characteristics are abstracted or exaggerated in order to create a
humor or sarcasm effect. For numerous caricature related applications such as
attribute recognition and caricature editing, face parsing is an essential
pre-processing step that provides a complete facial structure understanding.
However, current state-of-the-art face parsing methods require large amounts of
labeled data on the pixel-level and such process for caricature is tedious and
labor-intensive. For real photos, there are numerous labeled datasets for face
parsing. Thus, we formulate caricature face parsing as a domain adaptation
problem, where real photos play the role of the source domain, adapting to the
target caricatures. Specifically, we first leverage a spatial transformer based
network to enable shape domain shifts. A feed-forward style transfer network is
then utilized to capture texture-level domain gaps. With these two steps, we
synthesize face caricatures from real photos, and thus we can use parsing
ground truths of the original photos to learn the parsing model. Experimental
results on the synthetic and real caricatures demonstrate the effectiveness of
the proposed domain adaptation algorithm. Code is available at:
https://github.com/ZJULearning/CariFaceParsing .Comment: Accepted in ICIP 2019, code and model are available at
https://github.com/ZJULearning/CariFaceParsin
TET-GAN: Text Effects Transfer via Stylization and Destylization
Text effects transfer technology automatically makes the text dramatically
more impressive. However, previous style transfer methods either study the
model for general style, which cannot handle the highly-structured text effects
along the glyph, or require manual design of subtle matching criteria for text
effects. In this paper, we focus on the use of the powerful representation
abilities of deep neural features for text effects transfer. For this purpose,
we propose a novel Texture Effects Transfer GAN (TET-GAN), which consists of a
stylization subnetwork and a destylization subnetwork. The key idea is to train
our network to accomplish both the objective of style transfer and style
removal, so that it can learn to disentangle and recombine the content and
style features of text effects images. To support the training of our network,
we propose a new text effects dataset with as much as 64 professionally
designed styles on 837 characters. We show that the disentangled feature
representations enable us to transfer or remove all these styles on arbitrary
glyphs using one network. Furthermore, the flexible network design empowers
TET-GAN to efficiently extend to a new text style via one-shot learning where
only one example is required. We demonstrate the superiority of the proposed
method in generating high-quality stylized text over the state-of-the-art
methods.Comment: Accepted by AAAI 2019. Code and dataset will be available at
http://www.icst.pku.edu.cn/struct/Projects/TETGAN.htm
- …