8,927 research outputs found
Characterizing and Improving Stability in Neural Style Transfer
Recent progress in style transfer on images has focused on improving the
quality of stylized images and speed of methods. However, real-time methods are
highly unstable resulting in visible flickering when applied to videos. In this
work we characterize the instability of these methods by examining the solution
set of the style transfer objective. We show that the trace of the Gram matrix
representing style is inversely related to the stability of the method. Then,
we present a recurrent convolutional network for real-time video style transfer
which incorporates a temporal consistency loss and overcomes the instability of
prior methods. Our networks can be applied at any resolution, do not re- quire
optical flow at test time, and produce high quality, temporally consistent
stylized videos in real-time
Manipulating Attributes of Natural Scenes via Hallucination
In this study, we explore building a two-stage framework for enabling users
to directly manipulate high-level attributes of a natural scene. The key to our
approach is a deep generative network which can hallucinate images of a scene
as if they were taken at a different season (e.g. during winter), weather
condition (e.g. in a cloudy day) or time of the day (e.g. at sunset). Once the
scene is hallucinated with the given attributes, the corresponding look is then
transferred to the input image while preserving the semantic details intact,
giving a photo-realistic manipulation result. As the proposed framework
hallucinates what the scene will look like, it does not require any reference
style image as commonly utilized in most of the appearance or style transfer
approaches. Moreover, it allows to simultaneously manipulate a given scene
according to a diverse set of transient attributes within a single model,
eliminating the need of training multiple networks per each translation task.
Our comprehensive set of qualitative and quantitative results demonstrate the
effectiveness of our approach against the competing methods.Comment: Accepted for publication in ACM Transactions on Graphic
Optical Flow Distillation: Towards Efficient and Stable Video Style Transfer
Video style transfer techniques inspire many exciting applications on mobile
devices. However, their efficiency and stability are still far from
satisfactory. To boost the transfer stability across frames, optical flow is
widely adopted, despite its high computational complexity, e.g. occupying over
97% inference time. This paper proposes to learn a lightweight video style
transfer network via knowledge distillation paradigm. We adopt two teacher
networks, one of which takes optical flow during inference while the other does
not. The output difference between these two teacher networks highlights the
improvements made by optical flow, which is then adopted to distill the target
student network. Furthermore, a low-rank distillation loss is employed to
stabilize the output of student network by mimicking the rank of input videos.
Extensive experiments demonstrate that our student network without an optical
flow module is still able to generate stable video and runs much faster than
the teacher network
- …