342 research outputs found
VToonify: Controllable High-Resolution Portrait Video Style Transfer
Generating high-quality artistic portrait videos is an important and
desirable task in computer graphics and vision. Although a series of successful
portrait image toonification models built upon the powerful StyleGAN have been
proposed, these image-oriented methods have obvious limitations when applied to
videos, such as the fixed frame size, the requirement of face alignment,
missing non-facial details and temporal inconsistency. In this work, we
investigate the challenging controllable high-resolution portrait video style
transfer by introducing a novel VToonify framework. Specifically, VToonify
leverages the mid- and high-resolution layers of StyleGAN to render
high-quality artistic portraits based on the multi-scale content features
extracted by an encoder to better preserve the frame details. The resulting
fully convolutional architecture accepts non-aligned faces in videos of
variable size as input, contributing to complete face regions with natural
motions in the output. Our framework is compatible with existing StyleGAN-based
image toonification models to extend them to video toonification, and inherits
appealing features of these models for flexible style control on color and
intensity. This work presents two instantiations of VToonify built upon Toonify
and DualStyleGAN for collection-based and exemplar-based portrait video style
transfer, respectively. Extensive experimental results demonstrate the
effectiveness of our proposed VToonify framework over existing methods in
generating high-quality and temporally-coherent artistic portrait videos with
flexible style controls.Comment: ACM Transactions on Graphics (SIGGRAPH Asia 2022). Code:
https://github.com/williamyang1991/VToonify Project page:
https://www.mmlab-ntu.com/project/vtoonify
Deep Learning for Free-Hand Sketch: A Survey
Free-hand sketches are highly illustrative, and have been widely used by
humans to depict objects or stories from ancient times to the present. The
recent prevalence of touchscreen devices has made sketch creation a much easier
task than ever and consequently made sketch-oriented applications increasingly
popular. The progress of deep learning has immensely benefited free-hand sketch
research and applications. This paper presents a comprehensive survey of the
deep learning techniques oriented at free-hand sketch data, and the
applications that they enable. The main contents of this survey include: (i) A
discussion of the intrinsic traits and unique challenges of free-hand sketch,
to highlight the essential differences between sketch data and other data
modalities, e.g., natural photos. (ii) A review of the developments of
free-hand sketch research in the deep learning era, by surveying existing
datasets, research topics, and the state-of-the-art methods through a detailed
taxonomy and experimental evaluation. (iii) Promotion of future work via a
discussion of bottlenecks, open problems, and potential research directions for
the community.Comment: This paper is accepted by IEEE TPAM
- …