497 research outputs found
Recycle-GAN: Unsupervised Video Retargeting
We introduce a data-driven approach for unsupervised video retargeting that
translates content from one domain to another while preserving the style native
to a domain, i.e., if contents of John Oliver's speech were to be transferred
to Stephen Colbert, then the generated content/speech should be in Stephen
Colbert's style. Our approach combines both spatial and temporal information
along with adversarial losses for content translation and style preservation.
In this work, we first study the advantages of using spatiotemporal constraints
over spatial constraints for effective retargeting. We then demonstrate the
proposed approach for the problems where information in both space and time
matters such as face-to-face translation, flower-to-flower, wind and cloud
synthesis, sunrise and sunset.Comment: ECCV 2018; Please refer to project webpage for videos -
http://www.cs.cmu.edu/~aayushb/Recycle-GA
Pose-to-Motion: Cross-Domain Motion Retargeting with Pose Prior
Creating believable motions for various characters has long been a goal in
computer graphics. Current learning-based motion synthesis methods depend on
extensive motion datasets, which are often challenging, if not impossible, to
obtain. On the other hand, pose data is more accessible, since static posed
characters are easier to create and can even be extracted from images using
recent advancements in computer vision. In this paper, we utilize this
alternative data source and introduce a neural motion synthesis approach
through retargeting. Our method generates plausible motions for characters that
have only pose data by transferring motion from an existing motion capture
dataset of another character, which can have drastically different skeletons.
Our experiments show that our method effectively combines the motion features
of the source character with the pose features of the target character, and
performs robustly with small or noisy pose data sets, ranging from a few
artist-created poses to noisy poses estimated directly from images.
Additionally, a conducted user study indicated that a majority of participants
found our retargeted motion to be more enjoyable to watch, more lifelike in
appearance, and exhibiting fewer artifacts. Project page:
https://cyanzhao42.github.io/pose2motionComment: Project page: https://cyanzhao42.github.io/pose2motio
VToonify: Controllable High-Resolution Portrait Video Style Transfer
Generating high-quality artistic portrait videos is an important and
desirable task in computer graphics and vision. Although a series of successful
portrait image toonification models built upon the powerful StyleGAN have been
proposed, these image-oriented methods have obvious limitations when applied to
videos, such as the fixed frame size, the requirement of face alignment,
missing non-facial details and temporal inconsistency. In this work, we
investigate the challenging controllable high-resolution portrait video style
transfer by introducing a novel VToonify framework. Specifically, VToonify
leverages the mid- and high-resolution layers of StyleGAN to render
high-quality artistic portraits based on the multi-scale content features
extracted by an encoder to better preserve the frame details. The resulting
fully convolutional architecture accepts non-aligned faces in videos of
variable size as input, contributing to complete face regions with natural
motions in the output. Our framework is compatible with existing StyleGAN-based
image toonification models to extend them to video toonification, and inherits
appealing features of these models for flexible style control on color and
intensity. This work presents two instantiations of VToonify built upon Toonify
and DualStyleGAN for collection-based and exemplar-based portrait video style
transfer, respectively. Extensive experimental results demonstrate the
effectiveness of our proposed VToonify framework over existing methods in
generating high-quality and temporally-coherent artistic portrait videos with
flexible style controls.Comment: ACM Transactions on Graphics (SIGGRAPH Asia 2022). Code:
https://github.com/williamyang1991/VToonify Project page:
https://www.mmlab-ntu.com/project/vtoonify
- …