2,243 research outputs found
Sketching-out virtual humans: From 2d storyboarding to immediate 3d character animation
Virtual beings are playing a remarkable role in today’s public entertainment, while ordinary users are still treated as audiences due to the lack of appropriate expertise, equipment, and computer skills. In this paper, we present a fast and intuitive storyboarding interface, which enables users to sketch-out 3D virtual humans, 2D/3D animations, and character intercommunication. We devised an intuitive “stick figurefleshing-outskin mapping” graphical animation pipeline, which realises the whole process of key framing, 3D pose reconstruction, virtual human modelling, motion path/timing control, and the final animation synthesis by almost pure 2D sketching. A “creative model-based method” is developed, which emulates a human perception process, to generate the 3D human bodies of variational sizes, shapes, and fat distributions. Meanwhile, our current system also supports the sketch-based crowd animation and the storyboarding of the 3D multiple character intercommunication. This system has been formally tested by various users on Tablet PC. After minimal training, even a beginner can create vivid virtual humans and animate them within minutes
DeepSketch2Face: A Deep Learning Based Sketching System for 3D Face and Caricature Modeling
Face modeling has been paid much attention in the field of visual computing.
There exist many scenarios, including cartoon characters, avatars for social
media, 3D face caricatures as well as face-related art and design, where
low-cost interactive face modeling is a popular approach especially among
amateur users. In this paper, we propose a deep learning based sketching system
for 3D face and caricature modeling. This system has a labor-efficient
sketching interface, that allows the user to draw freehand imprecise yet
expressive 2D lines representing the contours of facial features. A novel CNN
based deep regression network is designed for inferring 3D face models from 2D
sketches. Our network fuses both CNN and shape based features of the input
sketch, and has two independent branches of fully connected layers generating
independent subsets of coefficients for a bilinear face representation. Our
system also supports gesture based interactions for users to further manipulate
initial face models. Both user studies and numerical results indicate that our
sketching system can help users create face models quickly and effectively. A
significantly expanded face database with diverse identities, expressions and
levels of exaggeration is constructed to promote further research and
evaluation of face modeling techniques.Comment: 12 pages, 16 figures, to appear in SIGGRAPH 201
Local 3D Editing via 3D Distillation of CLIP Knowledge
3D content manipulation is an important computer vision task with many
real-world applications (e.g., product design, cartoon generation, and 3D
Avatar editing). Recently proposed 3D GANs can generate diverse photorealistic
3D-aware contents using Neural Radiance fields (NeRF). However, manipulation of
NeRF still remains a challenging problem since the visual quality tends to
degrade after manipulation and suboptimal control handles such as 2D semantic
maps are used for manipulations. While text-guided manipulations have shown
potential in 3D editing, such approaches often lack locality. To overcome these
problems, we propose Local Editing NeRF (LENeRF), which only requires text
inputs for fine-grained and localized manipulation. Specifically, we present
three add-on modules of LENeRF, the Latent Residual Mapper, the Attention Field
Network, and the Deformation Network, which are jointly used for local
manipulations of 3D features by estimating a 3D attention field. The 3D
attention field is learned in an unsupervised way, by distilling the zero-shot
mask generation capability of CLIP to the 3D space with multi-view guidance. We
conduct diverse experiments and thorough evaluations both quantitatively and
qualitatively.Comment: conference: CVPR 202
RaBit: Parametric Modeling of 3D Biped Cartoon Characters with a Topological-consistent Dataset
Assisting people in efficiently producing visually plausible 3D characters
has always been a fundamental research topic in computer vision and computer
graphics. Recent learning-based approaches have achieved unprecedented accuracy
and efficiency in the area of 3D real human digitization. However, none of the
prior works focus on modeling 3D biped cartoon characters, which are also in
great demand in gaming and filming. In this paper, we introduce 3DBiCar, the
first large-scale dataset of 3D biped cartoon characters, and RaBit, the
corresponding parametric model. Our dataset contains 1,500 topologically
consistent high-quality 3D textured models which are manually crafted by
professional artists. Built upon the data, RaBit is thus designed with a
SMPL-like linear blend shape model and a StyleGAN-based neural UV-texture
generator, simultaneously expressing the shape, pose, and texture. To
demonstrate the practicality of 3DBiCar and RaBit, various applications are
conducted, including single-view reconstruction, sketch-based modeling, and 3D
cartoon animation. For the single-view reconstruction setting, we find a
straightforward global mapping from input images to the output UV-based texture
maps tends to lose detailed appearances of some local parts (e.g., nose, ears).
Thus, a part-sensitive texture reasoner is adopted to make all important local
areas perceived. Experiments further demonstrate the effectiveness of our
method both qualitatively and quantitatively. 3DBiCar and RaBit are available
at gaplab.cuhk.edu.cn/projects/RaBit.Comment: CVPR 2023, Project page: https://gaplab.cuhk.edu.cn/projects/RaBit
- …