8,292 research outputs found
Training and Tuning Generative Neural Radiance Fields for Attribute-Conditional 3D-Aware Face Generation
Generative Neural Radiance Fields (GNeRF) based 3D-aware GANs have
demonstrated remarkable capabilities in generating high-quality images while
maintaining strong 3D consistency. Notably, significant advancements have been
made in the domain of face generation. However, most existing models prioritize
view consistency over disentanglement, resulting in limited semantic/attribute
control during generation. To address this limitation, we propose a conditional
GNeRF model incorporating specific attribute labels as input to enhance the
controllability and disentanglement abilities of 3D-aware generative models.
Our approach builds upon a pre-trained 3D-aware face model, and we introduce a
Training as Init and Optimizing for Tuning (TRIOT) method to train a
conditional normalized flow module to enable the facial attribute editing, then
optimize the latent vector to improve attribute-editing precision further. Our
extensive experiments demonstrate that our model produces high-quality edits
with superior view consistency while preserving non-target regions. Code is
available at https://github.com/zhangqianhui/TT-GNeRF.Comment: 13 page
Bidirectionally Deformable Motion Modulation For Video-based Human Pose Transfer
Video-based human pose transfer is a video-to-video generation task that
animates a plain source human image based on a series of target human poses.
Considering the difficulties in transferring highly structural patterns on the
garments and discontinuous poses, existing methods often generate
unsatisfactory results such as distorted textures and flickering artifacts. To
address these issues, we propose a novel Deformable Motion Modulation (DMM)
that utilizes geometric kernel offset with adaptive weight modulation to
simultaneously perform feature alignment and style transfer. Different from
normal style modulation used in style transfer, the proposed modulation
mechanism adaptively reconstructs smoothed frames from style codes according to
the object shape through an irregular receptive field of view. To enhance the
spatio-temporal consistency, we leverage bidirectional propagation to extract
the hidden motion information from a warped image sequence generated by noisy
poses. The proposed feature propagation significantly enhances the motion
prediction ability by forward and backward propagation. Both quantitative and
qualitative experimental results demonstrate superiority over the
state-of-the-arts in terms of image fidelity and visual continuity. The source
code is publicly available at github.com/rocketappslab/bdmm.Comment: ICCV 202
Lagrangian Neural Style Transfer for Fluids
Artistically controlling the shape, motion and appearance of fluid
simulations pose major challenges in visual effects production. In this paper,
we present a neural style transfer approach from images to 3D fluids formulated
in a Lagrangian viewpoint. Using particles for style transfer has unique
benefits compared to grid-based techniques. Attributes are stored on the
particles and hence are trivially transported by the particle motion. This
intrinsically ensures temporal consistency of the optimized stylized structure
and notably improves the resulting quality. Simultaneously, the expensive,
recursive alignment of stylization velocity fields of grid approaches is
unnecessary, reducing the computation time to less than an hour and rendering
neural flow stylization practical in production settings. Moreover, the
Lagrangian representation improves artistic control as it allows for
multi-fluid stylization and consistent color transfer from images, and the
generality of the method enables stylization of smoke and liquids likewise.Comment: ACM Transaction on Graphics (SIGGRAPH 2020), additional materials:
http://www.byungsoo.me/project/lnst/index.htm
Visual modeling and simulation of multiscale phenomena
Many large-scale systems seen in real life, such as human crowds, fluids, and granular materials, exhibit complicated motion at many different scales, from a characteristic global behavior to important small-scale detail. Such multiscale systems are computationally expensive for traditional simulation techniques to capture over the full range of scales. In this dissertation, I present novel techniques for scalable and efficient simulation of these large, complex phenomena for visual computing applications. These techniques are based on a new approach of representing a complex system by coupling together separate models for its large-scale and fine-scale dynamics. In fluid simulation, it remains a challenge to efficiently simulate fine local detail such as foam, ripples, and turbulence without compromising the accuracy of the large-scale flow. I present two techniques for this problem that combine physically-based numerical simulation for the global flow with efficient local models for detail. For surface features, I propose the use of texture synthesis, guided by the physical characteristics of the macroscopic flow. For turbulence in the fluid motion itself, I present a technique that tracks the transfer of energy from the mean flow to the turbulent fluctuations and synthesizes these fluctuations procedurally, allowing extremely efficient visual simulation of turbulent fluids. Another large class of problems which are not easily handled by traditional approaches is the simulation of very large aggregates of discrete entities, such as dense pedestrian crowds and granular materials. I present a technique for crowd simulation that couples a discrete per-agent model of individual navigation with a novel continuum formulation for the collective motion of pedestrians. This approach allows simulation of dense crowds of a hundred thousand agents at near-real-time rates on desktop computers. I also present a technique for simulating granular materials, which generalizes this model and introduces a novel computational scheme for friction. This method efficiently reproduces a wide range of granular behavior and allows two-way interaction with simulated solid bodies. In all of these cases, the proposed techniques are typically an order of magnitude faster than comparable existing methods. Through these applications to a diverse set of challenging simulation problems, I demonstrate the benefits of the proposed approach, showing that it is a powerful and versatile technique for the simulation of a broad range of large and complex systems
- …