71 research outputs found
MVDream: Multi-view Diffusion for 3D Generation
We propose MVDream, a multi-view diffusion model that is able to generate
geometrically consistent multi-view images from a given text prompt. By
leveraging image diffusion models pre-trained on large-scale web datasets and a
multi-view dataset rendered from 3D assets, the resulting multi-view diffusion
model can achieve both the generalizability of 2D diffusion and the consistency
of 3D data. Such a model can thus be applied as a multi-view prior for 3D
generation via Score Distillation Sampling, where it greatly improves the
stability of existing 2D-lifting methods by solving the 3D consistency problem.
Finally, we show that the multi-view diffusion model can also be fine-tuned
under a few shot setting for personalized 3D generation, i.e. DreamBooth3D
application, where the consistency can be maintained after learning the subject
identity.Comment: Our project page is https://MV-Dream.github.i
OmniAvatar: Geometry-Guided Controllable 3D Head Synthesis
We present OmniAvatar, a novel geometry-guided 3D head synthesis model
trained from in-the-wild unstructured images that is capable of synthesizing
diverse identity-preserved 3D heads with compelling dynamic details under full
disentangled control over camera poses, facial expressions, head shapes,
articulated neck and jaw poses. To achieve such high level of disentangled
control, we first explicitly define a novel semantic signed distance function
(SDF) around a head geometry (FLAME) conditioned on the control parameters.
This semantic SDF allows us to build a differentiable volumetric correspondence
map from the observation space to a disentangled canonical space from all the
control parameters. We then leverage the 3D-aware GAN framework (EG3D) to
synthesize detailed shape and appearance of 3D full heads in the canonical
space, followed by a volume rendering step guided by the volumetric
correspondence map to output into the observation space. To ensure the control
accuracy on the synthesized head shapes and expressions, we introduce a
geometry prior loss to conform to head SDF and a control loss to conform to the
expression code. Further, we enhance the temporal realism with dynamic details
conditioned upon varying expressions and joint poses. Our model can synthesize
more preferable identity-preserved 3D heads with compelling dynamic details
compared to the state-of-the-art methods both qualitatively and quantitatively.
We also provide an ablation study to justify many of our system design choices
- …