57 research outputs found
SceneTex: High-Quality Texture Synthesis for Indoor Scenes via Diffusion Priors
We propose SceneTex, a novel method for effectively generating high-quality
and style-consistent textures for indoor scenes using depth-to-image diffusion
priors. Unlike previous methods that either iteratively warp 2D views onto a
mesh surface or distillate diffusion latent features without accurate geometric
and style cues, SceneTex formulates the texture synthesis task as an
optimization problem in the RGB space where style and geometry consistency are
properly reflected. At its core, SceneTex proposes a multiresolution texture
field to implicitly encode the mesh appearance. We optimize the target texture
via a score-distillation-based objective function in respective RGB renderings.
To further secure the style consistency across views, we introduce a
cross-attention decoder to predict the RGB values by cross-attending to the
pre-sampled reference locations in each instance. SceneTex enables various and
accurate texture synthesis for 3D-FRONT scenes, demonstrating significant
improvements in visual quality and prompt fidelity over the prior texture
generation methods.Comment: Project website: https://daveredrum.github.io/SceneTex
LADIS: Language Disentanglement for 3D Shape Editing
Natural language interaction is a promising direction for democratizing 3D
shape design. However, existing methods for text-driven 3D shape editing face
challenges in producing decoupled, local edits to 3D shapes. We address this
problem by learning disentangled latent representations that ground language in
3D geometry. To this end, we propose a complementary tool set including a novel
network architecture, a disentanglement loss, and a new editing procedure.
Additionally, to measure edit locality, we define a new metric that we call
part-wise edit precision. We show that our method outperforms existing SOTA
methods by 20% in terms of edit locality, and up to 6.6% in terms of language
reference resolution accuracy. Our work suggests that by solely disentangling
language representations, downstream 3D shape editing can become more local to
relevant parts, even if the model was never given explicit part-based
supervision
SMIL: Multimodal Learning with Severely Missing Modality
A common assumption in multimodal learning is the completeness of training
data, i.e., full modalities are available in all training examples. Although
there exists research endeavor in developing novel methods to tackle the
incompleteness of testing data, e.g., modalities are partially missing in
testing examples, few of them can handle incomplete training modalities. The
problem becomes even more challenging if considering the case of severely
missing, e.g., 90% training examples may have incomplete modalities. For the
first time in the literature, this paper formally studies multimodal learning
with missing modality in terms of flexibility (missing modalities in training,
testing, or both) and efficiency (most training data have incomplete modality).
Technically, we propose a new method named SMIL that leverages Bayesian
meta-learning in uniformly achieving both objectives. To validate our idea, we
conduct a series of experiments on three popular benchmarks: MM-IMDb, CMU-MOSI,
and avMNIST. The results prove the state-of-the-art performance of SMIL over
existing methods and generative baselines including autoencoders and generative
adversarial networks. Our code is available at
https://github.com/mengmenm/SMIL.Comment: In AAAI 2021 (9 pages
- …