978 research outputs found
Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation
We present a novel alignment-before-generation approach to tackle the
challenging task of generating general 3D shapes based on 2D images or texts.
Directly learning a conditional generative model from images or texts to 3D
shapes is prone to producing inconsistent results with the conditions because
3D shapes have an additional dimension whose distribution significantly differs
from that of 2D images and texts. To bridge the domain gap among the three
modalities and facilitate multi-modal-conditioned 3D shape generation, we
explore representing 3D shapes in a shape-image-text-aligned space. Our
framework comprises two models: a Shape-Image-Text-Aligned Variational
Auto-Encoder (SITA-VAE) and a conditional Aligned Shape Latent Diffusion Model
(ASLDM). The former model encodes the 3D shapes into the shape latent space
aligned to the image and text and reconstructs the fine-grained 3D neural
fields corresponding to given shape embeddings via the transformer-based
decoder. The latter model learns a probabilistic mapping function from the
image or text space to the latent shape space. Our extensive experiments
demonstrate that our proposed approach can generate higher-quality and more
diverse 3D shapes that better semantically conform to the visual or textural
conditional inputs, validating the effectiveness of the
shape-image-text-aligned space for cross-modality 3D shape generation.Comment: 20 pages, 11 figure
MFM-Net: Unpaired Shape Completion Network with Multi-stage Feature Matching
Unpaired 3D object completion aims to predict a complete 3D shape from an
incomplete input without knowing the correspondence between the complete and
incomplete shapes during training. To build the correspondence between two data
modalities, previous methods usually apply adversarial training to match the
global shape features extracted by the encoder. However, this ignores the
correspondence between multi-scaled geometric information embedded in the
pyramidal hierarchy of the decoder, which makes previous methods struggle to
generate high-quality complete shapes. To address this problem, we propose a
novel unpaired shape completion network, named MFM-Net, using multi-stage
feature matching, which decomposes the learning of geometric correspondence
into multi-stages throughout the hierarchical generation process in the point
cloud decoder. Specifically, MFM-Net adopts a dual path architecture to
establish multiple feature matching channels in different layers of the
decoder, which is then combined with the adversarial learning to merge the
distribution of features from complete and incomplete modalities. In addition,
a refinement is applied to enhance the details. As a result, MFM-Net makes use
of a more comprehensive understanding to establish the geometric correspondence
between complete and incomplete shapes in a local-to-global perspective, which
enables more detailed geometric inference for generating high-quality complete
shapes. We conduct comprehensive experiments on several datasets, and the
results show that our method outperforms previous methods of unpaired point
cloud completion with a large margin
OneGAN: Simultaneous Unsupervised Learning of Conditional Image Generation, Foreground Segmentation, and Fine-Grained Clustering
We present a method for simultaneously learning, in an unsupervised manner,
(i) a conditional image generator, (ii) foreground extraction and segmentation,
(iii) clustering into a two-level class hierarchy, and (iv) object removal and
background completion, all done without any use of annotation. The method
combines a Generative Adversarial Network and a Variational Auto-Encoder, with
multiple encoders, generators and discriminators, and benefits from solving all
tasks at once. The input to the training scheme is a varied collection of
unlabeled images from the same domain, as well as a set of background images
without a foreground object. In addition, the image generator can mix the
background from one image, with a foreground that is conditioned either on that
of a second image or on the index of a desired cluster. The method obtains
state of the art results in comparison to the literature methods, when compared
to the current state of the art in each of the tasks.Comment: To be published in the European Conference on Computer Vision (ECCV)
202
- …