4,195 research outputs found
Leveraging 2D data to learn textured 3D mesh generation
Numerous methods have been proposed for probabilistic generative modelling of
3D objects. However, none of these is able to produce textured objects, which
renders them of limited use for practical tasks. In this work, we present the
first generative model of textured 3D meshes. Training such a model would
traditionally require a large dataset of textured meshes, but unfortunately,
existing datasets of meshes lack detailed textures. We instead propose a new
training methodology that allows learning from collections of 2D images without
any 3D information. To do so, we train our model to explain a distribution of
images by modelling each image as a 3D foreground object placed in front of a
2D background. Thus, it learns to generate meshes that when rendered, produce
images similar to those in its training set.
A well-known problem when generating meshes with deep networks is the
emergence of self-intersections, which are problematic for many use-cases. As a
second contribution we therefore introduce a new generation process for 3D
meshes that guarantees no self-intersections arise, based on the physical
intuition that faces should push one another out of the way as they move.
We conduct extensive experiments on our approach, reporting quantitative and
qualitative results on both synthetic data and natural images. These show our
method successfully learns to generate plausible and diverse textured 3D
samples for five challenging object classes
MeshAdv: Adversarial Meshes for Visual Recognition
Highly expressive models such as deep neural networks (DNNs) have been widely
applied to various applications. However, recent studies show that DNNs are
vulnerable to adversarial examples, which are carefully crafted inputs aiming
to mislead the predictions. Currently, the majority of these studies have
focused on perturbation added to image pixels, while such manipulation is not
physically realistic. Some works have tried to overcome this limitation by
attaching printable 2D patches or painting patterns onto surfaces, but can be
potentially defended because 3D shape features are intact. In this paper, we
propose meshAdv to generate "adversarial 3D meshes" from objects that have rich
shape features but minimal textural variation. To manipulate the shape or
texture of the objects, we make use of a differentiable renderer to compute
accurate shading on the shape and propagate the gradient. Extensive experiments
show that the generated 3D meshes are effective in attacking both classifiers
and object detectors. We evaluate the attack under different viewpoints. In
addition, we design a pipeline to perform black-box attack on a photorealistic
renderer with unknown rendering parameters.Comment: Published in IEEE CVPR201
Neural 3D Mesh Renderer
For modeling the 3D world behind 2D images, which 3D representation is most
appropriate? A polygon mesh is a promising candidate for its compactness and
geometric properties. However, it is not straightforward to model a polygon
mesh from 2D images using neural networks because the conversion from a mesh to
an image, or rendering, involves a discrete operation called rasterization,
which prevents back-propagation. Therefore, in this work, we propose an
approximate gradient for rasterization that enables the integration of
rendering into neural networks. Using this renderer, we perform single-image 3D
mesh reconstruction with silhouette image supervision and our system
outperforms the existing voxel-based approach. Additionally, we perform
gradient-based 3D mesh editing operations, such as 2D-to-3D style transfer and
3D DeepDream, with 2D supervision for the first time. These applications
demonstrate the potential of the integration of a mesh renderer into neural
networks and the effectiveness of our proposed renderer
Towards High-Fidelity 3D Face Reconstruction from In-the-Wild Images Using Graph Convolutional Networks
3D Morphable Model (3DMM) based methods have achieved great success in
recovering 3D face shapes from single-view images. However, the facial textures
recovered by such methods lack the fidelity as exhibited in the input images.
Recent work demonstrates high-quality facial texture recovering with generative
networks trained from a large-scale database of high-resolution UV maps of face
textures, which is hard to prepare and not publicly available. In this paper,
we introduce a method to reconstruct 3D facial shapes with high-fidelity
textures from single-view images in-the-wild, without the need to capture a
large-scale face texture database. The main idea is to refine the initial
texture generated by a 3DMM based method with facial details from the input
image. To this end, we propose to use graph convolutional networks to
reconstruct the detailed colors for the mesh vertices instead of reconstructing
the UV map. Experiments show that our method can generate high-quality results
and outperforms state-of-the-art methods in both qualitative and quantitative
comparisons.Comment: Accepted to CVPR 2020. The source code is available at
https://github.com/FuxiCV/3D-Face-GCN
Learning Material-Aware Local Descriptors for 3D Shapes
Material understanding is critical for design, geometric modeling, and
analysis of functional objects. We enable material-aware 3D shape analysis by
employing a projective convolutional neural network architecture to learn
material- aware descriptors from view-based representations of 3D points for
point-wise material classification or material- aware retrieval. Unfortunately,
only a small fraction of shapes in 3D repositories are labeled with physical
mate- rials, posing a challenge for learning methods. To address this
challenge, we crowdsource a dataset of 3080 3D shapes with part-wise material
labels. We focus on furniture models which exhibit interesting structure and
material variabil- ity. In addition, we also contribute a high-quality expert-
labeled benchmark of 115 shapes from Herman-Miller and IKEA for evaluation. We
further apply a mesh-aware con- ditional random field, which incorporates
rotational and reflective symmetries, to smooth our local material predic-
tions across neighboring surface patches. We demonstrate the effectiveness of
our learned descriptors for automatic texturing, material-aware retrieval, and
physical simulation. The dataset and code will be publicly available.Comment: 3DV 201
TM-NET: Deep Generative Networks for Textured Meshes
We introduce TM-NET, a novel deep generative model for synthesizing textured
meshes in a part-aware manner. Once trained, the network can generate novel
textured meshes from scratch or predict textures for a given 3D mesh, without
image guidance. Plausible and diverse textures can be generated for the same
mesh part, while texture compatibility between parts in the same shape is
achieved via conditional generation. Specifically, our method produces texture
maps for individual shape parts, each as a deformable box, leading to a natural
UV map with minimal distortion. The network separately embeds part geometry
(via a PartVAE) and part texture (via a TextureVAE) into their respective
latent spaces, so as to facilitate learning texture probability distributions
conditioned on geometry. We introduce a conditional autoregressive model for
texture generation, which can be conditioned on both part geometry and textures
already generated for other parts to achieve texture compatibility. To produce
high-frequency texture details, our TextureVAE operates in a high-dimensional
latent space via dictionary-based vector quantization. We also exploit
transparencies in the texture as an effective means to model complex shape
structures including topological details. Extensive experiments demonstrate the
plausibility, quality, and diversity of the textures and geometries generated
by our network, while avoiding inconsistency issues that are common to novel
view synthesis methods
- …