76 research outputs found
3D ShapeNets: A Deep Representation for Volumetric Shapes
3D shape is a crucial but heavily underutilized cue in today's computer
vision systems, mostly due to the lack of a good generic shape representation.
With the recent availability of inexpensive 2.5D depth sensors (e.g. Microsoft
Kinect), it is becoming increasingly important to have a powerful 3D shape
representation in the loop. Apart from category recognition, recovering full 3D
shapes from view-based 2.5D depth maps is also a critical part of visual
understanding. To this end, we propose to represent a geometric 3D shape as a
probability distribution of binary variables on a 3D voxel grid, using a
Convolutional Deep Belief Network. Our model, 3D ShapeNets, learns the
distribution of complex 3D shapes across different object categories and
arbitrary poses from raw CAD data, and discovers hierarchical compositional
part representations automatically. It naturally supports joint object
recognition and shape completion from 2.5D depth maps, and it enables active
object recognition through view planning. To train our 3D deep learning model,
we construct ModelNet -- a large-scale 3D CAD model dataset. Extensive
experiments show that our 3D deep representation enables significant
performance improvement over the-state-of-the-arts in a variety of tasks.Comment: to be appeared in CVPR 201
3D Face Synthesis Driven by Personality Impression
Synthesizing 3D faces that give certain personality impressions is commonly
needed in computer games, animations, and virtual world applications for
producing realistic virtual characters. In this paper, we propose a novel
approach to synthesize 3D faces based on personality impression for creating
virtual characters. Our approach consists of two major steps. In the first
step, we train classifiers using deep convolutional neural networks on a
dataset of images with personality impression annotations, which are capable of
predicting the personality impression of a face. In the second step, given a 3D
face and a desired personality impression type as user inputs, our approach
optimizes the facial details against the trained classifiers, so as to
synthesize a face which gives the desired personality impression. We
demonstrate our approach for synthesizing 3D faces giving desired personality
impressions on a variety of 3D face models. Perceptual studies show that the
perceived personality impressions of the synthesized faces agree with the
target personality impressions specified for synthesizing the faces. Please
refer to the supplementary materials for all results.Comment: 8pages;6 figure
Adversarially Tuned Scene Generation
Generalization performance of trained computer vision systems that use
computer graphics (CG) generated data is not yet effective due to the concept
of 'domain-shift' between virtual and real data. Although simulated data
augmented with a few real world samples has been shown to mitigate domain shift
and improve transferability of trained models, guiding or bootstrapping the
virtual data generation with the distributions learnt from target real world
domain is desired, especially in the fields where annotating even few real
images is laborious (such as semantic labeling, and intrinsic images etc.). In
order to address this problem in an unsupervised manner, our work combines
recent advances in CG (which aims to generate stochastic scene layouts coupled
with large collections of 3D object models) and generative adversarial training
(which aims train generative models by measuring discrepancy between generated
and real data in terms of their separability in the space of a deep
discriminatively-trained classifier). Our method uses iterative estimation of
the posterior density of prior distributions for a generative graphical model.
This is done within a rejection sampling framework. Initially, we assume
uniform distributions as priors on the parameters of a scene described by a
generative graphical model. As iterations proceed the prior distributions get
updated to distributions that are closer to the (unknown) distributions of
target data. We demonstrate the utility of adversarially tuned scene generation
on two real-world benchmark datasets (CityScapes and CamVid) for traffic scene
semantic labeling with a deep convolutional net (DeepLab). We realized
performance improvements by 2.28 and 3.14 points (using the IoU metric) between
the DeepLab models trained on simulated sets prepared from the scene generation
models before and after tuning to CityScapes and CamVid respectively.Comment: 9 pages, accepted at CVPR 201
SurfNet: Generating 3D shape surfaces using deep residual networks
3D shape models are naturally parameterized using vertices and faces, \ie,
composed of polygons forming a surface. However, current 3D learning paradigms
for predictive and generative tasks using convolutional neural networks focus
on a voxelized representation of the object. Lifting convolution operators from
the traditional 2D to 3D results in high computational overhead with little
additional benefit as most of the geometry information is contained on the
surface boundary. Here we study the problem of directly generating the 3D shape
surface of rigid and non-rigid shapes using deep convolutional neural networks.
We develop a procedure to create consistent `geometry images' representing the
shape surface of a category of 3D objects. We then use this consistent
representation for category-specific shape surface generation from a parametric
representation or an image by developing novel extensions of deep residual
networks for the task of geometry image generation. Our experiments indicate
that our network learns a meaningful representation of shape surfaces allowing
it to interpolate between shape orientations and poses, invent new shape
surfaces and reconstruct 3D shape surfaces from previously unseen images.Comment: CVPR 2017 pape
Interactive 3D Modeling with a Generative Adversarial Network
This paper proposes the idea of using a generative adversarial network (GAN)
to assist a novice user in designing real-world shapes with a simple interface.
The user edits a voxel grid with a painting interface (like Minecraft). Yet, at
any time, he/she can execute a SNAP command, which projects the current voxel
grid onto a latent shape manifold with a learned projection operator and then
generates a similar, but more realistic, shape using a learned generator
network. Then the user can edit the resulting shape and snap again until he/she
is satisfied with the result. The main advantage of this approach is that the
projection and generation operators assist novice users to create 3D models
characteristic of a background distribution of object shapes, but without
having to specify all the details. The core new research idea is to use a GAN
to support this application. 3D GANs have previously been used for shape
generation, interpolation, and completion, but never for interactive modeling.
The new challenge for this application is to learn a projection operator that
takes an arbitrary 3D voxel model and produces a latent vector on the shape
manifold from which a similar and realistic shape can be generated. We develop
algorithms for this and other steps of the SNAP processing pipeline and
integrate them into a simple modeling tool. Experiments with these algorithms
and tool suggest that GANs provide a promising approach to computer-assisted
interactive modeling.Comment: Published at International Conference on 3D Vision 2017
(http://irc.cs.sdu.edu.cn/3dv/index.html
- …