111,109 research outputs found
Generating 3D faces using Convolutional Mesh Autoencoders
Learned 3D representations of human faces are useful for computer vision
problems such as 3D face tracking and reconstruction from images, as well as
graphics applications such as character generation and animation. Traditional
models learn a latent representation of a face using linear subspaces or
higher-order tensor generalizations. Due to this linearity, they can not
capture extreme deformations and non-linear expressions. To address this, we
introduce a versatile model that learns a non-linear representation of a face
using spectral convolutions on a mesh surface. We introduce mesh sampling
operations that enable a hierarchical mesh representation that captures
non-linear variations in shape and expression at multiple scales within the
model. In a variational setting, our model samples diverse realistic 3D faces
from a multivariate Gaussian distribution. Our training data consists of 20,466
meshes of extreme expressions captured over 12 different subjects. Despite
limited training data, our trained model outperforms state-of-the-art face
models with 50% lower reconstruction error, while using 75% fewer parameters.
We also show that, replacing the expression space of an existing
state-of-the-art face model with our autoencoder, achieves a lower
reconstruction error. Our data, model and code are available at
http://github.com/anuragranj/com
Single View Reconstruction for Human Face and Motion with Priors
Single view reconstruction is fundamentally an under-constrained problem. We aim to develop new approaches to model human face and motion with model priors that restrict the space of possible solutions. First, we develop a novel approach to recover the 3D shape from a single view image under challenging conditions, such as large variations in illumination and pose. The problem is addressed by employing the techniques of non-linear manifold embedding and alignment. Specifically, the local image models for each patch of facial images and the local surface models for each patch of 3D shape are learned using a non-linear dimensionality reduction technique, and the correspondences between these local models are then learned by a manifold alignment method. Local models successfully remove the dependency of large training databases for human face modeling. By combining the local shapes, the global shape of a face can be reconstructed directly from a single linear system of equations via least square.
Unfortunately, this learning-based approach cannot be successfully applied to the problem of human motion modeling due to the internal and external variations in single view video-based marker-less motion capture. Therefore, we introduce a new model-based approach for capturing human motion using a stream of depth images from a single depth sensor. While a depth sensor provides metric 3D information, using a single sensor, instead of a camera array, results in a view-dependent and incomplete measurement of object motion. We develop a novel two-stage template fitting algorithm that is invariant to subject size and view-point variations, and robust to occlusions. Starting from a known pose, our algorithm first estimates a body configuration through temporal registration, which is used to search the template motion database for a best match. The best match body configuration as well as its corresponding surface mesh model are deformed to fit the input depth map, filling in the part that is occluded from the input and compensating for differences in pose and body-size between the input image and the template. Our approach does not require any makers, user-interaction, or appearance-based tracking.
Experiments show that our approaches can achieve good modeling results for human face and motion, and are capable of dealing with variety of challenges in single view reconstruction, e.g., occlusion
Learning quadrangulated patches for 3D shape parameterization and completion
We propose a novel 3D shape parameterization by surface patches, that are
oriented by 3D mesh quadrangulation of the shape. By encoding 3D surface detail
on local patches, we learn a patch dictionary that identifies principal surface
features of the shape. Unlike previous methods, we are able to encode surface
patches of variable size as determined by the user. We propose novel methods
for dictionary learning and patch reconstruction based on the query of a noisy
input patch with holes. We evaluate the patch dictionary towards various
applications in 3D shape inpainting, denoising and compression. Our method is
able to predict missing vertices and inpaint moderately sized holes. We
demonstrate a complete pipeline for reconstructing the 3D mesh from the patch
encoding. We validate our shape parameterization and reconstruction methods on
both synthetic shapes and real world scans. We show that our patch dictionary
performs successful shape completion of complicated surface textures.Comment: To be presented at International Conference on 3D Vision 2017, 201
Latent Partition Implicit with Surface Codes for 3D Representation
Deep implicit functions have shown remarkable shape modeling ability in
various 3D computer vision tasks. One drawback is that it is hard for them to
represent a 3D shape as multiple parts. Current solutions learn various
primitives and blend the primitives directly in the spatial space, which still
struggle to approximate the 3D shape accurately. To resolve this problem, we
introduce a novel implicit representation to represent a single 3D shape as a
set of parts in the latent space, towards both highly accurate and plausibly
interpretable shape modeling. Our insight here is that both the part learning
and the part blending can be conducted much easier in the latent space than in
the spatial space. We name our method Latent Partition Implicit (LPI), because
of its ability of casting the global shape modeling into multiple local part
modeling, which partitions the global shape unity. LPI represents a shape as
Signed Distance Functions (SDFs) using surface codes. Each surface code is a
latent code representing a part whose center is on the surface, which enables
us to flexibly employ intrinsic attributes of shapes or additional surface
properties. Eventually, LPI can reconstruct both the shape and the parts on the
shape, both of which are plausible meshes. LPI is a multi-level representation,
which can partition a shape into different numbers of parts after training. LPI
can be learned without ground truth signed distances, point normals or any
supervision for part partition. LPI outperforms the latest methods under the
widely used benchmarks in terms of reconstruction accuracy and modeling
interpretability. Our code, data and models are available at
https://github.com/chenchao15/LPI.Comment: 20pages,14figures. Accepted by ECCV 202
Learning Shape Priors for Single-View 3D Completion and Reconstruction
The problem of single-view 3D shape completion or reconstruction is
challenging, because among the many possible shapes that explain an
observation, most are implausible and do not correspond to natural objects.
Recent research in the field has tackled this problem by exploiting the
expressiveness of deep convolutional networks. In fact, there is another level
of ambiguity that is often overlooked: among plausible shapes, there are still
multiple shapes that fit the 2D image equally well; i.e., the ground truth
shape is non-deterministic given a single-view input. Existing fully supervised
approaches fail to address this issue, and often produce blurry mean shapes
with smooth surfaces but no fine details.
In this paper, we propose ShapeHD, pushing the limit of single-view shape
completion and reconstruction by integrating deep generative models with
adversarially learned shape priors. The learned priors serve as a regularizer,
penalizing the model only if its output is unrealistic, not if it deviates from
the ground truth. Our design thus overcomes both levels of ambiguity
aforementioned. Experiments demonstrate that ShapeHD outperforms state of the
art by a large margin in both shape completion and shape reconstruction on
multiple real datasets.Comment: ECCV 2018. The first two authors contributed equally to this work.
Project page: http://shapehd.csail.mit.edu
Multilinear Wavelets: A Statistical Shape Space for Human Faces
We present a statistical model for D human faces in varying expression,
which decomposes the surface of the face using a wavelet transform, and learns
many localized, decorrelated multilinear models on the resulting coefficients.
Using this model we are able to reconstruct faces from noisy and occluded D
face scans, and facial motion sequences. Accurate reconstruction of face shape
is important for applications such as tele-presence and gaming. The localized
and multi-scale nature of our model allows for recovery of fine-scale detail
while retaining robustness to severe noise and occlusion, and is
computationally efficient and scalable. We validate these properties
experimentally on challenging data in the form of static scans and motion
sequences. We show that in comparison to a global multilinear model, our model
better preserves fine detail and is computationally faster, while in comparison
to a localized PCA model, our model better handles variation in expression, is
faster, and allows us to fix identity parameters for a given subject.Comment: 10 pages, 7 figures; accepted to ECCV 201
Linear Local Models for Monocular Reconstruction of Deformable Surfaces
Recovering the 3D shape of a nonrigid surface from a single viewpoint is known to be both ambiguous and challenging. Resolving the ambiguities typically requires prior knowledge about the most likely deformations that the surface may undergo. It often takes the form of a global deformation model that can be learned from training data. While effective, this approach suffers from the fact that a new model must be learned for each new surface, which means acquiring new training data and may be impractical. In this paper, we replace the global models by linear local ones for surface patches, which can be assembled to represent arbitrary surface shapes as long as they are made of the same material. Not only do they eliminate the need to retrain the model for different surface shapes, they also let us formulate 3D shape reconstruction from correspondences as either an algebraic problem that can be solved in closed-form or a convex optimization problem whose solution can be found using standard numerical packages. We present quantitative results on synthetic data, as well as qualitative ones on real images
- …