47,902 research outputs found
MonoPerfCap: Human Performance Capture from Monocular Video
We present the first marker-less approach for temporally coherent 3D
performance capture of a human with general clothing from monocular video. Our
approach reconstructs articulated human skeleton motion as well as medium-scale
non-rigid surface deformations in general scenes. Human performance capture is
a challenging problem due to the large range of articulation, potentially fast
motion, and considerable non-rigid deformations, even from multi-view data.
Reconstruction from monocular video alone is drastically more challenging,
since strong occlusions and the inherent depth ambiguity lead to a highly
ill-posed reconstruction problem. We tackle these challenges by a novel
approach that employs sparse 2D and 3D human pose detections from a
convolutional neural network using a batch-based pose estimation strategy.
Joint recovery of per-batch motion allows to resolve the ambiguities of the
monocular reconstruction problem based on a low dimensional trajectory
subspace. In addition, we propose refinement of the surface geometry based on
fully automatically extracted silhouettes to enable medium-scale non-rigid
alignment. We demonstrate state-of-the-art performance capture results that
enable exciting applications such as video editing and free viewpoint video,
previously infeasible from monocular video. Our qualitative and quantitative
evaluation demonstrates that our approach significantly outperforms previous
monocular methods in terms of accuracy, robustness and scene complexity that
can be handled.Comment: Accepted to ACM TOG 2018, to be presented on SIGGRAPH 201
Structure from Articulated Motion: Accurate and Stable Monocular 3D Reconstruction without Training Data
Recovery of articulated 3D structure from 2D observations is a challenging
computer vision problem with many applications. Current learning-based
approaches achieve state-of-the-art accuracy on public benchmarks but are
restricted to specific types of objects and motions covered by the training
datasets. Model-based approaches do not rely on training data but show lower
accuracy on these datasets. In this paper, we introduce a model-based method
called Structure from Articulated Motion (SfAM), which can recover multiple
object and motion types without training on extensive data collections. At the
same time, it performs on par with learning-based state-of-the-art approaches
on public benchmarks and outperforms previous non-rigid structure from motion
(NRSfM) methods. SfAM is built upon a general-purpose NRSfM technique while
integrating a soft spatio-temporal constraint on the bone lengths. We use
alternating optimization strategy to recover optimal geometry (i.e., bone
proportions) together with 3D joint positions by enforcing the bone lengths
consistency over a series of frames. SfAM is highly robust to noisy 2D
annotations, generalizes to arbitrary objects and does not rely on training
data, which is shown in extensive experiments on public benchmarks and real
video sequences. We believe that it brings a new perspective on the domain of
monocular 3D recovery of articulated structures, including human motion
capture.Comment: 21 pages, 8 figures, 2 table
A closed-form solution to estimate uncertainty in non-rigid structure from motion
Semi-Definite Programming (SDP) with low-rank prior has been widely applied
in Non-Rigid Structure from Motion (NRSfM). Based on a low-rank constraint, it
avoids the inherent ambiguity of basis number selection in conventional
base-shape or base-trajectory methods. Despite the efficiency in deformable
shape reconstruction, it remains unclear how to assess the uncertainty of the
recovered shape from the SDP process. In this paper, we present a statistical
inference on the element-wise uncertainty quantification of the estimated
deforming 3D shape points in the case of the exact low-rank SDP problem. A
closed-form uncertainty quantification method is proposed and tested. Moreover,
we extend the exact low-rank uncertainty quantification to the approximate
low-rank scenario with a numerical optimal rank selection method, which enables
solving practical application in SDP based NRSfM scenario. The proposed method
provides an independent module to the SDP method and only requires the
statistic information of the input 2D tracked points. Extensive experiments
prove that the output 3D points have identical normal distribution to the 2D
trackings, the proposed method and quantify the uncertainty accurately, and
supports that it has desirable effects on routinely SDP low-rank based NRSfM
solver.Comment: 9 pages, 2 figure
End-to-end Recovery of Human Shape and Pose
We describe Human Mesh Recovery (HMR), an end-to-end framework for
reconstructing a full 3D mesh of a human body from a single RGB image. In
contrast to most current methods that compute 2D or 3D joint locations, we
produce a richer and more useful mesh representation that is parameterized by
shape and 3D joint angles. The main objective is to minimize the reprojection
loss of keypoints, which allow our model to be trained using images in-the-wild
that only have ground truth 2D annotations. However, the reprojection loss
alone leaves the model highly under constrained. In this work we address this
problem by introducing an adversary trained to tell whether a human body
parameter is real or not using a large database of 3D human meshes. We show
that HMR can be trained with and without using any paired 2D-to-3D supervision.
We do not rely on intermediate 2D keypoint detections and infer 3D pose and
shape parameters directly from image pixels. Our model runs in real-time given
a bounding box containing the person. We demonstrate our approach on various
images in-the-wild and out-perform previous optimization based methods that
output 3D meshes and show competitive results on tasks such as 3D joint
location estimation and part segmentation.Comment: CVPR 2018, Project page with code: https://akanazawa.github.io/hmr
On the Two-View Geometry of Unsynchronized Cameras
We present new methods for simultaneously estimating camera geometry and time
shift from video sequences from multiple unsynchronized cameras. Algorithms for
simultaneous computation of a fundamental matrix or a homography with unknown
time shift between images are developed. Our methods use minimal correspondence
sets (eight for fundamental matrix and four and a half for homography) and
therefore are suitable for robust estimation using RANSAC. Furthermore, we
present an iterative algorithm that extends the applicability on sequences
which are significantly unsynchronized, finding the correct time shift up to
several seconds. We evaluated the methods on synthetic and wide range of real
world datasets and the results show a broad applicability to the problem of
camera synchronization.Comment: 12 pages, 9 figures, Computer Vision and Pattern Recognition (CVPR)
201
Depth Superresolution using Motion Adaptive Regularization
Spatial resolution of depth sensors is often significantly lower compared to
that of conventional optical cameras. Recent work has explored the idea of
improving the resolution of depth using higher resolution intensity as a side
information. In this paper, we demonstrate that further incorporating temporal
information in videos can significantly improve the results. In particular, we
propose a novel approach that improves depth resolution, exploiting the
space-time redundancy in the depth and intensity using motion-adaptive low-rank
regularization. Experiments confirm that the proposed approach substantially
improves the quality of the estimated high-resolution depth. Our approach can
be a first component in systems using vision techniques that rely on high
resolution depth information
- …