1,839 research outputs found
Structure from Articulated Motion: Accurate and Stable Monocular 3D Reconstruction without Training Data
Recovery of articulated 3D structure from 2D observations is a challenging
computer vision problem with many applications. Current learning-based
approaches achieve state-of-the-art accuracy on public benchmarks but are
restricted to specific types of objects and motions covered by the training
datasets. Model-based approaches do not rely on training data but show lower
accuracy on these datasets. In this paper, we introduce a model-based method
called Structure from Articulated Motion (SfAM), which can recover multiple
object and motion types without training on extensive data collections. At the
same time, it performs on par with learning-based state-of-the-art approaches
on public benchmarks and outperforms previous non-rigid structure from motion
(NRSfM) methods. SfAM is built upon a general-purpose NRSfM technique while
integrating a soft spatio-temporal constraint on the bone lengths. We use
alternating optimization strategy to recover optimal geometry (i.e., bone
proportions) together with 3D joint positions by enforcing the bone lengths
consistency over a series of frames. SfAM is highly robust to noisy 2D
annotations, generalizes to arbitrary objects and does not rely on training
data, which is shown in extensive experiments on public benchmarks and real
video sequences. We believe that it brings a new perspective on the domain of
monocular 3D recovery of articulated structures, including human motion
capture.Comment: 21 pages, 8 figures, 2 table
Real-time 3D reconstruction of non-rigid shapes with a single moving camera
© . This manuscript version is made available under the CC-BY-NC-ND 4.0 license http://creativecommons.org/licenses/by-nc-nd/4.0/This paper describes a real-time sequential method to simultaneously recover the camera motion and the 3D shape of deformable objects from a calibrated monocular video. For this purpose, we consider the Navier-Cauchy equations used in 3D linear elasticity and solved by finite elements, to model the time-varying shape per frame. These equations are embedded in an extended Kalman filter, resulting in sequential Bayesian estimation approach. We represent the shape, with unknown material properties, as a combination of elastic elements whose nodal points correspond to salient points in the image. The global rigidity of the shape is encoded by a stiffness matrix, computed after assembling each of these elements. With this piecewise model, we can linearly relate the 3D displacements with the 3D acting forces that cause the object deformation, assumed to be normally distributed. While standard finite-element-method techniques require imposing boundary conditions to solve the resulting linear system, in this work we eliminate this requirement by modeling the compliance matrix with a generalized pseudoinverse that enforces a pre-fixed rank. Our framework also ensures surface continuity without the need for a post-processing step to stitch all the piecewise reconstructions into a global smooth shape. We present experimental results using both synthetic and real videos for different scenarios ranging from isometric to elastic deformations. We also show the consistency of the estimation with respect to 3D ground truth data, include several experiments assessing robustness against artifacts and finally, provide an experimental validation of our performance in real time at frame rate for small mapsPeer ReviewedPostprint (author's final draft
Factored Neural Representation for Scene Understanding
A long-standing goal in scene understanding is to obtain interpretable and
editable representations that can be directly constructed from a raw monocular
RGB-D video, without requiring specialized hardware setup or priors. The
problem is significantly more challenging in the presence of multiple moving
and/or deforming objects. Traditional methods have approached the setup with a
mix of simplifications, scene priors, pretrained templates, or known
deformation models. The advent of neural representations, especially neural
implicit representations and radiance fields, opens the possibility of
end-to-end optimization to collectively capture geometry, appearance, and
object motion. However, current approaches produce global scene encoding,
assume multiview capture with limited or no motion in the scenes, and do not
facilitate easy manipulation beyond novel view synthesis. In this work, we
introduce a factored neural scene representation that can directly be learned
from a monocular RGB-D video to produce object-level neural presentations with
an explicit encoding of object movement (e.g., rigid trajectory) and/or
deformations (e.g., nonrigid movement). We evaluate ours against a set of
neural approaches on both synthetic and real data to demonstrate that the
representation is efficient, interpretable, and editable (e.g., change object
trajectory). The project webpage is available at:
- …