Sequential non-rigid structure from motion using physical priors

© 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.We propose a new approach to simultaneously recover camera pose and 3D shape of non-rigid and potentially extensible surfaces from a monocular image sequence. For this purpose, we make use of the Extended Kalman Filter based Simultaneous Localization And Mapping (EKF-SLAM) formulation, a Bayesian optimization framework traditionally used in mobile robotics for estimating camera pose and reconstructing rigid scenarios. In order to extend the problem to a deformable domain we represent the object's surface mechanics by means of Navier's equations, which are solved using a Finite Element Method (FEM). With these main ingredients, we can further model the material's stretching, allowing us to go a step further than most of current techniques, typically constrained to surfaces undergoing isometric deformations. We extensively validate our approach in both real and synthetic experiments, and demonstrate its advantages with respect to competing methods. More specifically, we show that besides simultaneously retrieving camera pose and non-rigid shape, our approach is adequate for both isometric and extensible surfaces, does not require neither batch processing all the frames nor tracking points over the whole sequence and runs at several frames per second.Peer ReviewedPostprint (author's final draft

Geometry-Aware Network for Non-Rigid Shape Prediction from a Single View

We propose a method for predicting the 3D shape of a deformable surface from a single view. By contrast with previous approaches, we do not need a pre-registered template of the surface, and our method is robust to the lack of texture and partial occlusions. At the core of our approach is a {\it geometry-aware} deep architecture that tackles the problem as usually done in analytic solutions: first perform 2D detection of the mesh and then estimate a 3D shape that is geometrically consistent with the image. We train this architecture in an end-to-end manner using a large dataset of synthetic renderings of shapes under different levels of deformation, material properties, textures and lighting conditions. We evaluate our approach on a test split of this dataset and available real benchmarks, consistently improving state-of-the-art solutions with a significantly lower computational time.Comment: Accepted at CVPR 201

Real-time 3D reconstruction of non-rigid shapes with a single moving camera

© . This manuscript version is made available under the CC-BY-NC-ND 4.0 license http://creativecommons.org/licenses/by-nc-nd/4.0/This paper describes a real-time sequential method to simultaneously recover the camera motion and the 3D shape of deformable objects from a calibrated monocular video. For this purpose, we consider the Navier-Cauchy equations used in 3D linear elasticity and solved by finite elements, to model the time-varying shape per frame. These equations are embedded in an extended Kalman filter, resulting in sequential Bayesian estimation approach. We represent the shape, with unknown material properties, as a combination of elastic elements whose nodal points correspond to salient points in the image. The global rigidity of the shape is encoded by a stiffness matrix, computed after assembling each of these elements. With this piecewise model, we can linearly relate the 3D displacements with the 3D acting forces that cause the object deformation, assumed to be normally distributed. While standard finite-element-method techniques require imposing boundary conditions to solve the resulting linear system, in this work we eliminate this requirement by modeling the compliance matrix with a generalized pseudoinverse that enforces a pre-fixed rank. Our framework also ensures surface continuity without the need for a post-processing step to stitch all the piecewise reconstructions into a global smooth shape. We present experimental results using both synthetic and real videos for different scenarios ranging from isometric to elastic deformations. We also show the consistency of the estimation with respect to 3D ground truth data, include several experiments assessing robustness against artifacts and finally, provide an experimental validation of our performance in real time at frame rate for small mapsPeer ReviewedPostprint (author's final draft

A bayesian approach to simultaneously recover camera pose and non-rigid shape from monocular images

© . This manuscript version is made available under the CC-BY-NC-ND 4.0 license http://creativecommons.org/licenses/by-nc-nd/4.0/In this paper we bring the tools of the Simultaneous Localization and Map Building (SLAM) problem from a rigid to a deformable domain and use them to simultaneously recover the 3D shape of non-rigid surfaces and the sequence of poses of a moving camera. Under the assumption that the surface shape may be represented as a weighted sum of deformation modes, we show that the problem of estimating the modal weights along with the camera poses, can be probabilistically formulated as a maximum a posteriori estimate and solved using an iterative least squares optimization. In addition, the probabilistic formulation we propose is very general and allows introducing different constraints without requiring any extra complexity. As a proof of concept, we show that local inextensibility constraints that prevent the surface from stretching can be easily integrated. An extensive evaluation on synthetic and real data, demonstrates that our method has several advantages over current non-rigid shape from motion approaches. In particular, we show that our solution is robust to large amounts of noise and outliers and that it does not need to track points over the whole sequence nor to use an initialization close from the ground truth.Peer ReviewedPostprint (author's final draft

Multi-Level Shape Representation Using Global Deformations and Locally Adaptive Finite Elements

We present a model-based method for the multi-level shape, pose estimation and abstraction of an object’s surface from range data. The surface shape is estimated based on the parameters of a superquadric that is subjected to global deformations (tapering and bending) and a varying number of levels of local deformations. Local deformations are implemented using locally adaptive finite elements whose shape functions are piecewise cubic functions with C1 continuity. The surface pose is estimated based on the model\u27s translational and rotational degrees of freedom. The algorithm first does a coarse fit, solving for a first approximation to the translation, rotation and global deformation parameters and then does several passes of mesh refinement, by locally subdividing triangles based on the distance between the given datapoints and the model. The adaptive finite element algorithm ensures that during subdivision the desirable finite element mesh generation properties of conformity, non-degeneracy and smoothness are maintained. Each pass of the algorithm uses physics-based modeling techniques to iteratively adjust the global and local parameters of the model in response to forces that are computed from approximation errors between the model and the data. We present results demonstrating the multi-level shape representation for both sparse and dense range data

Dynamic Multivariate Simplex Splines For Volume Representation And Modeling

Volume representation and modeling of heterogeneous objects acquired from real world are very challenging research tasks and playing fundamental roles in many potential applications, e.g., volume reconstruction, volume simulation and volume registration. In order to accurately and efficiently represent and model the real-world objects, this dissertation proposes an integrated computational framework based on dynamic multivariate simplex splines (DMSS) that can greatly improve the accuracy and efficacy of modeling and simulation of heterogenous objects. The framework can not only reconstruct with high accuracy geometric, material, and other quantities associated with heterogeneous real-world models, but also simulate the complicated dynamics precisely by tightly coupling these physical properties into simulation. The integration of geometric modeling and material modeling is the key to the success of representation and modeling of real-world objects. The proposed framework has been successfully applied to multiple research areas, such as volume reconstruction and visualization, nonrigid volume registration, and physically based modeling and simulation

Finite element surface registration incorporating curvature, volume preservation, and statistical model information

We present a novel method for nonrigid registration of 3D surfaces and images. The method can be used to register surfaces by means of their distance images, or to register medical images directly. It is formulated as a minimization problem of a sum of several terms representing the desired properties of a registration result: smoothness, volume preservation, matching of the surface, its curvature, and possible other feature images, as well as consistency with previous registration results of similar objects, represented by a statistical deformation model. While most of these concepts are already known, we present a coherent continuous formulation of these constraints, including the statistical deformation model. This continuous formulation renders the registration method independent of its discretization. The finite element discretization we present is, while independent of the registration functional, the second main contribution of this paper. The local discontinuous Galerkin method has not previously been used in image registration, and it provides an efficient and general framework to discretize each of the terms of our functional. Computational efficiency and modest memory consumption are achieved thanks to parallelization and locally adaptive mesh refinement. This allows for the first time the use of otherwise prohibitively large 3D statistical deformation models

A spring force formulation for elastically deformable models

Cataloged from PDF version of article.Continuous deformable models are generally represented using a grid of control points. The elastic properties are then modeled using the interactions between these points. The formulations based on elasticity theory express these interactions using stiffness matrices. These matrices store the elastic properties of the models and they should be evolved in time according to changing elastic properties of the models. However, forming the stiffness matrices at any step of an animation is very difficult and sometimes the differential equations that should be solved to produce animation become ill-conditioned. Instead of modeling the elasticities using stiffness matrices, the interactions between model points could be expressed in terms of external spring forces. In this paper, a spring force formulation for animating elastically deformable models is presented. In this formulation, elastic properties of the materials are represented as external spring forces as opposed to forming complicated stiffness matrices. (C) 1997 Elsevier Science Ltd

Análise de Movimento Não Rígido em Visão por Computador

Neste artigo são apresentadas várias metodologias actualmente existentes, no domínio da Visão por Computador, para a análise de movimento não rígido e são indicados diversos exemplos de aplicações. Assim o movimento não rígido é classificado e, para cada classe resultante, são indicadas as restrições e as condições inerentes e verificados alguns trabalhos realizados no seu âmbito. Como as questões de análise de movimento e modelização da forma se tornam inseparáveis quando se considera o movimento do tipo não rígido, a modelização sugere uma classificação possível da forma não rígida e do movimento. Assim são também apresentados modelos de forma para objectos deformáveis e indicados vários exemplos de aplicações. Com este estudo, de certo modo aprofundado, das várias metodologias, e suas aplicações, existentes no domínio da análise de movimento não rígido, espera-se contribuir para o seu desenvolvimento, dada a actual carência de boas revisões do estado da arte neste domínio.In this article several methodologies actually existent, in the Computer Vision domain, for non-rigid movement analysis are presented and several examples of applications are indicated. Thus the non-rigid movement is classified and, for each resulting class, the restrictions and the inherent conditions are presented and some works accomplished in its ambit are verified. As the questions of movement and shape analysis becomes non-separable when its considered the movement of the non-rigid type, the shape models also suggests a possible classification of the non-rigid shape and of the movement. Thus shape models for deformable objects will be presented and some examples of applications indicated. With this study, in certain way deep, of several methodologies, and its applications, existent in the domain of the non-rigid movement analysis, the authors hope to contribute for its development, given the actual lack of good state of the art revisions in this domain

Probabilistic simultaneous pose and non-rigid shape recovery

We present an algorithm to simultaneously recover non-rigid shape and camera poses from point correspondences between a reference shape and a sequence of input images. The key novel contribution of our approach is in bringing the tools of the probabilistic SLAM methodology from a rigid to a deformable domain. Under the assumption that the shape may be represented as a weighted sum of deformation modes, we show that the problem of estimating the modal weights along with the camera poses, may be probabilistically formulated as a maximum a posterior estimate and solved using an iterative least squares optimization. An extensive evaluation on synthetic and real data, shows that our approach has several significant advantages over current approaches, such as performing robustly under large amounts of noise and outliers, and neither requiring to track points over the whole sequence nor initializations close from the ground truth solution.Postprint (author’s final draft