Tracking an elastic object with an RGB-D sensor for a pizza chef robot

This paper presents a method to track in real-time a 3D object which undergoes large deformations such as elastic ones, and fast rigid motions, using the point cloud data provided by a RGB-D sensor. This solution would contribute to robotic humanoid manipulation purposes. Our framework relies on a prior visual segmentation of the object in the image. The segmented point cloud is then registered first in a rigid manner and then by non-rigidly fitting the mesh, based on the Finite Element Method to model elasticity and on geometrical point-to-point correspondences to compute external forces exerted on the mesh. The real-time performance of the system is demonstrated on real data involving challenging deformations and motions, for a pizza dough to be ideally manipulated by a chef robot

Real-time tracking of 3D elastic objects with an RGB-D sensor

This paper presents a method to track in real-time a 3D textureless object which undergoes large deformations such as elastic ones, and rigid motions, using the point cloud data provided by an RGB-D sensor. This solution is expected to be useful for enhanced manipulation of humanoid robotic systems. Our framework relies on a prior visual segmentation of the object in the image. The segmented point cloud is registered first in a rigid manner and then by non-rigidly fitting the mesh, based on the Finite Element Method to model elasticity, and on geometrical point-to-point correspondences to compute external forces exerted on the mesh. The real-time performance of the system is demonstrated on synthetic and real data involving challenging deformations and motions

Parameterizing Deformable Surfaces for Monocular 3--D Tracking

e propose a deformable surface parameterization that is generic and lets us automatically build registered shape databases. This allows us to directly derive low-dimensional shape models using a simple dimensionality reduction technique. This addresses one of the biggest difficulties in example-based shape modeling: Building the required database, which is often a difficult and painstaking process. We incorporate the resulting models into a monocular tracking system that we use to capture the complex deformations of objects such as sheets of papers or expanding balloons from single video sequences

Closed-Form Solution to Non-rigid 3D Surface Registration

We present a closed-form solution to the problem of recovering the 3D shape of a non-rigid inelastic surface from 3D-to-2D correspondences. This lets us detect and reconstruct such a, surface by matching individual images against a reference configuration, which is in contrast to all existing approaches that require initial shape estimates and track deformations from image to image

Capturing Deformations of Interacting Non-rigid Objects Using RGB-D Data

International audienceThis paper presents a method for tracking multiple interacting deformable objects undergoing rigid motions, elastic deformations and contacts, using image and point cloud data provided by an RGB-D sensor. A joint registration framework is proposed, based on physical Finite Element Method (FEM) elastic and interaction models. It first relies on a visual segmentation of the considered objects in the RGB images. The different segmented point clouds are then processed to estimate rigid transformations with on an ICP algorithm, and to determine geometrical point-to-point correspondences with the meshes. External forces resulting from these correspondences and between the current and the rigidly transformed mesh can then be derived. It provides both non-rigid and rigid data cues. A classical collision detection and response model is also integrated, giving contact forces between the objects. The deformations of the objects are estimated by solving a dynamic system balancing these external and contact forces with the internal or regularization forces computed through the FEM elastic model. This approach has been here tested on different scenarios involving two or three interacting deformable objects of various shapes, with promising results

Surface Deformation Models for Non-Rigid 3--D Shape Recovery

3--D detection and shape recovery of a non-rigid surface from video sequences require deformation models to effectively take advantage of potentially noisy image data. Here we introduce an approach to creating such models for deformable 3--D surfaces. We exploit the fact that the shape of an inextensible triangulated mesh can be parameterized in terms of a small subset of the angles between its facets. We use this set of angles to create a representative set of potential shapes, which we feed to a simple dimensionality reduction technique to produce low-dimensional 3--D deformation models. We show that these models can be used to accurately model a wide range of deforming 3--D surfaces from video sequences acquired under realistic conditions

Learning and recovering 3D surface deformations

Recovering the 3D deformations of a non-rigid surface from a single viewpoint has applications in many domains such as sports, entertainment, and medical imaging. Unfortunately, without any knowledge of the possible deformations that the object of interest can undergo, it is severely under-constrained, and extremely different shapes can have very similar appearances when reprojected onto an image plane. In this thesis, we first exhibit the ambiguities of the reconstruction problem when relying on correspondences between a reference image for which we know the shape and an input image. We then propose several approaches to overcoming these ambiguities. The core idea is that some a priori knowledge about how a surface can deform must be introduced to solve them. We therefore present different ways to formulate that knowledge that range from very generic constraints to models specifically designed for a particular object or material. First, we propose generally applicable constraints formulated as motion models. Such models simply link the deformations of the surface from one image to the next in a video sequence. The obvious advantage is that they can be used independently of the physical properties of the object of interest. However, to be effective, they require the presence of texture over the whole surface, and, additionally, do not prevent error accumulation from frame to frame. To overcome these weaknesses, we propose to introduce statistical learning techniques that let us build a model from a large set of training examples, that is, in our case, known 3D deformations. The resulting model then essentially performs linear or non-linear interpolation between the training examples. Following this approach, we first propose a linear global representation that models the behavior of the whole surface. As is the case with all statistical learning techniques, the applicability of this representation is limited by the fact that acquiring training data is far from trivial. A large surface can undergo many subtle deformations, and thus a large amount of training data must be available to build an accurate model. We therefore propose an automatic way of generating such training examples in the case of inextensible surfaces. Furthermore, we show that the resulting linear global models can be incorporated into a closed-form solution to the shape recovery problem. This lets us not only track deformations from frame to frame, but also reconstruct surfaces from individual images. The major drawback of global representations is that they can only model the behavior of a specific surface, which forces us to re-train a new model for every new shape, even though it is made of a material observed before. To overcome this issue, and simultaneously reduce the amount of required training data, we propose local deformation models. Such models describe the behavior of small portions of a surface, and can be combined to form arbitrary global shapes. For this purpose, we study both linear and non-linear statistical learning methods, and show that, whereas the latter are better suited for traking deformations from frame to frame, the former can also be used for reconstruction from a single image