The Alignment Between 3-D Data and Articulated Shapes with Bending Surfaces

International audienceIn this paper we address the problem of aligning 3-D data with articulated shapes. This problem resides at the core of many motion tracking methods with applications in human motion capture, action recognition, medical-image analysis, etc. We describe an articulated and bending surface representation well suited for this task as well as a method which aligns (or registers) such a surface to 3-D data. Articulated objects, e.g., humans and animals, are covered with clothes and skin which may be seen as textured surfaces. These surfaces are both articulated and deformable and one realistic way to model them is to assume that they bend in the neighborhood of the shape's joints. We will introduce a surface-bending model as a function of the articulated-motion parameters. This combined articulated-motion and surface-bending model better predicts the observed phenomena in the data and therefore is well suited for surface registration. Given a set of sparse 3-D data (gathered with a stereo camera pair) and a textured, articulated, and bending surface, we describe a register-and-fit method that proceeds as follows. First, the data-to-surface registration problem is formalized as a classifier and is carried out using an EM algorithm. Second, the data-to-surface fitting problem is carried out by minimizing the distance from the registered data points to the surface over the joint variables. In order to illustrate the method we applied it to the problem of hand tracking. A hand model with 27 degrees of freedom is successfully registered and fitted to a sequence of 3-D data points gathered with a stereo camera pair

Virtual Clay for Direct Hand Manipulation

International audienceIn order to make virtual modeling as easy as real clay manipulation, we describe a realtime virtual clay model, specially designed for direct hand manipulation. We build on a previous layered model for clay, extending it to handle local properties such as colour or fluidity, to deal with an arbitrary number of tools, and to capture twist effects due to rotating tools. The resulting clay model is the first step towards a more long term goal, namely direct interaction through video tracking of the user's hands

A Statistical-Topological Feature Combination for Recognition of Isolated Hand Gestures from Kinect Based Depth Images

International audienceReliable hand gesture recognition is an important problem for automatic sign language recognition for the people with hearing and speech disabilities. In this paper, we create a new benchmark database of multi-oriented, isolated ASL numeric images using recently launched Kinect V2. Further, we design an effective statistical-topological feature combinations for recognition of the hand gestures using the available V1 sensor dataset and also over the new V2 dataset. For V1, our best accuracy is 98.4% which is comparable with the best one reported so far and for V2 we achieve an accuracy of 92.2% which is first of its kind

Articulated Motion Capture from 3-D Points and Normals

International audienceIn this paper we address the problem of tracking the motion of articulated objects from their 2-D silhouettes gathered with several cameras. The vast majority of existing approaches relies on a single camera or on stereo. We describe a new method which requires at least two cameras. The method relies on (i) building 3-D observations (points and normals) from image silhouettes and on (ii) fitting an articulated object model to these observations by minimizing their discrepancies. The objective function sums up these discrepancies while it takes into account both the scaled algebraic distance from data points to the model surface and the offset in orientation between observed normals and model normals. The combination of a feed-forward reconstruction technique with a robust model-tracking method results in a reliable and efficient method for articulated motion capture

Gesture imitation and recognition using Kinect sensor and extreme learning machines

This study presents a framework that recognizes and imitates human upper-body motions in real time. The framework consists of two parts. In the first part, a transformation algorithm is applied to 3D human motion data captured by a Kinect. The data are then converted into the robot’s joint angles by the algorithm. The human upper-body motions are successfully imitated by the NAO humanoid robot in real time. In the second part, the human action recognition algorithm is implemented for upper-body gestures. A human action dataset is also created for the upper-body movements. Each action is performed 10 times by twenty-four users. The collected joint angles are divided into six action classes. Extreme Learning Machines (ELMs) are used to classify the human actions. Additionally, the Feed-Forward Neural Networks (FNNs) and K-Nearest Neighbor (K-NN) classifiers are used for comparison. According to the comparative results, ELMs produce a good human action recognition performance

Designing and evolving hands-on interaction prototypes for virtual reality

Best Paper AwardInternational audienceWe analyse and compare several prototypes of the HandNavigator, a peripheral device allowing a user to interact with a virtual environment by controlling a virtual hand with fine dexterity. Our prototypes, as easy to manipulate as a computer mouse, integrate a large panel of small sensors enabling the simultaneous control of a large number of degrees of freedom. Based on this architecture, we address the problems of designing a device where physical phenomena, physiological behavior and device structure are all tightly combined and significantly influence the overall interaction. The issues addressed include the generation of degrees of freedom and decoupling of virtual hand and finger movement, the influence of device shape and sensor type on the decoupling necessary for various tasks, dexterity, and performance. Using these different prototypes, we are able to perform complex tasks, such as virtual sculpture and manipulation of deformable objects in a natural way. The choice of the sensors and of their placement on the device show their influence on the dexterity of the virtual hand and on the range of configurations that can be achieved and addressed by a prototype

Design of an immersive peripheral for object grasping

Technical Committee Best Paper AwardInternational audienceDuring product development processes, simulations involving user's grasping operations are of increasing interest to incorporate more quantitative information in DFA (Design For Assembly) or immersive simulations. We present several prototypes of an immersive peripheral device for controlling a virtual hand with fine dexterity. These prototypes are derived from the analysis of a grasping action to define the structure and main features of this device. The prototypes, as easy to manipulate as a computer mouse, enable the simultaneous control of a large number of degrees of freedom (dofs). The design issues, where physical phenomena, physiological behavior and device structure are all tightly combined and significantly influence the overall interaction, are reviewed. These issues include the generation of dofs, monitoring kinematics, force reduction during virtual hand and finger movements, and the influence of device design, sensor types and their placement on the interaction and on the range of configurations that can be achieved for grasping tasks, dexterity, and performance. Examples of grasping tasks show the effect of these immersive devices to reach user-friendly and efficient interactions with objects bringing new insight to the interaction with virtual products

Fast Non-Rigid Surface Detection, Registration and Realistic Augmentation

We present a real-time method for detecting deformable surfaces, with no need whatsoever for a priori pose knowledge. Our method starts from a set of wide baseline point matches between an undeformed image of the object and the image in which it is to be detected. The matches are used not only to detect but also to compute a precise mapping from one to the other. The algorithm is robust to large deformations, lighting changes, motion blur, and occlusions. It runs at 10 frames per second on a 2.8 GHz PC.We demonstrate its applicability by using it to realistically modify the texture of a deforming surface and to handle complex illumination effects. Combining deformable meshes with a well designed robust estimator is key to dealing with the large number of parameters involved in modeling deformable surfaces and rejecting erroneous matches for error rates of more than 90%, which is considerably more than what is required in practice

Bridging the Gap between Detection and Tracking for 3D Human Motion Recovery

The aim of this thesis is to build a system able to automatically and robustly track human motion in 3–D starting from monocular input. To this end two approaches are introduced, which tackle two different types of motion: The first is useful to analyze activities for which a characteristic pose, or key-pose, can be detected, as for example in the walking case. On the other hand the second can be used for cases in which such pose is not defined but there is a clear relation between some easily measurable image quantities and the body configuration, as for example in the skating case where the trajectory followed by a subject is highly correlated to how the subject articulates. In the first proposed technique we combine detection and tracking techniques to achieve robust 3D motion recovery of people seen from arbitrary viewpoints by a single and potentially moving camera. We rely on detecting key postures, which can be done reliably, using a motion model to infer 3D poses between consecutive detections, and finally refining them over the whole sequence using a generative model. We demonstrate our approach in the cases of golf motions filmed using a static camera and walking motions acquired using a potentially moving one. We will show that this approach, although monocular, is both metrically accurate because it integrates information over many frames and robust because it can recover from a few misdetections. The second approach is based on the fact that the articulated body models used to represent human motion typically have many degrees of freedom, usually expressed as joint angles that are highly correlated. The true range of motion can therefore be represented by latent variables that span a low-dimensional space. This has often been used to make motion tracking easier. However, learning the latent space in a problem independent way makes it non trivial to initialize the tracking process by picking appropriate initial values for the latent variables, and thus for the pose. In this thesis, it will be shown that by directly using observable quantities as latent variables, this issue can be eliminated