Gesture recognition using a depth camera for human robot collaboration on assembly line

International audienceWe present a framework and preliminary experimental results for technical gestures recognition using a RGB-D camera. We have studied a collaborative task between a robot and an operator: the assembly of a motor hoses. The goal is to enable the robot to understand which task has just been executed by a human operator in order to anticipate on his actions, to adapt his speed and react properly if an unusual event occurs. The depth camera is placed above the operator, to minimize the possible occlusion on an assembly line, and we track the head and the hands of the operator using the geodesic distance between the head and the pixels of his torso. To describe his movements we used the shape of the shortest routes joining the head and the hands. We then used a discreet HMM to learn and recognize five gestures performed during the motor hoses assembly. By using gesture from the same operator for the learning and the recognition, we reach a good recognition rate of 93%. These results are encouraging and ongoing work will lead us to experiment our set up on a larger pool of operators and recognize the gesture in real time

Learning Generative Models for Multi-Activity Body Pose Estimation

We present a method to simultaneously estimate 3D body pose and action categories from monocular video sequences. Our approach learns a generative model of the relationship of body pose and image appearance using a sparse kernel regressor. Body poses are modelled on a low-dimensional manifold obtained by Locally Linear Embedding dimensionality reduction. In addition, we learn a prior model of likely body poses and a dynamical model in this pose manifold. Sparse kernel regressors capture the nonlinearities of this mapping efficiently. Within a Recursive Bayesian Sampling framework, the potentially multimodal posterior probability distributions can then be inferred. An activity-switching mechanism based on learned transfer functions allows for inference of the performed activity class, along with the estimation of body pose and 2D image location of the subject. Using a rough foreground segmentation, we compare Binary PCA and distance transforms to encode the appearance. As a postprocessing step, the globally optimal trajectory through the entire sequence is estimated, yielding a single pose estimate per frame that is consistent throughout the sequence. We evaluate the algorithm on challenging sequences with subjects that are alternating between running and walking movements. Our experiments show how the dynamical model helps to track through poorly segmented low-resolution image sequences where tracking otherwise fails, while at the same time reliably classifying the activity typ

Tracking human motion with multiple cameras using articulated ICP with hard constraints

Questa tesi propone un nuovo algoritmo basato su ICP per il tracking di un modello scheletrico articolato di un corpo umano. L\u2019algoritmo proposto prende in input immagini calibrate di un soggetto, calcola la ricostruzione volumetrica e la linea mediale del corpo e quindi posiziona in modo adeguato il modello, composto di segmenti, in ogni frame usando una versione di ICP modificata (versione che usa una strategia di attraversamento alberi gerarchica che mantiene connessi tutti i segmenti del modello nei giunti relativi). L\u2019approccio proposto usa limiti cinematica per i giunti e un filtro di Kalman esteso per fare il tracking del modello. Il primo contributo originale di questa tesi \ue8 l\u2019algoritmo per trovare i punti sullo scheletro di un volume tridimensionale. L\u2019algoritmo, usando una tecnica di slicing trova l\u2019asse mediale di un volume 3D in modo veloce utilizzando il processore della scheda grafica e le texture units della scheda stessa. Questo algoritmo produce ottimi risultati per quanto riguarda la qualit\ue0 e le prestazioni se comparato con altri algoritmi in letteratura. Un altro contributo originale \ue8 l\u2019introduzione di una nuova strategia di tracking basata su un approccio gerarchico dell\u2019algoritmo ICP, utilizzato per trovare le congruenze tra un modello di corpo umano composto da soli segmenti e un insieme di punti 3D. L\u2019algoritmo usa una versione di ICP dove tutti i punti 3D sono pesati in funzione del segmento del corpo preso in considerazione dall\u2019algoritmo in quel momento. L\u2019applicazione di queste tecniche dimostra la bont\ue0 del metodo e le prestazioni ottenute in termini di qualit\ue0 della stima della posa sono comparabili con altri lavori in letteratura. I risultati presentati nella tesi dimostrano la fattibilit\ue0 dell\u2019approccio generale, che si intende utilizzare in un sistema completo per il tracking di corpi umani senza l\u2019uso di marcatori. In futuro il lavoro pu\uf2 essere esteso ottimizzando l\u2019implementazione e la codifica in modo da poter ottenere prestazioni real-time.This thesis proposed a new ICP-based algorithm for tracking articulated skeletal model of a human body. The proposed algorithm takes as input multiple calibrated views of the subject, computes a volumetric reconstruction and the centerlines of the body and fits the skeletal body model in each frame using a hierarchic tree traversal version of the ICP algorithm that preserves the connection of the segments at the joints. The proposed approach uses the kinematic constraints and an Extended Kalman Filter to track the body pose. The first contribution is a new algorithm to find the skeletal points of a 3D volume. The algorithm using a slicing technique find the medial axis of a volume in a fast way using the graphic card processor and the texture units. This algorithm produce good results in quality and performance compared to other works in literature. Another contribution is the introduction of a new tracking strategy based on a hierarchical application of the ICP standard algorithm to find the match between a stick body model and a set of 3D points. The algorithm use a traversing version of ICP where also all the 3D points are weighted in such a way every limbs of the model can best fit on the right portion of the body. The application of these techniques shown the feasibility of the method and the performances obtained in terms of quality of estimate pose are comparable with other works in literature. The results presented here demonstrate the feasibility of the approach, which is is intended to be used in complete system for vision-based markerless human body tracking. Future work will aimed at optimizing the implementation, in order to achieve real-time performances

Detección automática de manos con caminos geodésicos en datos multi-modales

Treballs Finals de Grau d'Enginyeria Informàtica, Facultat de Matemàtiques, Universitat de Barcelona, Any:2013, Director: Sergio Escalera Guerrero i Albert Clapés SintesIn the last years, with the appearance of the multi-modal RGB-Depth information provided by the low cost KinectTM sensor, new ways of solving Computer Vision challenges have come and new strategies have been proposed. In this work the main problem is automatic hand detection in multi-modal RGB-Depth visual data. This task involves several difficulties due to the changes in illumination, viewport variations and articulated nature of the human body as well as its high flexibility. In order to solve it the present work proposes an accurate and efficient method based on hypothesis that the hand landmarks remain at a nearly constant geodesic distance from automatically located anatomical reference point. In a given frame, the human body is segmented first in the depth image. Then, a graph representation of the body is built in which the geodesic paths are computed from the reference point. The dense optical flow vectors on the corresponding RGB image are used to reduce ambiguities of the geodesic paths’ connectivity, allowing to eliminate false edges interconnecting different body parts. Finally, as the result, exact coordinates of both hands are obtained without involving costly learning procedures