Hybrid One-Shot 3D Hand Pose Estimation by Exploiting Uncertainties

Model-based approaches to 3D hand tracking have been shown to perform well in a wide range of scenarios. However, they require initialisation and cannot recover easily from tracking failures that occur due to fast hand motions. Data-driven approaches, on the other hand, can quickly deliver a solution, but the results often suffer from lower accuracy or missing anatomical validity compared to those obtained from model-based approaches. In this work we propose a hybrid approach for hand pose estimation from a single depth image. First, a learned regressor is employed to deliver multiple initial hypotheses for the 3D position of each hand joint. Subsequently, the kinematic parameters of a 3D hand model are found by deliberately exploiting the inherent uncertainty of the inferred joint proposals. This way, the method provides anatomically valid and accurate solutions without requiring manual initialisation or suffering from track losses. Quantitative results on several standard datasets demonstrate that the proposed method outperforms state-of-the-art representatives of the model-based, data-driven and hybrid paradigms.Comment: BMVC 2015 (oral); see also http://lrs.icg.tugraz.at/research/hybridhape

Simultaneous Hand Pose and Skeleton Bone-Lengths Estimation from a Single Depth Image

Articulated hand pose estimation is a challenging task for human-computer interaction. The state-of-the-art hand pose estimation algorithms work only with one or a few subjects for which they have been calibrated or trained. Particularly, the hybrid methods based on learning followed by model fitting or model based deep learning do not explicitly consider varying hand shapes and sizes. In this work, we introduce a novel hybrid algorithm for estimating the 3D hand pose as well as bone-lengths of the hand skeleton at the same time, from a single depth image. The proposed CNN architecture learns hand pose parameters and scale parameters associated with the bone-lengths simultaneously. Subsequently, a new hybrid forward kinematics layer employs both parameters to estimate 3D joint positions of the hand. For end-to-end training, we combine three public datasets NYU, ICVL and MSRA-2015 in one unified format to achieve large variation in hand shapes and sizes. Among hybrid methods, our method shows improved accuracy over the state-of-the-art on the combined dataset and the ICVL dataset that contain multiple subjects. Also, our algorithm is demonstrated to work well with unseen images.Comment: This paper has been accepted and presented in 3DV-2017 conference held at Qingdao, China. http://irc.cs.sdu.edu.cn/3dv

Staged Probabilistic Regression for Hand Orientation Inference

Learning the global hand orientation from 2D monocular images is a challenging task, as the projected hand shape is affected by a number of variations. These include inter-person hand shape and size variations, intra-person pose and style variations and self-occlusion due to varying hand orientation. Given a hand orientation dataset containing these variations, a single regressor proves to be limited for learning the mapping of hand silhouette images onto the orientation angles. We address this by proposing a staged probabilistic regressor (SPORE) which consists of multiple expert regressors, each one learning a subset of variations from the dataset. Inspired by Boosting, the novelty of our method comes from the staged probabilistic learning, where each stage consists of training and adding an expert regressor to the intermediate ensemble of expert regressors. Unlike Boosting, we marginalize the posterior prediction probabilities from each expert regressor by learning a marginalization weights regressor, where the weights are extracted during training using a KullbackLeibler divergence-based optimization. We extend and evaluate our proposed framework for inferring hand orientation and pose simultaneously. In comparison to the state-of-the-art of hand orientation inference, multi-layered Random Forest marginalization and Boosting, our proposed method proves to be more accurate. Moreover, experimental results reveal that simultaneously learning hand orientation and pose from 2D monocular images significantly improves the pose classification performance