Hand-Object Interaction Detection with Fully Convolutional Networks

Schröder M, Ritter H. Hand-Object Interaction Detection with Fully Convolutional Networks. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. 2017: 18-25.Detecting hand-object interactions is a challenging problem with many applications in the human-computer interaction domain. We present a real-time method that automatically detects hand-object interactions in RGBD sensor data and tracks the object’s rigid pose over time. The detection is performed using a fully convolutional neural network, which is purposefully trained to discern the relationship between hands and objects and which predicts pixel-wise class probabilities. This output is used in a probabilistic pixel labeling strategy that explicitly accounts for the uncertainty of the prediction. Based on the labeling of object pixels, the object is tracked over time using modelbased registration. We evaluate the accuracy and generalizability of our approach and make our annotated RGBD dataset as well as our trained models publicly available