3D Human Pose Estimation using Spatio-Temporal Networks with Explicit Occlusion Training

Estimating 3D poses from a monocular video is still a challenging task, despite the significant progress that has been made in recent years. Generally, the performance of existing methods drops when the target person is too small/large, or the motion is too fast/slow relative to the scale and speed of the training data. Moreover, to our knowledge, many of these methods are not designed or trained under severe occlusion explicitly, making their performance on handling occlusion compromised. Addressing these problems, we introduce a spatio-temporal network for robust 3D human pose estimation. As humans in videos may appear in different scales and have various motion speeds, we apply multi-scale spatial features for 2D joints or keypoints prediction in each individual frame, and multi-stride temporal convolutional net-works (TCNs) to estimate 3D joints or keypoints. Furthermore, we design a spatio-temporal discriminator based on body structures as well as limb motions to assess whether the predicted pose forms a valid pose and a valid movement. During training, we explicitly mask out some keypoints to simulate various occlusion cases, from minor to severe occlusion, so that our network can learn better and becomes robust to various degrees of occlusion. As there are limited 3D ground-truth data, we further utilize 2D video data to inject a semi-supervised learning capability to our network. Experiments on public datasets validate the effectiveness of our method, and our ablation studies show the strengths of our network\'s individual submodules.Comment: 8 pages, AAAI 202

마스크 언어 모델을 이용한 3차원 손 좌표의 미세조정

학위논문(석사) -- 서울대학교대학원 : 공과대학 컴퓨터공학부, 2021.8. 문병로.3D Hand Pose Estimation 문제는 한 장의 2차원 image를 이용하여 3차원의 손 좌표를 추정하는 문제로, 2차원 image에서 손의 일부가 가려지는 경우들 때문에 현존하는 접근방식으로 풀기에 까다로운 문제이다. 최근에 연구자들은 많은 양의 데이터를 모으거나, 합성하여 이 문제를 해결하려 했다. 하지만, 이러한 데이터 기반 접근들은 각 관절들이 독립적이라 가정하고 문제를 풀거나, 물리적으로 드러나는 관절들의 연결 관계만 가지고서 문제를 해결하려 했기때문에 성능 향상에 한계가 있었다. 이러한 문제를 완화하기 위해, 손 관절의 3차원 좌표들 간의 관계를 학습시키고 이를 기반으로 미세조정을 하여 전반적인 성능을 끌어 올리는 방법을 이 논문에서 제안 한다. 자연어 처리 분야에서 가장 많이 쓰이는 BERT 모델에서 영감을 받았으며, BERT를 이용하여 보이지 않는 손에 대하여 잘 추정하도록 하는 모듈을 추가함으로써 기존에 있던 접근방식들 보다 더 좋은 결과를 실험에서 얻을 수 있었다. 또한, 물리적인 관절간의 연결 관계에 갇혀있지 않고, 모델이 데이터로부터 각 관절간의 영향력을 파악하여 위치를 세부 조정하게 하였다. 이렇게 학습한 연결 관계를 시각화 하여 이 논문에 일부 소개하였고, 이를 통해 눈에 보이는 물리적인 연결관계 뿐만 아니라 관계없어 보이는 관절들 간에도 영향을 주고받고 있다고 가정하는 것이 훨씬 더 좋은 접근방법임을 관찰할 수 있었다.Accurately estimating hand/body pose from a single viewpoint under occlusion is challenging for most of the current approaches. Recent approaches have tried to address the occlusion problem by collecting or synthesizing images having joint occlusions. However, the data-driven approaches failed to tackle the occlusion because they assumed that joints are independent or they only used explicit joint connection. To mitigate this problem, I propose a method that learns joint relations and refines the occluded information based on their relation. Inspired by BERT in Natural Language Processing, I pre-train a refinement module and add it at the end of the proposed framework. Refinement improves not only the accuracy of occluded joints but also the accuracy of whole joints. In addition, instead of using a physical connection between joints, the proposed model learns their relation from the data. I visualized the learned joint relation in this paper, and it implies that assuming explicit connection hinders the model from accurately predicting joint locations accurately.1 INTRODUCTION 1 2 RELATEDWORKS 3 3 PRELIMINARIES 5 3.1 Attention Mechanism 5 3.2 Transformer 5 3.3 Masked Language Model 6 4 Method 9 4.1 Problem Definition 9 4.2 3D Hand Pose Estimation Framework 9 4.2.1 Dense Representation Module 10 4.2.2 3D Regression Module 10 4.2.3 Joint Refinement Module 11 4.3 Pre-training 11 4.3.1 Stacked HourGlass 12 4.3.2 Joint Refinement Module 13 4.4 Training 13 5 EXPERIMENTS 17 5.1 Dataset 17 5.2 Experimental Results 18 5.2.1 Quantative Results 18 5.2.2 Qualitative Results 18 5.2.3 Computational Complexity 19 6 CONCLUSION 26 Bibliography 27 Abstract (In Korean) 34석

A Grid-based Representation for Human Action Recognition

Human action recognition (HAR) in videos is a fundamental research topic in computer vision. It consists mainly in understanding actions performed by humans based on a sequence of visual observations. In recent years, HAR have witnessed significant progress, especially with the emergence of deep learning models. However, most of existing approaches for action recognition rely on information that is not always relevant for this task, and are limited in the way they fuse the temporal information. In this paper, we propose a novel method for human action recognition that encodes efficiently the most discriminative appearance information of an action with explicit attention on representative pose features, into a new compact grid representation. Our GRAR (Grid-based Representation for Action Recognition) method is tested on several benchmark datasets demonstrating that our model can accurately recognize human actions, despite intra-class appearance variations and occlusion challenges.Comment: Accepted on 25th International Conference on Pattern Recognition (ICPR 2020