1,720 research outputs found
3D Human Pose Estimation using Spatio-Temporal Networks with Explicit Occlusion Training
Estimating 3D poses from a monocular video is still a challenging task,
despite the significant progress that has been made in recent years. Generally,
the performance of existing methods drops when the target person is too
small/large, or the motion is too fast/slow relative to the scale and speed of
the training data. Moreover, to our knowledge, many of these methods are not
designed or trained under severe occlusion explicitly, making their performance
on handling occlusion compromised. Addressing these problems, we introduce a
spatio-temporal network for robust 3D human pose estimation. As humans in
videos may appear in different scales and have various motion speeds, we apply
multi-scale spatial features for 2D joints or keypoints prediction in each
individual frame, and multi-stride temporal convolutional net-works (TCNs) to
estimate 3D joints or keypoints. Furthermore, we design a spatio-temporal
discriminator based on body structures as well as limb motions to assess
whether the predicted pose forms a valid pose and a valid movement. During
training, we explicitly mask out some keypoints to simulate various occlusion
cases, from minor to severe occlusion, so that our network can learn better and
becomes robust to various degrees of occlusion. As there are limited 3D
ground-truth data, we further utilize 2D video data to inject a semi-supervised
learning capability to our network. Experiments on public datasets validate the
effectiveness of our method, and our ablation studies show the strengths of our
network\'s individual submodules.Comment: 8 pages, AAAI 202
๋ง์คํฌ ์ธ์ด ๋ชจ๋ธ์ ์ด์ฉํ 3์ฐจ์ ์ ์ขํ์ ๋ฏธ์ธ์กฐ์
ํ์๋
ผ๋ฌธ(์์ฌ) -- ์์ธ๋ํ๊ต๋ํ์ : ๊ณต๊ณผ๋ํ ์ปดํจํฐ๊ณตํ๋ถ, 2021.8. ๋ฌธ๋ณ๋ก.3D Hand Pose Estimation ๋ฌธ์ ๋ ํ ์ฅ์ 2์ฐจ์ image๋ฅผ ์ด์ฉํ์ฌ 3์ฐจ์์ ์ ์ขํ๋ฅผ ์ถ์ ํ๋ ๋ฌธ์ ๋ก, 2์ฐจ์ image์์ ์์ ์ผ๋ถ๊ฐ ๊ฐ๋ ค์ง๋ ๊ฒฝ์ฐ๋ค ๋๋ฌธ์ ํ์กดํ๋ ์ ๊ทผ๋ฐฉ์์ผ๋ก ํ๊ธฐ์ ๊น๋ค๋ก์ด ๋ฌธ์ ์ด๋ค. ์ต๊ทผ์ ์ฐ๊ตฌ์๋ค์ ๋ง์ ์์ ๋ฐ์ดํฐ๋ฅผ ๋ชจ์ผ๊ฑฐ๋, ํฉ์ฑํ์ฌ ์ด ๋ฌธ์ ๋ฅผ ํด๊ฒฐํ๋ ค ํ๋ค. ํ์ง๋ง, ์ด๋ฌํ ๋ฐ์ดํฐ ๊ธฐ๋ฐ ์ ๊ทผ๋ค์ ๊ฐ ๊ด์ ๋ค์ด ๋
๋ฆฝ์ ์ด๋ผ ๊ฐ์ ํ๊ณ ๋ฌธ์ ๋ฅผ ํ๊ฑฐ๋, ๋ฌผ๋ฆฌ์ ์ผ๋ก ๋๋ฌ๋๋ ๊ด์ ๋ค์ ์ฐ๊ฒฐ ๊ด๊ณ๋ง ๊ฐ์ง๊ณ ์ ๋ฌธ์ ๋ฅผ ํด๊ฒฐํ๋ ค ํ๊ธฐ๋๋ฌธ์ ์ฑ๋ฅ ํฅ์์ ํ๊ณ๊ฐ ์์๋ค.
์ด๋ฌํ ๋ฌธ์ ๋ฅผ ์ํํ๊ธฐ ์ํด, ์ ๊ด์ ์ 3์ฐจ์ ์ขํ๋ค ๊ฐ์ ๊ด๊ณ๋ฅผ ํ์ต์ํค๊ณ ์ด๋ฅผ ๊ธฐ๋ฐ์ผ๋ก ๋ฏธ์ธ์กฐ์ ์ ํ์ฌ ์ ๋ฐ์ ์ธ ์ฑ๋ฅ์ ๋์ด ์ฌ๋ฆฌ๋ ๋ฐฉ๋ฒ์ ์ด ๋
ผ๋ฌธ์์ ์ ์ ํ๋ค. ์์ฐ์ด ์ฒ๋ฆฌ ๋ถ์ผ์์ ๊ฐ์ฅ ๋ง์ด ์ฐ์ด๋ BERT ๋ชจ๋ธ์์ ์๊ฐ์ ๋ฐ์์ผ๋ฉฐ, BERT๋ฅผ ์ด์ฉํ์ฌ ๋ณด์ด์ง ์๋ ์์ ๋ํ์ฌ ์ ์ถ์ ํ๋๋ก ํ๋ ๋ชจ๋์ ์ถ๊ฐํจ์ผ๋ก์จ ๊ธฐ์กด์ ์๋ ์ ๊ทผ๋ฐฉ์๋ค ๋ณด๋ค ๋ ์ข์ ๊ฒฐ๊ณผ๋ฅผ ์คํ์์ ์ป์ ์ ์์๋ค. ๋ํ, ๋ฌผ๋ฆฌ์ ์ธ ๊ด์ ๊ฐ์ ์ฐ๊ฒฐ ๊ด๊ณ์ ๊ฐํ์์ง ์๊ณ , ๋ชจ๋ธ์ด ๋ฐ์ดํฐ๋ก๋ถํฐ ๊ฐ ๊ด์ ๊ฐ์ ์ํฅ๋ ฅ์ ํ์
ํ์ฌ ์์น๋ฅผ ์ธ๋ถ ์กฐ์ ํ๊ฒ ํ์๋ค. ์ด๋ ๊ฒ ํ์ตํ ์ฐ๊ฒฐ ๊ด๊ณ๋ฅผ ์๊ฐํ ํ์ฌ ์ด ๋
ผ๋ฌธ์ ์ผ๋ถ ์๊ฐํ์๊ณ , ์ด๋ฅผ ํตํด ๋์ ๋ณด์ด๋ ๋ฌผ๋ฆฌ์ ์ธ ์ฐ๊ฒฐ๊ด๊ณ ๋ฟ๋ง ์๋๋ผ ๊ด๊ณ์์ด ๋ณด์ด๋ ๊ด์ ๋ค ๊ฐ์๋ ์ํฅ์ ์ฃผ๊ณ ๋ฐ๊ณ ์๋ค๊ณ ๊ฐ์ ํ๋ ๊ฒ์ด ํจ์ฌ ๋ ์ข์ ์ ๊ทผ๋ฐฉ๋ฒ์์ ๊ด์ฐฐํ ์ ์์๋ค.Accurately estimating hand/body pose from a single viewpoint under occlusion is challenging for most of the current approaches. Recent approaches have tried to address the occlusion problem by collecting or synthesizing images having joint occlusions. However, the data-driven approaches failed to tackle the occlusion because they assumed that joints are independent or they only used explicit joint connection.
To mitigate this problem, I propose a method that learns joint relations and refines the occluded information based on their relation. Inspired by BERT in Natural Language Processing, I pre-train a refinement module and add it at the end of the proposed framework. Refinement improves not only the accuracy of occluded joints but also the accuracy of whole joints. In addition, instead of using a physical connection between joints, the proposed model learns their relation from the data. I visualized the learned joint relation in this paper, and it implies that assuming explicit connection hinders the model from accurately predicting joint locations accurately.1 INTRODUCTION 1
2 RELATEDWORKS 3
3 PRELIMINARIES 5
3.1 Attention Mechanism 5
3.2 Transformer 5
3.3 Masked Language Model 6
4 Method 9
4.1 Problem Definition 9
4.2 3D Hand Pose Estimation Framework 9
4.2.1 Dense Representation Module 10
4.2.2 3D Regression Module 10
4.2.3 Joint Refinement Module 11
4.3 Pre-training 11
4.3.1 Stacked HourGlass 12
4.3.2 Joint Refinement Module 13
4.4 Training 13
5 EXPERIMENTS 17
5.1 Dataset 17
5.2 Experimental Results 18
5.2.1 Quantative Results 18
5.2.2 Qualitative Results 18
5.2.3 Computational Complexity 19
6 CONCLUSION 26
Bibliography 27
Abstract (In Korean) 34์
A Grid-based Representation for Human Action Recognition
Human action recognition (HAR) in videos is a fundamental research topic in
computer vision. It consists mainly in understanding actions performed by
humans based on a sequence of visual observations. In recent years, HAR have
witnessed significant progress, especially with the emergence of deep learning
models. However, most of existing approaches for action recognition rely on
information that is not always relevant for this task, and are limited in the
way they fuse the temporal information. In this paper, we propose a novel
method for human action recognition that encodes efficiently the most
discriminative appearance information of an action with explicit attention on
representative pose features, into a new compact grid representation. Our GRAR
(Grid-based Representation for Action Recognition) method is tested on several
benchmark datasets demonstrating that our model can accurately recognize human
actions, despite intra-class appearance variations and occlusion challenges.Comment: Accepted on 25th International Conference on Pattern Recognition
(ICPR 2020
Enhanced Spatio-Temporal Context for Temporally Consistent Robust 3D Human Motion Recovery from Monocular Videos
Recovering temporally consistent 3D human body pose, shape and motion from a
monocular video is a challenging task due to (self-)occlusions, poor lighting
conditions, complex articulated body poses, depth ambiguity, and limited
availability of annotated data. Further, doing a simple perframe estimation is
insufficient as it leads to jittery and implausible results. In this paper, we
propose a novel method for temporally consistent motion estimation from a
monocular video. Instead of using generic ResNet-like features, our method uses
a body-aware feature representation and an independent per-frame pose and
camera initialization over a temporal window followed by a novel
spatio-temporal feature aggregation by using a combination of self-similarity
and self-attention over the body-aware features and the perframe
initialization. Together, they yield enhanced spatiotemporal context for every
frame by considering remaining past and future frames. These features are used
to predict the pose and shape parameters of the human body model, which are
further refined using an LSTM. Experimental results on the publicly available
benchmark data show that our method attains significantly lower acceleration
error and outperforms the existing state-of-the-art methods over all key
quantitative evaluation metrics, including complex scenarios like partial
occlusion, complex poses and even relatively low illumination
- โฆ