3D Skeleton-based human action recognition has attracted increasing attention
in recent years. Most of the existing work focuses on supervised learning which
requires a large number of labeled action sequences that are often expensive
and time-consuming to annotate. In this paper, we address self-supervised 3D
action representation learning for skeleton-based action recognition. We
investigate self-supervised representation learning and design a novel skeleton
cloud colorization technique that is capable of learning spatial and temporal
skeleton representations from unlabeled skeleton sequence data. We represent a
skeleton action sequence as a 3D skeleton cloud and colorize each point in the
cloud according to its temporal and spatial orders in the original
(unannotated) skeleton sequence. Leveraging the colorized skeleton point cloud,
we design an auto-encoder framework that can learn spatial-temporal features
from the artificial color labels of skeleton joints effectively. Specifically,
we design a two-steam pretraining network that leverages fine-grained and
coarse-grained colorization to learn multi-scale spatial-temporal features. In
addition, we design a Masked Skeleton Cloud Repainting task that can pretrain
the designed auto-encoder framework to learn informative representations. We
evaluate our skeleton cloud colorization approach with linear classifiers
trained under different configurations, including unsupervised,
semi-supervised, fully-supervised, and transfer learning settings. Extensive
experiments on NTU RGB+D, NTU RGB+D 120, PKU-MMD, NW-UCLA, and UWA3D datasets
show that the proposed method outperforms existing unsupervised and
semi-supervised 3D action recognition methods by large margins and achieves
competitive performance in supervised 3D action recognition as well.Comment: This work is an extension of our ICCV 2021 paper [arXiv:2108.01959]
https://openaccess.thecvf.com/content/ICCV2021/html/Yang_Skeleton_Cloud_Colorization_for_Unsupervised_3D_Action_Representation_Learning_ICCV_2021_paper.htm