144 research outputs found
Multimodal Multipart Learning for Action Recognition in Depth Videos
The articulated and complex nature of human actions makes the task of action
recognition difficult. One approach to handle this complexity is dividing it to
the kinetics of body parts and analyzing the actions based on these partial
descriptors. We propose a joint sparse regression based learning method which
utilizes the structured sparsity to model each action as a combination of
multimodal features from a sparse set of body parts. To represent dynamics and
appearance of parts, we employ a heterogeneous set of depth and skeleton based
features. The proper structure of multimodal multipart features are formulated
into the learning framework via the proposed hierarchical mixed norm, to
regularize the structured features of each part and to apply sparsity between
them, in favor of a group feature selection. Our experimental results expose
the effectiveness of the proposed learning method in which it outperforms other
methods in all three tested datasets while saturating one of them by achieving
perfect accuracy
Skeleton-Based Human Action Recognition with Global Context-Aware Attention LSTM Networks
Human action recognition in 3D skeleton sequences has attracted a lot of
research attention. Recently, Long Short-Term Memory (LSTM) networks have shown
promising performance in this task due to their strengths in modeling the
dependencies and dynamics in sequential data. As not all skeletal joints are
informative for action recognition, and the irrelevant joints often bring noise
which can degrade the performance, we need to pay more attention to the
informative ones. However, the original LSTM network does not have explicit
attention ability. In this paper, we propose a new class of LSTM network,
Global Context-Aware Attention LSTM (GCA-LSTM), for skeleton based action
recognition. This network is capable of selectively focusing on the informative
joints in each frame of each skeleton sequence by using a global context memory
cell. To further improve the attention capability of our network, we also
introduce a recurrent attention mechanism, with which the attention performance
of the network can be enhanced progressively. Moreover, we propose a stepwise
training scheme in order to train our network effectively. Our approach
achieves state-of-the-art performance on five challenging benchmark datasets
for skeleton based action recognition
Learning Scene Flow With Skeleton Guidance For 3D Action Recognition
Among the existing modalities for 3D action recognition, 3D flow has been
poorly examined, although conveying rich motion information cues for human
actions. Presumably, its susceptibility to noise renders it intractable, thus
challenging the learning process within deep models. This work demonstrates the
use of 3D flow sequence by a deep spatiotemporal model and further proposes an
incremental two-level spatial attention mechanism, guided from skeleton domain,
for emphasizing motion features close to the body joint areas and according to
their informativeness. Towards this end, an extended deep skeleton model is
also introduced to learn the most discriminant action motion dynamics, so as to
estimate an informativeness score for each joint. Subsequently, a late fusion
scheme is adopted between the two models for learning the high level
cross-modal correlations. Experimental results on the currently largest and
most challenging dataset NTU RGB+D, demonstrate the effectiveness of the
proposed approach, achieving state-of-the-art results.Comment: 18 pages, 3 figures, 3 tables, conferenc
ViLP: Knowledge Exploration using Vision, Language, and Pose Embeddings for Video Action Recognition
Video Action Recognition (VAR) is a challenging task due to its inherent
complexities. Though different approaches have been explored in the literature,
designing a unified framework to recognize a large number of human actions is
still a challenging problem. Recently, Multi-Modal Learning (MML) has
demonstrated promising results in this domain. In literature, 2D skeleton or
pose modality has often been used for this task, either independently or in
conjunction with the visual information (RGB modality) present in videos.
However, the combination of pose, visual information, and text attributes has
not been explored yet, though text and pose attributes independently have been
proven to be effective in numerous computer vision tasks. In this paper, we
present the first pose augmented Vision-language model (VLM) for VAR. Notably,
our scheme achieves an accuracy of 92.81% and 73.02% on two popular human video
action recognition benchmark datasets, UCF-101 and HMDB-51, respectively, even
without any video data pre-training, and an accuracy of 96.11% and 75.75% after
kinetics pre-training.Comment: 7 pages, 3 figures, 2 Table
- …