6 research outputs found
Regional Attention with Architecture-Rebuilt 3D Network for RGB-D Gesture Recognition
Human gesture recognition has drawn much attention in the area of computer
vision. However, the performance of gesture recognition is always influenced by
some gesture-irrelevant factors like the background and the clothes of
performers. Therefore, focusing on the regions of hand/arm is important to the
gesture recognition. Meanwhile, a more adaptive architecture-searched network
structure can also perform better than the block-fixed ones like Resnet since
it increases the diversity of features in different stages of the network
better. In this paper, we propose a regional attention with
architecture-rebuilt 3D network (RAAR3DNet) for gesture recognition. We replace
the fixed Inception modules with the automatically rebuilt structure through
the network via Neural Architecture Search (NAS), owing to the different shape
and representation ability of features in the early, middle, and late stage of
the network. It enables the network to capture different levels of feature
representations at different layers more adaptively. Meanwhile, we also design
a stackable regional attention module called dynamic-static Attention (DSA),
which derives a Gaussian guidance heatmap and dynamic motion map to highlight
the hand/arm regions and the motion information in the spatial and temporal
domains, respectively. Extensive experiments on two recent large-scale RGB-D
gesture datasets validate the effectiveness of the proposed method and show it
outperforms state-of-the-art methods. The codes of our method are available at:
https://github.com/zhoubenjia/RAAR3DNet.Comment: Accepted by AAAI 202
Multi-stage Factorized Spatio-Temporal Representation for RGB-D Action and Gesture Recognition
RGB-D action and gesture recognition remain an interesting topic in
human-centered scene understanding, primarily due to the multiple granularities
and large variation in human motion. Although many RGB-D based action and
gesture recognition approaches have demonstrated remarkable results by
utilizing highly integrated spatio-temporal representations across multiple
modalities (i.e., RGB and depth data), they still encounter several challenges.
Firstly, vanilla 3D convolution makes it hard to capture fine-grained motion
differences between local clips under different modalities. Secondly, the
intricate nature of highly integrated spatio-temporal modeling can lead to
optimization difficulties. Thirdly, duplicate and unnecessary information can
add complexity and complicate entangled spatio-temporal modeling. To address
the above issues, we propose an innovative heuristic architecture called
Multi-stage Factorized Spatio-Temporal (MFST) for RGB-D action and gesture
recognition. The proposed MFST model comprises a 3D Central Difference
Convolution Stem (CDC-Stem) module and multiple factorized spatio-temporal
stages. The CDC-Stem enriches fine-grained temporal perception, and the
multiple hierarchical spatio-temporal stages construct dimension-independent
higher-order semantic primitives. Specifically, the CDC-Stem module captures
bottom-level spatio-temporal features and passes them successively to the
following spatio-temporal factored stages to capture the hierarchical spatial
and temporal features through the Multi- Scale Convolution and Transformer
(MSC-Trans) hybrid block and Weight-shared Multi-Scale Transformer (WMS-Trans)
block. The seamless integration of these innovative designs results in a robust
spatio-temporal representation that outperforms state-of-the-art approaches on
RGB-D action and gesture recognition datasets.Comment: ACM MM'2
Searching multi-rate and multi-modal temporal enhanced networks for gesture recognition
Abstract
Gesture recognition has attracted considerable attention owing to its great potential in applications. Although the great progress has been made recently in multi-modal learning methods, existing methods still lack effective integration to fully explore synergies among spatio-temporal modalities effectively for gesture recognition. The problems are partially due to the fact that the existing manually designed network architectures have low efficiency in the joint learning of multi-modalities. In this paper, we propose the first neural architecture search (NAS)-based method for RGB-D gesture recognition. The proposed method includes two key components: 1) enhanced temporal representation via the proposed 3D Central Difference Convolution (3D-CDC) family, which is able to capture rich temporal context via aggregating temporal difference information; and 2) optimized backbones for multi-sampling-rate branches and lateral connections among varied modalities. The resultant multi-modal multi-rate network provides a new perspective to understand the relationship between RGB and depth modalities and their temporal dynamics. Comprehensive experiments are performed on three benchmark datasets (IsoGD, NvGesture, and EgoGesture), demonstrating the state-of-the-art performance in both single- and multi-modality settings. The code is available at https://github.com/ZitongYu/3DCDC-NAS