1,966 research outputs found
Combined CNN Transformer Encoder for Enhanced Fine-grained Human Action Recognition
Fine-grained action recognition is a challenging task in computer vision. As
fine-grained datasets have small inter-class variations in spatial and temporal
space, fine-grained action recognition model requires good temporal reasoning
and discrimination of attribute action semantics. Leveraging on CNN's ability
in capturing high level spatial-temporal feature representations and
Transformer's modeling efficiency in capturing latent semantics and global
dependencies, we investigate two frameworks that combine CNN vision backbone
and Transformer Encoder to enhance fine-grained action recognition: 1) a
vision-based encoder to learn latent temporal semantics, and 2) a multi-modal
video-text cross encoder to exploit additional text input and learn cross
association between visual and text semantics. Our experimental results show
that both our Transformer encoder frameworks effectively learn latent temporal
semantics and cross-modality association, with improved recognition performance
over CNN vision model. We achieve new state-of-the-art performance on the
FineGym benchmark dataset for both proposed architectures.Comment: The Ninth Workshop on Fine-Grained Visual Categorization (FGVC9) @
CVPR202
SR-GNN: Spatial Relation-aware Graph Neural Network for Fine-Grained Image Categorization
Over the past few years, a significant progress has been made in deep
convolutional neural networks (CNNs)-based image recognition. This is mainly
due to the strong ability of such networks in mining discriminative object pose
and parts information from texture and shape. This is often inappropriate for
fine-grained visual classification (FGVC) since it exhibits high intra-class
and low inter-class variances due to occlusions, deformation, illuminations,
etc. Thus, an expressive feature representation describing global structural
information is a key to characterize an object/ scene. To this end, we propose
a method that effectively captures subtle changes by aggregating context-aware
features from most relevant image-regions and their importance in
discriminating fine-grained categories avoiding the bounding-box and/or
distinguishable part annotations. Our approach is inspired by the recent
advancement in self-attention and graph neural networks (GNNs) approaches to
include a simple yet effective relation-aware feature transformation and its
refinement using a context-aware attention mechanism to boost the
discriminability of the transformed feature in an end-to-end learning process.
Our model is evaluated on eight benchmark datasets consisting of fine-grained
objects and human-object interactions. It outperforms the state-of-the-art
approaches by a significant margin in recognition accuracy.Comment: Accepted manuscript - IEEE Transaction on Image Processin
Self-Supervised 3D Action Representation Learning with Skeleton Cloud Colorization
3D Skeleton-based human action recognition has attracted increasing attention
in recent years. Most of the existing work focuses on supervised learning which
requires a large number of labeled action sequences that are often expensive
and time-consuming to annotate. In this paper, we address self-supervised 3D
action representation learning for skeleton-based action recognition. We
investigate self-supervised representation learning and design a novel skeleton
cloud colorization technique that is capable of learning spatial and temporal
skeleton representations from unlabeled skeleton sequence data. We represent a
skeleton action sequence as a 3D skeleton cloud and colorize each point in the
cloud according to its temporal and spatial orders in the original
(unannotated) skeleton sequence. Leveraging the colorized skeleton point cloud,
we design an auto-encoder framework that can learn spatial-temporal features
from the artificial color labels of skeleton joints effectively. Specifically,
we design a two-steam pretraining network that leverages fine-grained and
coarse-grained colorization to learn multi-scale spatial-temporal features. In
addition, we design a Masked Skeleton Cloud Repainting task that can pretrain
the designed auto-encoder framework to learn informative representations. We
evaluate our skeleton cloud colorization approach with linear classifiers
trained under different configurations, including unsupervised,
semi-supervised, fully-supervised, and transfer learning settings. Extensive
experiments on NTU RGB+D, NTU RGB+D 120, PKU-MMD, NW-UCLA, and UWA3D datasets
show that the proposed method outperforms existing unsupervised and
semi-supervised 3D action recognition methods by large margins and achieves
competitive performance in supervised 3D action recognition as well.Comment: This work is an extension of our ICCV 2021 paper [arXiv:2108.01959]
https://openaccess.thecvf.com/content/ICCV2021/html/Yang_Skeleton_Cloud_Colorization_for_Unsupervised_3D_Action_Representation_Learning_ICCV_2021_paper.htm
SR-GNN: Spatial Relation-aware Graph Neural Network for Fine-Grained Image Categorization
Over the past few years, a significant progress has been made in deep
convolutional neural networks (CNNs)-based image recognition. This is mainly
due to the strong ability of such networks in mining discriminative object pose
and parts information from texture and shape. This is often inappropriate for
fine-grained visual classification (FGVC) since it exhibits high intra-class
and low inter-class variances due to occlusions, deformation, illuminations,
etc. Thus, an expressive feature representation describing global structural
information is a key to characterize an object/ scene. To this end, we propose
a method that effectively captures subtle changes by aggregating context-aware
features from most relevant image-regions and their importance in
discriminating fine-grained categories avoiding the bounding-box and/or
distinguishable part annotations. Our approach is inspired by the recent
advancement in self-attention and graph neural networks (GNNs) approaches to
include a simple yet effective relation-aware feature transformation and its
refinement using a context-aware attention mechanism to boost the
discriminability of the transformed feature in an end-to-end learning process.
Our model is evaluated on eight benchmark datasets consisting of fine-grained
objects and human-object interactions. It outperforms the state-of-the-art
approaches by a significant margin in recognition accuracy.Comment: Accepted manuscript - IEEE Transaction on Image Processin
- …