38 research outputs found
Action tube extraction based 3D-CNN for RGB-D action recognition
In this paper we propose a novel action tube extractor for RGB-D action recognition in trimmed videos. The action tube extractor takes as input a video and outputs an action tube. The method consists of two parts: spatial tube extraction and temporal sampling. The first part is built upon MobileNet-SSD and its role is to define the spatial region where the action takes place. The second part is based on the structural similarity index (SSIM) and is designed to remove frames without obvious motion from the primary action tube. The final extracted action tube has two benefits: 1) a higher ratio of ROI (subjects of action) to background; 2) most frames contain obvious motion change. We propose to use a two-stream (RGB and Depth) I3D architecture as our 3D-CNN model. Our approach outperforms the state-of-the-art methods on the OA and NTU RGB-D datasets. © 2018 IEEE.Peer ReviewedPostprint (published version
Guest editors' introduction to the special section on learning with Shared information for computer vision and multimedia analysis
The twelve papers in this special section focus on learning systems with shared information for computer vision and multimedia communication analysis. In the real world, a realistic setting for computer vision or multimedia recognition problems is that we have some classes containing lots of training data and many classes containing a small amount of training data. Therefore, how to use frequent classes to help learning rare classes for which it is harder to collect the training data is an open question. Learning with shared information is an emerging topic in machine learning, computer vision and multimedia analysis. There are different levels of components that can be shared during concept modeling and machine learning stages, such as sharing generic object parts, sharing attributes, sharing transformations, sharing regularization parameters and sharing training examples, etc. Regarding the specific methods, multi-task learning, transfer learning and deep learning can be seen as using different strategies to share information. These learning with shared information methods are very effective in solving real-world large-scale problems
Graph Distillation for Action Detection with Privileged Modalities
We propose a technique that tackles action detection in multimodal videos
under a realistic and challenging condition in which only limited training data
and partially observed modalities are available. Common methods in transfer
learning do not take advantage of the extra modalities potentially available in
the source domain. On the other hand, previous work on multimodal learning only
focuses on a single domain or task and does not handle the modality discrepancy
between training and testing. In this work, we propose a method termed graph
distillation that incorporates rich privileged information from a large-scale
multimodal dataset in the source domain, and improves the learning in the
target domain where training data and modalities are scarce. We evaluate our
approach on action classification and detection tasks in multimodal videos, and
show that our model outperforms the state-of-the-art by a large margin on the
NTU RGB+D and PKU-MMD benchmarks. The code is released at
http://alan.vision/eccv18_graph/.Comment: ECCV 201
Deep-Temporal LSTM for Daily Living Action Recognition
In this paper, we propose to improve the traditional use of RNNs by employing
a many to many model for video classification. We analyze the importance of
modeling spatial layout and temporal encoding for daily living action
recognition. Many RGB methods focus only on short term temporal information
obtained from optical flow. Skeleton based methods on the other hand show that
modeling long term skeleton evolution improves action recognition accuracy. In
this work, we propose a deep-temporal LSTM architecture which extends standard
LSTM and allows better encoding of temporal information. In addition, we
propose to fuse 3D skeleton geometry with deep static appearance. We validate
our approach on public available CAD60, MSRDailyActivity3D and NTU-RGB+D,
achieving competitive performance as compared to the state-of-the art.Comment: Submitted in conferenc
View-invariant action recognition
Human action recognition is an important problem in computer vision. It has a
wide range of applications in surveillance, human-computer interaction,
augmented reality, video indexing, and retrieval. The varying pattern of
spatio-temporal appearance generated by human action is key for identifying the
performed action. We have seen a lot of research exploring this dynamics of
spatio-temporal appearance for learning a visual representation of human
actions. However, most of the research in action recognition is focused on some
common viewpoints, and these approaches do not perform well when there is a
change in viewpoint. Human actions are performed in a 3-dimensional environment
and are projected to a 2-dimensional space when captured as a video from a
given viewpoint. Therefore, an action will have a different spatio-temporal
appearance from different viewpoints. The research in view-invariant action
recognition addresses this problem and focuses on recognizing human actions
from unseen viewpoints