13 research outputs found
Action Sets: Weakly Supervised Action Segmentation without Ordering Constraints
Action detection and temporal segmentation of actions in videos are topics of
increasing interest. While fully supervised systems have gained much attention
lately, full annotation of each action within the video is costly and
impractical for large amounts of video data. Thus, weakly supervised action
detection and temporal segmentation methods are of great importance. While most
works in this area assume an ordered sequence of occurring actions to be given,
our approach only uses a set of actions. Such action sets provide much less
supervision since neither action ordering nor the number of action occurrences
are known. In exchange, they can be easily obtained, for instance, from
meta-tags, while ordered sequences still require human annotation. We introduce
a system that automatically learns to temporally segment and label actions in a
video, where the only supervision that is used are action sets. An evaluation
on three datasets shows that our method still achieves good results although
the amount of supervision is significantly smaller than for other related
methods.Comment: CVPR 201
Actor and Action Video Segmentation from a Sentence
This paper strives for pixel-level segmentation of actors and their actions
in video content. Different from existing works, which all learn to segment
from a fixed vocabulary of actor and action pairs, we infer the segmentation
from a natural language input sentence. This allows to distinguish between
fine-grained actors in the same super-category, identify actor and action
instances, and segment pairs that are outside of the actor and action
vocabulary. We propose a fully-convolutional model for pixel-level actor and
action segmentation using an encoder-decoder architecture optimized for video.
To show the potential of actor and action video segmentation from a sentence,
we extend two popular actor and action datasets with more than 7,500 natural
language descriptions. Experiments demonstrate the quality of the
sentence-guided segmentations, the generalization ability of our model, and its
advantage for traditional actor and action segmentation compared to the
state-of-the-art.Comment: Accepted to CVPR 2018 as ora
Multiscale human activity recognition and anticipation network
Deep convolutional neural networks have been leveraged to achieve huge improvements in video understanding and human activity recognition performance in the past decade. However, most existing methods focus on activities that have similar time scales, leaving the task of action recognition on multiscale human behaviors less explored. In this study, a two-stream multiscale human activity recognition and anticipation (MS-HARA) network is proposed, which is jointly optimized using a multitask learning method. The MS-HARA network fuses the two streams of the network using an efficient temporal-channel attention (TCA)-based fusion approach to improve the model's representational ability for both temporal and spatial features. We investigate the multiscale human activities from two basic categories, namely, midterm activities and long-term activities. The network is designed to function as part of a real-time processing framework to support interaction and mutual understanding between humans and intelligent machines. It achieves state-of-the-art results on several datasets for different tasks and different application domains. The midterm and long-term action recognition and anticipation performance, as well as the network fusion, are extensively tested to show the efficiency of the proposed network. The results show that the MS-HARA network can easily be extended to different application domains