26 research outputs found
Transductive Zero-Shot Action Recognition by Word-Vector Embedding
The number of categories for action recognition is growing rapidly and it has
become increasingly hard to label sufficient training data for learning
conventional models for all categories. Instead of collecting ever more data
and labelling them exhaustively for all categories, an attractive alternative
approach is zero-shot learning" (ZSL). To that end, in this study we construct
a mapping between visual features and a semantic descriptor of each action
category, allowing new categories to be recognised in the absence of any visual
training data. Existing ZSL studies focus primarily on still images, and
attribute-based semantic representations. In this work, we explore word-vectors
as the shared semantic space to embed videos and category labels for ZSL action
recognition. This is a more challenging problem than existing ZSL of still
images and/or attributes, because the mapping between video spacetime features
of actions and the semantic space is more complex and harder to learn for the
purpose of generalising over any cross-category domain shift. To solve this
generalisation problem in ZSL action recognition, we investigate a series of
synergistic strategies to improve upon the standard ZSL pipeline. Most of these
strategies are transductive in nature which means access to testing data in the
training phase.Comment: Accepted by IJC
Spatial-Aware Object Embeddings for Zero-Shot Localization and Classification of Actions
We aim for zero-shot localization and classification of human actions in
video. Where traditional approaches rely on global attribute or object
classification scores for their zero-shot knowledge transfer, our main
contribution is a spatial-aware object embedding. To arrive at spatial
awareness, we build our embedding on top of freely available actor and object
detectors. Relevance of objects is determined in a word embedding space and
further enforced with estimated spatial preferences. Besides local object
awareness, we also embed global object awareness into our embedding to maximize
actor and object interaction. Finally, we exploit the object positions and
sizes in the spatial-aware embedding to demonstrate a new spatio-temporal
action retrieval scenario with composite queries. Action localization and
classification experiments on four contemporary action video datasets support
our proposal. Apart from state-of-the-art results in the zero-shot localization
and classification settings, our spatial-aware embedding is even competitive
with recent supervised action localization alternatives.Comment: ICC