9 research outputs found
Ordered Pooling of Optical Flow Sequences for Action Recognition
Training of Convolutional Neural Networks (CNNs) on long video sequences is
computationally expensive due to the substantial memory requirements and the
massive number of parameters that deep architectures demand. Early fusion of
video frames is thus a standard technique, in which several consecutive frames
are first agglomerated into a compact representation, and then fed into the CNN
as an input sample. For this purpose, a summarization approach that represents
a set of consecutive RGB frames by a single dynamic image to capture pixel
dynamics is proposed recently. In this paper, we introduce a novel ordered
representation of consecutive optical flow frames as an alternative and argue
that this representation captures the action dynamics more effectively than RGB
frames. We provide intuitions on why such a representation is better for action
recognition. We validate our claims on standard benchmark datasets and
demonstrate that using summaries of flow images lead to significant
improvements over RGB frames while achieving accuracy comparable to the
state-of-the-art on UCF101 and HMDB datasets.Comment: Accepted in WACV 201
Non-Linear Temporal Subspace Representations for Activity Recognition
Representations that can compactly and effectively capture the temporal
evolution of semantic content are important to computer vision and machine
learning algorithms that operate on multi-variate time-series data. We
investigate such representations motivated by the task of human action
recognition. Here each data instance is encoded by a multivariate feature (such
as via a deep CNN) where action dynamics are characterized by their variations
in time. As these features are often non-linear, we propose a novel pooling
method, kernelized rank pooling, that represents a given sequence compactly as
the pre-image of the parameters of a hyperplane in a reproducing kernel Hilbert
space, projections of data onto which captures their temporal order. We develop
this idea further and show that such a pooling scheme can be cast as an
order-constrained kernelized PCA objective. We then propose to use the
parameters of a kernelized low-rank feature subspace as the representation of
the sequences. We cast our formulation as an optimization problem on
generalized Grassmann manifolds and then solve it efficiently using Riemannian
optimization techniques. We present experiments on several action recognition
datasets using diverse feature modalities and demonstrate state-of-the-art
results.Comment: Accepted at the IEEE International Conference on Computer Vision and
Pattern Recognition, CVPR, 2018. arXiv admin note: substantial text overlap
with arXiv:1705.0858
Discriminative Video Representation Learning
Representation learning is a fundamental research problem in the area of machine learning, refining the raw data to discover representations needed for various applications. However, real-world data, particularly video data, is neither mathematically nor computationally convenient to process due to its semantic redundancy and complexity. Video data, as opposed to images, includes temporal correlation and motion dynamics, but the ground truth label is normally limited to category labels, which makes the video representation learning a challenging problem. To this end, this thesis addresses the problem of video representation learning, specifically discriminative video representation learning, which focuses on capturing useful data distributions and reliable feature representations improving the performance of varied downstream tasks. We argue that neither all frames in one video nor all dimensions in one feature vector are useful and should be equally treated for video representation learning. Based on this argument, several novel algorithms are investigated in this thesis under multiple application scenarios, such as action recognition, action detection and one-class video anomaly detection. These proposed video representation learning methods produce discriminative video features in both deep and non-deep learning setups. Specifically, they are presented in the form of: 1) an early fusion layer that adopts a temporal ranking SVM formulation, agglomerating several optical flow images from consecutive frames into a novel compact representation, named as dynamic optical flow images; 2) an intermediate feature aggregation layer that applies weakly-supervised contrastive learning techniques, learning discriminative video representations via contrasting positive and negative samples from a sequence; 3) a new formulation for one-class feature learning that learns a set of discriminative subspaces with orthonormal hyperplanes to flexibly bound the one-class data distribution using Riemannian optimisation methods. We provide extensive experiments to gain intuitions into why the learned representations are discriminative and useful. All the proposed methods in this thesis are evaluated on standard publicly available benchmarks, demonstrating state-of-the-art performance