3,430 research outputs found
Second-order Temporal Pooling for Action Recognition
Deep learning models for video-based action recognition usually generate
features for short clips (consisting of a few frames); such clip-level features
are aggregated to video-level representations by computing statistics on these
features. Typically zero-th (max) or the first-order (average) statistics are
used. In this paper, we explore the benefits of using second-order statistics.
Specifically, we propose a novel end-to-end learnable feature aggregation
scheme, dubbed temporal correlation pooling that generates an action descriptor
for a video sequence by capturing the similarities between the temporal
evolution of clip-level CNN features computed across the video. Such a
descriptor, while being computationally cheap, also naturally encodes the
co-activations of multiple CNN features, thereby providing a richer
characterization of actions than their first-order counterparts. We also
propose higher-order extensions of this scheme by computing correlations after
embedding the CNN features in a reproducing kernel Hilbert space. We provide
experiments on benchmark datasets such as HMDB-51 and UCF-101, fine-grained
datasets such as MPII Cooking activities and JHMDB, as well as the recent
Kinetics-600. Our results demonstrate the advantages of higher-order pooling
schemes that when combined with hand-crafted features (as is standard practice)
achieves state-of-the-art accuracy.Comment: Accepted in the International Journal of Computer Vision (IJCV
RGB-T Tracking Based on Mixed Attention
RGB-T tracking involves the use of images from both visible and thermal
modalities. The primary objective is to adaptively leverage the relatively
dominant modality in varying conditions to achieve more robust tracking
compared to single-modality tracking. An RGB-T tracker based on mixed attention
mechanism to achieve complementary fusion of modalities (referred to as MACFT)
is proposed in this paper. In the feature extraction stage, we utilize
different transformer backbone branches to extract specific and shared
information from different modalities. By performing mixed attention operations
in the backbone to enable information interaction and self-enhancement between
the template and search images, it constructs a robust feature representation
that better understands the high-level semantic features of the target. Then,
in the feature fusion stage, a modality-adaptive fusion is achieved through a
mixed attention-based modality fusion network, which suppresses the low-quality
modality noise while enhancing the information of the dominant modality.
Evaluation on multiple RGB-T public datasets demonstrates that our proposed
tracker outperforms other RGB-T trackers on general evaluation metrics while
also being able to adapt to longterm tracking scenarios.Comment: 14 pages, 10 figure
Dense Feature Aggregation and Pruning for RGBT Tracking
How to perform effective information fusion of different modalities is a core
factor in boosting the performance of RGBT tracking. This paper presents a
novel deep fusion algorithm based on the representations from an end-to-end
trained convolutional neural network. To deploy the complementarity of features
of all layers, we propose a recursive strategy to densely aggregate these
features that yield robust representations of target objects in each modality.
In different modalities, we propose to prune the densely aggregated features of
all modalities in a collaborative way. In a specific, we employ the operations
of global average pooling and weighted random selection to perform channel
scoring and selection, which could remove redundant and noisy features to
achieve more robust feature representation. Experimental results on two RGBT
tracking benchmark datasets suggest that our tracker achieves clear
state-of-the-art against other RGB and RGBT tracking methods.Comment: arXiv admin note: text overlap with arXiv:1811.0985
Tracking by 3D Model Estimation of Unknown Objects in Videos
Most model-free visual object tracking methods formulate the tracking task as
object location estimation given by a 2D segmentation or a bounding box in each
video frame. We argue that this representation is limited and instead propose
to guide and improve 2D tracking with an explicit object representation, namely
the textured 3D shape and 6DoF pose in each video frame. Our representation
tackles a complex long-term dense correspondence problem between all 3D points
on the object for all video frames, including frames where some points are
invisible. To achieve that, the estimation is driven by re-rendering the input
video frames as well as possible through differentiable rendering, which has
not been used for tracking before. The proposed optimization minimizes a novel
loss function to estimate the best 3D shape, texture, and 6DoF pose. We improve
the state-of-the-art in 2D segmentation tracking on three different datasets
with mostly rigid objects
- …