4 research outputs found
Orthogonal Temporal Interpolation for Zero-Shot Video Recognition
Zero-shot video recognition (ZSVR) is a task that aims to recognize video
categories that have not been seen during the model training process. Recently,
vision-language models (VLMs) pre-trained on large-scale image-text pairs have
demonstrated impressive transferability for ZSVR. To make VLMs applicable to
the video domain, existing methods often use an additional temporal learning
module after the image-level encoder to learn the temporal relationships among
video frames. Unfortunately, for video from unseen categories, we observe an
abnormal phenomenon where the model that uses spatial-temporal feature performs
much worse than the model that removes temporal learning module and uses only
spatial feature. We conjecture that improper temporal modeling on video
disrupts the spatial feature of the video. To verify our hypothesis, we propose
Feature Factorization to retain the orthogonal temporal feature of the video
and use interpolation to construct refined spatial-temporal feature. The model
using appropriately refined spatial-temporal feature performs better than the
one using only spatial feature, which verifies the effectiveness of the
orthogonal temporal feature for the ZSVR task. Therefore, an Orthogonal
Temporal Interpolation module is designed to learn a better refined
spatial-temporal video feature during training. Additionally, a Matching Loss
is introduced to improve the quality of the orthogonal temporal feature. We
propose a model called OTI for ZSVR by employing orthogonal temporal
interpolation and the matching loss based on VLMs. The ZSVR accuracies on
popular video datasets (i.e., Kinetics-600, UCF101 and HMDB51) show that OTI
outperforms the previous state-of-the-art method by a clear margin
Joint Adversarial Domain Adaptation
Domain adaptation aims to transfer the enriched label knowledge from large amounts of source data to unlabeled target data. It has raised significant interest in multimedia analysis. Existing researches mainly focus on learning domain-wise transferable representations via statistical moment matching or adversarial adaptation techniques, while ignoring the class-wise mismatch across domains, resulting in inaccurate distribution alignment. To address this issue, we propose a Joint Adversarial Domain Adaptation (JADA) approach to simultaneously align domain-wise and class-wise distributions across source and target in a unified adversarial learning process. Specifically, JADA attempts to solve two complementary minimax problems jointly. The feature generator aims to not only fool the well-trained domain discriminator to learn domain-invariant features, but also minimize the disagreement between two distinct task-specific classifiers' predictions to synthesize target features near the support of source class-wisely. As a result, the learned transferable features will be equipped with more discriminative structures, and effectively avoid mode collapse. Additionally, JADA enables an efficient end-to-end training manner via a simple back-propagation scheme. Extensive experiments on several real-world cross-domain benchmarks, including VisDA-2017, ImageCLEF, Office-31 and digits, verify that JADA can gain remarkable improvements over other state-of-the-art deep domain adaptation approaches