27 research outputs found
Learning to Discriminate Information for Online Action Detection
From a streaming video, online action detection aims to identify actions in
the present. For this task, previous methods use recurrent networks to model
the temporal sequence of current action frames. However, these methods overlook
the fact that an input image sequence includes background and irrelevant
actions as well as the action of interest. For online action detection, in this
paper, we propose a novel recurrent unit to explicitly discriminate the
information relevant to an ongoing action from others. Our unit, named
Information Discrimination Unit (IDU), decides whether to accumulate input
information based on its relevance to the current action. This enables our
recurrent network with IDU to learn a more discriminative representation for
identifying ongoing actions. In experiments on two benchmark datasets, TVSeries
and THUMOS-14, the proposed method outperforms state-of-the-art methods by a
significant margin. Moreover, we demonstrate the effectiveness of our recurrent
unit by conducting comprehensive ablation studies.Comment: To appear in CVPR 202
Multi-Modal Few-Shot Temporal Action Detection via Vision-Language Meta-Adaptation
Few-shot (FS) and zero-shot (ZS) learning are two different approaches for
scaling temporal action detection (TAD) to new classes. The former adapts a
pretrained vision model to a new task represented by as few as a single video
per class, whilst the latter requires no training examples by exploiting a
semantic description of the new class. In this work, we introduce a new
multi-modality few-shot (MMFS) TAD problem, which can be considered as a
marriage of FS-TAD and ZS-TAD by leveraging few-shot support videos and new
class names jointly. To tackle this problem, we further introduce a novel
MUlti-modality PromPt mETa-learning (MUPPET) method. This is enabled by
efficiently bridging pretrained vision and language models whilst maximally
reusing already learned capacity. Concretely, we construct multi-modal prompts
by mapping support videos into the textual token space of a vision-language
model using a meta-learned adapter-equipped visual semantics tokenizer. To
tackle large intra-class variation, we further design a query feature
regulation scheme. Extensive experiments on ActivityNetv1.3 and THUMOS14
demonstrate that our MUPPET outperforms state-of-the-art alternative methods,
often by a large margin. We also show that our MUPPET can be easily extended to
tackle the few-shot object detection problem and again achieves the
state-of-the-art performance on MS-COCO dataset. The code will be available in
https://github.com/sauradip/MUPPETComment: Technical Repor
CLIP-TSA: CLIP-Assisted Temporal Self-Attention for Weakly-Supervised Video Anomaly Detection
Video anomaly detection (VAD) -- commonly formulated as a multiple-instance
learning problem in a weakly-supervised manner due to its labor-intensive
nature -- is a challenging problem in video surveillance where the frames of
anomaly need to be localized in an untrimmed video. In this paper, we first
propose to utilize the ViT-encoded visual features from CLIP, in contrast with
the conventional C3D or I3D features in the domain, to efficiently extract
discriminative representations in the novel technique. We then model long- and
short-range temporal dependencies and nominate the snippets of interest by
leveraging our proposed Temporal Self-Attention (TSA). The ablation study
conducted on each component confirms its effectiveness in the problem, and the
extensive experiments show that our proposed CLIP-TSA outperforms the existing
state-of-the-art (SOTA) methods by a large margin on two commonly-used
benchmark datasets in the VAD problem (UCF-Crime and ShanghaiTech Campus). The
source code will be made publicly available upon acceptance.Comment: Under Submissio
Rescaling Egocentric Vision:Collection Pipeline and Challenges for EPIC-KITCHENS-100
This paper introduces the pipeline to extend the largest dataset in egocentric vision, EPIC-KITCHENS. The effort culminates in EPIC-KITCHENS-100, a collection of 100 hours, 20M frames, 90K actions in 700 variable-length videos, capturing long-term unscripted activities in 45 environments, using head-mounted cameras. Compared to its previous version (Damen in Scaling egocentric vision: ECCV, 2018), EPIC-KITCHENS-100 has been annotated using a novel pipeline that allows denser (54% more actions per minute) and more complete annotations of fine-grained actions (+128% more action segments). This collection enables new challenges such as action detection and evaluating the “test of time”—i.e. whether models trained on data collected in 2018 can generalise to new footage collected two years later. The dataset is aligned with 6 challenges: action recognition (full and weak supervision), action detection, action anticipation, cross-modal retrieval (from captions), as well as unsupervised domain adaptation for action recognition. For each challenge, we define the task, provide baselines and evaluation metrics.Published versionResearch at Bristol is supported by Engineering and Physical Sciences Research Council (EPSRC) Doctoral Training Program (DTP), EPSRC Fellowship UMPIRE (EP/T004991/1). Research at Catania is sponsored by Piano della Ricerca 2016-2018 linea di Intervento 2 of DMI, by MISE - PON I&C 2014-2020, ENIGMA project (CUP: B61B19000520008) and by MIUR AIM - Attrazione e Mobilita Internazionale Linea 1 - AIM1893589 - CUP E64118002540007