27,185 research outputs found
Exploiting Prompt Caption for Video Grounding
Video grounding aims to locate a moment of interest matching the given query
sentence from an untrimmed video. Previous works ignore the \emph{sparsity
dilemma} in video annotations, which fails to provide the context information
between potential events and query sentences in the dataset. In this paper, we
contend that exploiting easily available captions which describe general
actions \ie, prompt captions (PC) defined in our paper, will significantly
boost the performance. To this end, we propose a Prompt Caption Network (PCNet)
for video grounding. Specifically, we first introduce dense video captioning to
generate dense captions and then obtain prompt captions by Non-Prompt Caption
Suppression (NPCS). To capture the potential information in prompt captions, we
propose Caption Guided Attention (CGA) project the semantic relations between
prompt captions and query sentences into temporal space and fuse them into
visual representations. Considering the gap between prompt captions and ground
truth, we propose Asymmetric Cross-modal Contrastive Learning (ACCL) for
constructing more negative pairs to maximize cross-modal mutual information.
Without bells and whistles, extensive experiments on three public datasets
(\ie, ActivityNet Captions, TACoS and ActivityNet-CG) demonstrate that our
method significantly outperforms state-of-the-art methods
Generation-Guided Multi-Level Unified Network for Video Grounding
Video grounding aims to locate the timestamps best matching the query
description within an untrimmed video. Prevalent methods can be divided into
moment-level and clip-level frameworks. Moment-level approaches directly
predict the probability of each transient moment to be the boundary in a global
perspective, and they usually perform better in coarse grounding. On the other
hand, clip-level ones aggregate the moments in different time windows into
proposals and then deduce the most similar one, leading to its advantage in
fine-grained grounding. In this paper, we propose a multi-level unified
framework to enhance performance by leveraging the merits of both moment-level
and clip-level methods. Moreover, a novel generation-guided paradigm in both
levels is adopted. It introduces a multi-modal generator to produce the
implicit boundary feature and clip feature, later regarded as queries to
calculate the boundary scores by a discriminator. The generation-guided
solution enhances video grounding from a two-unique-modals' match task to a
cross-modal attention task, which steps out of the previous framework and
obtains notable gains. The proposed Generation-guided Multi-level Unified
network (GMU) surpasses previous methods and reaches State-Of-The-Art on
various benchmarks with disparate features, e.g., Charades-STA, ActivityNet
captions
Learning to Ground Instructional Articles in Videos through Narrations
In this paper we present an approach for localizing steps of procedural
activities in narrated how-to videos. To deal with the scarcity of labeled data
at scale, we source the step descriptions from a language knowledge base
(wikiHow) containing instructional articles for a large variety of procedural
tasks. Without any form of manual supervision, our model learns to temporally
ground the steps of procedural articles in how-to videos by matching three
modalities: frames, narrations, and step descriptions. Specifically, our method
aligns steps to video by fusing information from two distinct pathways: i) {\em
direct} alignment of step descriptions to frames, ii) {\em indirect} alignment
obtained by composing steps-to-narrations with narrations-to-video
correspondences. Notably, our approach performs global temporal grounding of
all steps in an article at once by exploiting order information, and is trained
with step pseudo-labels which are iteratively refined and aggressively
filtered. In order to validate our model we introduce a new evaluation
benchmark -- HT-Step -- obtained by manually annotating a 124-hour subset of
HowTo100M\footnote{A test server is accessible at
\url{https://eval.ai/web/challenges/challenge-page/2082}.} with steps sourced
from wikiHow articles. Experiments on this benchmark as well as zero-shot
evaluations on CrossTask demonstrate that our multi-modality alignment yields
dramatic gains over several baselines and prior works. Finally, we show that
our inner module for matching narration-to-video outperforms by a large margin
the state of the art on the HTM-Align narration-video alignment benchmark.Comment: 17 pages, 4 figures and 10 table
- …