2 research outputs found
Dual Encoding for Zero-Example Video Retrieval
This paper attacks the challenging problem of zero-example video retrieval.
In such a retrieval paradigm, an end user searches for unlabeled videos by
ad-hoc queries described in natural language text with no visual example
provided. Given videos as sequences of frames and queries as sequences of
words, an effective sequence-to-sequence cross-modal matching is required. The
majority of existing methods are concept based, extracting relevant concepts
from queries and videos and accordingly establishing associations between the
two modalities. In contrast, this paper takes a concept-free approach,
proposing a dual deep encoding network that encodes videos and queries into
powerful dense representations of their own. Dual encoding is conceptually
simple, practically effective and end-to-end. As experiments on three
benchmarks, i.e. MSR-VTT, TRECVID 2016 and 2017 Ad-hoc Video Search show, the
proposed solution establishes a new state-of-the-art for zero-example video
retrieval.Comment: Accepted by CVPR 2019. Code and data are available at
https://github.com/danieljf24/dual_encodin
SEA: Sentence Encoder Assembly for Video Retrieval by Textual Queries
Retrieving unlabeled videos by textual queries, known as Ad-hoc Video Search
(AVS), is a core theme in multimedia data management and retrieval. The success
of AVS counts on cross-modal representation learning that encodes both query
sentences and videos into common spaces for semantic similarity computation.
Inspired by the initial success of previously few works in combining multiple
sentence encoders, this paper takes a step forward by developing a new and
general method for effectively exploiting diverse sentence encoders. The
novelty of the proposed method, which we term Sentence Encoder Assembly (SEA),
is two-fold. First, different from prior art that use only a single common
space, SEA supports text-video matching in multiple encoder-specific common
spaces. Such a property prevents the matching from being dominated by a
specific encoder that produces an encoding vector much longer than other
encoders. Second, in order to explore complementarities among the individual
common spaces, we propose multi-space multi-loss learning. As extensive
experiments on four benchmarks (MSR-VTT, TRECVID AVS 2016-2019, TGIF and MSVD)
show, SEA surpasses the state-of-the-art. In addition, SEA is extremely ease to
implement. All this makes SEA an appealing solution for AVS and promising for
continuously advancing the task by harvesting new sentence encoders.Comment: accepted for publication as a REGULAR paper in the IEEE Transactions
on Multimedi