233,650 research outputs found
Query-aware Long Video Localization and Relation Discrimination for Deep Video Understanding
The surge in video and social media content underscores the need for a deeper
understanding of multimedia data. Most of the existing mature video
understanding techniques perform well with short formats and content that
requires only shallow understanding, but do not perform well with long format
videos that require deep understanding and reasoning. Deep Video Understanding
(DVU) Challenge aims to push the boundaries of multimodal extraction, fusion,
and analytics to address the problem of holistically analyzing long videos and
extract useful knowledge to solve different types of queries. This paper
introduces a query-aware method for long video localization and relation
discrimination, leveraging an imagelanguage pretrained model. This model
adeptly selects frames pertinent to queries, obviating the need for a complete
movie-level knowledge graph. Our approach achieved first and fourth positions
for two groups of movie-level queries. Sufficient experiments and final
rankings demonstrate its effectiveness and robustness.Comment: ACM MM 2023 Grand Challeng
Contrastive Masked Autoencoders for Self-Supervised Video Hashing
Self-Supervised Video Hashing (SSVH) models learn to generate short binary
representations for videos without ground-truth supervision, facilitating
large-scale video retrieval efficiency and attracting increasing research
attention. The success of SSVH lies in the understanding of video content and
the ability to capture the semantic relation among unlabeled videos. Typically,
state-of-the-art SSVH methods consider these two points in a two-stage training
pipeline, where they firstly train an auxiliary network by instance-wise
mask-and-predict tasks and secondly train a hashing model to preserve the
pseudo-neighborhood structure transferred from the auxiliary network. This
consecutive training strategy is inflexible and also unnecessary. In this
paper, we propose a simple yet effective one-stage SSVH method called ConMH,
which incorporates video semantic information and video similarity relationship
understanding in a single stage. To capture video semantic information for
better hashing learning, we adopt an encoder-decoder structure to reconstruct
the video from its temporal-masked frames. Particularly, we find that a higher
masking ratio helps video understanding. Besides, we fully exploit the
similarity relationship between videos by maximizing agreement between two
augmented views of a video, which contributes to more discriminative and robust
hash codes. Extensive experiments on three large-scale video datasets (i.e.,
FCVID, ActivityNet and YFCC) indicate that ConMH achieves state-of-the-art
results. Code is available at https://github.com/huangmozhi9527/ConMH.Comment: This work is accepted by the AAAI 2023. 9 pages, 6 figures, 6 table
- …