4 research outputs found
Learning Segment Similarity and Alignment in Large-Scale Content Based Video Retrieval
With the explosive growth of web videos in recent years, large-scale
Content-Based Video Retrieval (CBVR) becomes increasingly essential in video
filtering, recommendation, and copyright protection. Segment-level CBVR
(S-CBVR) locates the start and end time of similar segments in finer
granularity, which is beneficial for user browsing efficiency and infringement
detection especially in long video scenarios. The challenge of S-CBVR task is
how to achieve high temporal alignment accuracy with efficient computation and
low storage consumption. In this paper, we propose a Segment Similarity and
Alignment Network (SSAN) in dealing with the challenge which is firstly trained
end-to-end in S-CBVR. SSAN is based on two newly proposed modules in video
retrieval: (1) An efficient Self-supervised Keyframe Extraction (SKE) module to
reduce redundant frame features, (2) A robust Similarity Pattern Detection
(SPD) module for temporal alignment. In comparison with uniform frame
extraction, SKE not only saves feature storage and search time, but also
introduces comparable accuracy and limited extra computation time. In terms of
temporal alignment, SPD localizes similar segments with higher accuracy and
efficiency than existing deep learning methods. Furthermore, we jointly train
SSAN with SKE and SPD and achieve an end-to-end improvement. Meanwhile, the two
key modules SKE and SPD can also be effectively inserted into other video
retrieval pipelines and gain considerable performance improvements.
Experimental results on public datasets show that SSAN can obtain higher
alignment accuracy while saving storage and online query computational cost
compared to existing methods.Comment: Accepted by ACM MM 202
Temporal Matching Kernel with Explicit Feature Maps
International audienceThis paper proposes a framework for content-based video retrieval that addresses various tasks as particular event retrieval , copy detection or video synchronization. Given a video query, the method is able to efficiently retrieve, from a large collection, similar video events or near-duplicates with temporarily consistent excerpts. As a byproduct of the representation, it provides a precise temporal alignment of the query and the detected video excerpts. Our method converts a series of frame descriptors into a single visual-temporal descriptor, called a temporal invariant match kernel. This representation takes into account the relative positions of the visual frames: the frame descriptors are jointly encoded with their timestamps. When matching two videos, the method produces a score function for all possible relative timestamps, which is maximized to obtain both the similarity score and the relative time offset. Then, we propose two complementary contributions to further improve the detection and localization performance. The first is a novel query expansion method that takes advantage of the joint descriptor/timestamp representation to automatically align the first result set and produce an enriched temporal query. In contrast to other query expansion methods proposed for videos, it preserves the localization capability. Second, we improve the localization trade-off between quality and representation size by using several complementary temporal match kernels. We evaluate our approach on benchmarks for particular event retrieval, copy detection and video synchronization. Our experiments show that our approach achieve excellent detection and localization results