Interactive video retrieval evaluation at a distance: comparing sixteen interactive video search systems in a remote setting at the 10th Video Browser Showdown

The Video Browser Showdown addresses difficult video search challenges through an annual interactive evaluation campaign attracting research teams focusing on interactive video retrieval. The campaign aims to provide insights into the performance of participating interactive video retrieval systems, tested by selected search tasks on large video collections. For the first time in its ten year history, the Video Browser Showdown 2021 was organized in a fully remote setting and hosted a record number of sixteen scoring systems. In this paper, we describe the competition setting, tasks and results and give an overview of state-of-the-art methods used by the competing systems. By looking at query result logs provided by ten systems, we analyze differences in retrieval model performances and browsing times before a correct submission. Through advances in data gathering methodology and tools, we provide a comprehensive analysis of ad-hoc video search tasks, discuss results, task design and methodological challenges. We highlight that almost all top performing systems utilize some sort of joint embedding for text-image retrieval and enable specification of temporal context in queries for known-item search. Whereas a combination of these techniques drive the currently top performing systems, we identify several future challenges for interactive video search engines and the Video Browser Showdown competition itself

Evaluation of Keyword-Based Search Models for Known-Item Search

Video retrieval over large datasets is still a very challenging task, which is getting even more relevant with the rapidly growing volume of unannotated data available. Know-item search, as one of the video retrieval tasks, is limited primarily due to the limited ability of users to formulate a suitable query and low efectivity of search models. This thesis focuses mainly on selected search models based on image classifcation, which we will also compare with a commercial solution. We will examine how to transform the network output and what models to use. Also, the efect of iterative user query reformulation on overall search efectivity will be investigated. We will also present a simple simulated user model for the generation of artifcial queries and supporting software for data collection and model evaluation in a web interface.

Evaluation of Keyword-Based Search Models for Known-Item Search

Video retrieval over large datasets is still a very challenging task, which is getting even more relevant with the rapidly growing volume of unannotated data available. Know-item search, as one of the video retrieval tasks, is limited primarily due to the limited ability of users to formulate a suitable query and low efectivity of search models. This thesis focuses mainly on selected search models based on image classifcation, which we will also compare with a commercial solution. We will examine how to transform the network output and what models to use. Also, the efect of iterative user query reformulation on overall search efectivity will be investigated. We will also present a simple simulated user model for the generation of artifcial queries and supporting software for data collection and model evaluation in a web interface. 1Vyhledávání ve videu nad rozsáhlými databázemi je stále velmi náročný úkol, který je s rychle rostoucím objemem dostupných neanotovaných dat ještě aktuálnější. Hledání známé scény, jako jeden z úkolů vyhledávání ve videu, je limitováno především omeze- nou schopností uživatelů zformulovat vhodný dotaz a nízkou efektivitou vyhledávacích modelů. Tato práce se zaměřuje zejména na vybrané modely spoléhající na klasifkaci snímků, které vyhodnotí a porovná i s komerčním řešením. Prozkoumáme, jak transfor- movat výstup sítě a jaký z modelů poté použít a také vliv iterativní reformulace uži- vatelského dotazu na efektivitu hledání. Představíme i jednoduchý model simulovaného uživatele pro generování dotazů a software, který ve webovém rozhraní umožňuje sběr dat a následné evaluace. 1Department of Software EngineeringKatedra softwarového inženýrstvíMatematicko-fyzikální fakultaFaculty of Mathematics and Physic

Rychlý podpisový protokol pro autentizaci proudu zpráv založený na podpisech využívajících hešovací funkce

Security of the data streaming over Internet becomes a challenge if re- quirements such as post-quantum-capable cryptography and complete de- centralisation must be addressed. This thesis develops a connection-less, re-broadcastable data streaming protocol that allows a wholly decentralised, petname-based quantum-robust authentication of streaming sources based solely on the post-quantum hash-based few-time signature schemes. As the main contribution, the thesis benchmarks various trade-offs given by the problematic ephemeral nature of identities based on the few-time signature schemes and by the desired networking properties of the streaming proto- col. The benchmarks show that the schemes are practically extensible to realistic use cases, with only minor overhead. The proof-of-concept proto- col implementation is provided as a Rust library, together with the example application for live audio broadcasting. 1Bezpečnost streamování dat na Internetu je problematická především v případech, kdy uživatelé vyžadují kompletní decentralizaci a odolnost proti kvantové kryptoanalýze. Tato diplomová práce navrhuje nespojovaný pro- tokol pro přenos proudů dat, který umožňuje plně decentralizovanou a postk- vantovou autentizaci odesílatelů dat pomocí "petnames", využívající pouze post-kvantovou kryptografii založenou na hešovacích funkcích. Hlavním pří- nosem je systematické vyhodnocení dopadu použití autentizace pomocí few- time digitálních podpisů odvozených z hashovacích funkci na klíčové vlast- nosti protokolu. Výsledný protokol je možné úpravou parametrů efektivně škálovat pro bezpečné použití v realistickém prostředí Internetu. Práce dále popisuje prototyp implementace, sestávající se z knihovny v jazyce Rust a ukázkové aplikace pro živé vysílání zvuku. 1Department of Software EngineeringKatedra softwarového inženýrstvíFaculty of Mathematics and PhysicsMatematicko-fyzikální fakult

On the User-centric Comparative Remote Evaluation of Interactive Video Search Systems

In the research of video retrieval systems, comparative assessments during dedicated retrieval competitions provide priceless insights into the performance of individual systems. The scope and depth of such evaluations are unfortunately hard to improve, due to the limitations by the set-up costs, logistics, and organization complexity of large events. We show that this easily impairs the statistical significance of the collected results, and the reproducibility of the competition outcomes. In this article, we present a methodology for remote comparative evaluations of content-based video retrieval systems and demonstrate that such evaluations scale-up to sizes that reliably produce statistically robust results, and propose additional measures that increase the replicability of the experiment. The proposed remote evaluation methodology forms a major contribution toward open science in interactive retrieval benchmarks. At the same time, the detailed evaluation reports form an interesting source of new observations about many subtle, previously inaccessible aspects of video retrieval

On the User-centric Comparative Remote Evaluation of Interactive Video Search Systems

In the research of video retrieval systems, comparative assessments during dedicated retrieval competitions provide priceless insights into the performance of individual systems. The scope and depth of such evaluations is unfortunately hard to improve, due to the limitations by the set-up costs, logistics and organization complexity of large events. We show that this easily impairs the statistical significance of the collected results, and the reproducibility of the competition outcomes. In this paper, we present a methodology for remote comparative evaluations of content-based video retrieval systems and demonstrate that such evaluations scale-up to sizes that reliably produce statistically robust results, and propose additional measures that increase the replicability of the experiment. The proposed remote evaluation methodology forms a major contribution towards open science in interactive retrieval benchmarks. At the same time, the detailed evaluation reports form an interesting source of new observations about many subtle, previously inaccessible aspects of video retrieval