This paper presents the competitive video retrieval capabilities of vitrivr.  The vitrivr stack is the continuation of the IMOTION system which participated to the Video Browser Showdown competitions since 2015. The primary focus of vitrivr and its participation in this competition is to simplify and generalize the system's individual components, making them easier to deploy and use. The entire vitrivr stack is made available as open source software

Gasser, Ralph

Giangreco, Ivan

Rossetto, Luca

Schuldt, Heiko

edoc

Competitive Video Retrieval with vitrivrLuca Rossetto, Ivan Giangreco, Ralph Gasser, and Heiko SchuldtDatabases and Information Systems Research GroupDepartment of Mathematics and Computer ScienceUniversity of Basel, Switzerland{firstname.lastname}@unibas.chAbstract. This paper presents the competitive video retrieval capabil-ities of vitrivr. The vitrivr stack is the continuation of the IMOTIONsystem which participated to the Video Browser Showdown competi-tions since 2015. The primary focus of vitrivr and its participation inthis competition is to simplify and generalize the system’s individualcomponents, making them easier to deploy and use. The entire vitrivrstack is made available as open source software.1 IntroductionIn this paper we present the current iteration of vitrivr [6], an open-sourcecontent-based multimedia retrieval stack. The vitrivr stack is the continuationof the IMOTION system [3, 5, 7, 8] which participated in previous iterations ofthe Video Browser Showdown [1]. Despite offering some new functionality, theprimary focus for this years participation lies in the simplification and general-ization of the retrieval stack in order to make it easier to adapt, deploy, and useby both experts and laymen. The vitrivr stack is available in its entirety fromhttps://vitrivr.org.The remainder of this paper is structured as follows: Section 2 provides a briefoverview of the overall system architecture and Section 3 summarizes all querytypes supported by vitrivr. Section 4 provides details on the functionalities in-troduced in the current version. In Section 5, we briefly outline our reasoningbehind the open sourcing of vitrivr and Section 6 concludes.2 Architectural OverviewThe vitrivr stack – like its predecessor IMOTION – consists of three primary sys-tem components: the storage layer ADAMpro [2], the retrieval engine Cineast [4],and a browser-based user interface. Additionally, a web server is used to servestatic content such as videos and thumbnail images. Additional details on thearchitecture of the entire stack can be found in [6].3 Interaction and Query TypesThe vitrivr stack offers various ways in which queries can be specified. Basically,they can for the most part be grouped into two categories: visual and textual.The visual query modes include Query-by-Sketch and Query-by-Example as wellas Relevance Feedback which are based on visual input such as user generatedsketches of a scene or one or multiple previously retrieved scenes. These queriesare performed based on data extracted directly from the video frames. Thetextual queries are based on information which can be extracted from the videocontent and represented as text, such as spoken language, text on screen, orthe provided textual video meta data. For this we use the ASR data providedwith the video data set as well as several object detectors to produce labels forthe shots. OCR is applied in order to make text which might appear on screensearchable as well.4 New FunctionalityWhile the IMOTION system that has participated in previous instances of VBShas always been a specialized piece of software, purpose-built for the competition,the functionality we added to vitrivr in preparation for this iteration of VBS aresuch that they are also useful for other use cases of vitrivr.4.1 New User InterfaceThe most salient difference to the IMOTION System of the previous year isthe new user interface. While still browser-based, the latest iteration of the UIis based upon the Angular framework1. Its modular structure makes it easy tocustomize the entire UI or parts thereof to shift its focus from general purposemultimedia retrieval to, in this case, competitive video retrieval.As in the past, the UI enables result streaming in order to be able to alreadypresent partial results to the user while the query is still being processed bythe backend. It, however, achieves this no longer via AJAX requests but ratheruses a WebSocket connection to Cineast. A REST-API is also available. Thestack still includes a web server which provides the static content such as shotthumbnails and the videos themselves, but it is no longer required to act as aproxy between the browser and Cineast. The screenshot in Figure 1 depicts thecurrent version of the UI.4.2 Approximate RetrievalThe underlying storage engine ADAMpro [2] supports multiple index structuresfor efficient vector space retrieval. Many of these index structures achieve theirhigh efficiency by approximating results rather than producing the true nearest1 https://angular.io/Fig. 1. Screenshot of the vitrivr UI for large-scale competitive video searchneighbors of a query vector. In previous system iterations, we only made use ofexact query results which lead to longer query times. In the current iterationof the system, the choice as to whether exact or approximate queries should beused can be made at query time. Hence, the user can sacrifice some accuracy togain major speed-ups.5 Open SourceThe entire vitrivr stack [6] is published under the MIT license, the source codeof all its components is available from their individual GitHub2 repositories,additional documentation can be found on https://vitrivr.org. Being a general-purpose multimedia retrieval stack, vitrivr has many applications outside of com-petitive video retrieval as it also supports other domains such as Images, Audio,and 3D-models. With this flexible open source stack, we hope to offer the com-munity the basis for future research in many areas and domains of multimediaretrieval.6 ConclusionsWith the competitive video retrieval version of vitrivr, we plan to continue thesuccessful participations with we had with the IMOTION system in the past. Itis our hope that by publishing the entire retrieval stack as open source software,we lower the entry hurdle for future participants and provide some accelerationfor the testing of new ideas in the context of large-scale video retrieval.2 https://github.com/vitrivrAcknowledgementsThis work was partly supported by the Chist-Era project IMOTION with con-tributions from the Swiss National Science Foundation (SNSF, contract no.20CH21 151571).References1. Claudiu Cobaˆrzan, Klaus Schoeffmann, Werner Bailer, Wolfgang Hu¨rst, AdamBlazˇek, Jakub Lokocˇ, Stefanos Vrochidis, Kai Uwe Barthel, and Luca Rossetto. In-teractive Video Search Tools: a detailed Analysis of the Video Browser Showdown2015. Multimedia Tools and Applications, 76(4):5539–5571, 2017.2. Ivan Giangreco and Heiko Schuldt. ADAMpro: Database Support for Big MultimediaRetrieval. Datenbank-Spektrum, 16(1):17–26, 2016.3. Luca Rossetto, Ivan Giangreco, Silvan Heller, Claudiu Ta˘nase, Heiko Schuldt,Ste´phane Dupont, Omar Seddati, Metin Sezgin, Ozan Can Altıok, and YusufSahilliog˘lu. IMOTION – Searching for Video Sequences using Multi-shot SketchQueries. In Proceedings of the 22nd International Conference on Multimedia Mod-eling (MMM 2016), volume 9517 of Lecture Notes in Computer Science, pages 377–382, Miami, FL, USA, January 2016. Springer.4. Luca Rossetto, Ivan Giangreco, and Heiko Schuldt. Cineast: a Multi-feature Sketch-based Video Retrieval Engine. In Proceedings of the 2014 IEEE International Sym-posium on Multimedia (ISM 2014), pages 18–23, Taichung, Taiwan, December 2014.IEEE Computer Society.5. Luca Rossetto, Ivan Giangreco, Heiko Schuldt, Ste´phane Dupont, Omar Seddati,Metin Sezgin, and Yusuf Sahilliog˘lu. IMOTION - A Content-based Video RetrievalEngine. In Proceedings of the 21st International Conference on Multimedia Modeling(MMM 2015), volume 8936 of Lecture Notes in Computer Science, pages 255–260,Sydney, NSW, Australia, January 2015. Springer.6. Luca Rossetto, Ivan Giangreco, Claudiu Ta˘nase, and Heiko Schuldt. vitrivr: A Flex-ible Retrieval Stack Supporting Multiple Query Modes for Searching in MultimediaCollections. In Proceedings of the 2016 ACM Conference on Multimedia Conference(ACM MM 2016), pages 1183–1186, Amsterdam, The Netherlands, October 2016.ACM.7. Luca Rossetto, Ivan Giangreco, Claudiu Ta˘nase, and Heiko Schuldt. MultimodalVideo Retrieval with the 2017 IMOTION System. In Proceedings of the 2017 ACMInternational Conference on Multimedia Retrieval (ICMR 2017), pages 457–460,Bucharest, Romania, June 2017. ACM.8. Luca Rossetto, Ivan Giangreco, Claudiu Ta˘nase, Heiko Schuldt, Ste´phane Dupont,and Omar Seddati. Enhanced Retrieval and Browsing in the IMOTION System. InProceedings of the 23rd International Conference on Multimedia Modeling (MMM2017), volume 10133 of Lecture Notes in Computer Science, pages 469–474, Reyk-javik, Iceland, January 2017. Springer.

Competitive Video Retrieval with vitrivr

https://edoc.unibas.ch/58217/1/20171229095620_5a460334032d4.pdf

Competitive Video Retrieval with vitrivr

Abstract

Similar works

Full text

Available Versions

edoc