Jónsson, Björn Thór

Khan, Omar Shahbaz

Koelma, Dennis C.

Larsen, Mathias Dybkjær

Poulsen, Liam Alex Sonto

Rudinac, Stevan

Worring, Marcel

Zahálka, Jan

English

Exquisitor is a scalable media exploration system based on interactive learning, which first took part in VBS in 2020. This paper presents an extension to Exquisitor, which supports operations on semantic classifiers to solve VBS tasks with temporal constraints. We outline the approach and present preliminary results, which indicate the potential of the approach

The IT University of Copenhagen's Repository

Exquisitor at the Video Browser Showdown 2021:Relationships Between Semantic ClassifiersOmar Shahbaz Khan1, Björn Þór Jónsson1, Mathias Larsen1, Liam Poulsen1,Dennis C. Koelma2, Stevan Rudinac2, Marcel Worring2, and Jan Zahálka31 IT University of Copenhagen, Denmark2 University of Amsterdam, Netherlands3 Czech Technical University in Prague, Czech RepublicAbstract. Exquisitor is a scalable media exploration system based oninteractive learning, which first took part in VBS in 2020. This paperpresents an extension to Exquisitor, which supports operations on se-mantic classifiers to solve VBS tasks with temporal constraints. We out-line the approach and present preliminary results, which indicate thepotential of the approach.Keywords: Interactive learning · Video browsing · Temporal relations.1 IntroductionThe Video Browser Showdown (VBS), now in its 10th anniversary edition, hasemerged as an important vehicle for the evolution of the multimedia field [5].During VBS, researchers are given a series of never-before-seen task descriptions,based on a collection of 7,475 video clips [9], and asked to interactively retrieveeither one specific video segment or multiple relevant segments, depending on thetask type. VBS allows researchers working on media exploration and search toolsto apply their techniques in a realistic setting and better understand the pros andcons of both the underlying techniques and the interfaces. The lessons learnedduring the competition can then inspire new methods and further research. Inaddition, the competitive setting makes for an exciting event where the rankingof systems can also give hints to their usability and applicability.Exquisitor, a prototype media exploration system based on interactive learn-ing, took part for the first time in VBS 2020, where it placed 5th out of 11systems [2]. The goal of Exquisitor, as applied to VBS, is to build a seman-tic classifier for the information need represented in each task, and use thatclassifier—along with metadata filters and a video timeline explorer—to solvethe task. Exquisitor uses the video segmentation supplied with the VBS col-lection and represents each video segment independently by semantic featuresderived from its keyframe. When building the semantic classifier, Exquisitor sug-gests keyframes to the user and asks for feedback on those suggestions. Once theuser spots a potentially relevant keyframe, the video explorer can then be usedto explore the actual content and internal structure of the full video clip.2 Khan, Jónsson, Larsen, Poulsen, Koelma, Rudinac, Worring, ZahálkaFor many VBS tasks, the task description applies to more than one video seg-ment, often focusing on different semantic concepts in different segments, andsometimes providing an explicit temporal relationship. Unsurprisingly, therefore,all the strongest VBS competitors provide temporal queries as a major tech-nique [4, 6, 10, 7]. Since video segmentation tends to split the video by semanticconcepts, a classifier built to find one segment may not find the other, and thesystem should provide support to utilise the relationship between concepts invideo segments.In this paper, we present a new version of Exquisitor, where the major ex-tension is the support for utilising relationships between semantic classifiers.While each semantic classifier is developed in the same manner as before, usingindependent video segments, the results of two semantic classifiers can now becombined in various ways, with an optional temporal relationship specification.In this paper we briefly outline the method and interface for combining two se-mantic models and show how two models combined could be used to solve twoVBS 2020 tasks, one of which the team failed to solve during the competition.We will present and evaluate the methods in more detail in a later publication.2 ExquisitorExquisitor is a user relevance feedback approach capable of handling large scalecollections in real time [3, 8]. The Exquisitor system used for VBS consists ofthree parts: (1) a web-based user interface for receiving and judging video sug-gestions; (2) an interactive learning server, which receives user judgments andproduces a new round of suggestions; and (3) a web server which serves videosand video thumbnails. Due to the computational efficiency of the system, allthree components can run locally on a laptop.Exquisitor Server: Exquisitor is fueled by a semantic model that combinesinteractive multimodal learning with cluster-based indexing. Each keyframe ineach modality is represented by an efficient representation containing the mostimportant semantic features, compressed using an index-based method [11]. Thisrepresentation is further clustered using a cluster-based indexing approach [1].When building a semantic classifier C, a linear SVM classifier is iteratively re-fined based on user interactions (positive and negative examples). In each roundof interaction, the resulting separating hyperplane forms k-farthest neighbourqueries posed to the cluster-based indexes. Finally, late fusion is performed onthe retrieved results, to produce the 25 top-ranked results to suggest to the user.Exquisitor Interface: The interface for building classifiers is shown in Figure 1.By hovering over a keyframe, the user can choose to view the video, submit it tothe VBS server, label it as a positive/negative example, or mark it as seen. Usingthe ‘next’ button, the user can also mark all videos as seen and get a full screenof new videos based on the current semantic classifier. Positive (green column)and negative (red column) examples are immediately used to update the model.Exquisitor at VBS 2021 3Fig. 1. Exquisitor’s interface for building semantic classifiers. See text for details.Interactive Learning and VBS: The tasks in VBS have three differentflavours: Textual Known-Item-Search (KIS) tasks present a gradually evolvingtext description matching a short video segment; Visual KIS tasks show thevideo clip sought; and Ad-hoc Video Search (AVS) tasks ask for all segmentsmatching a description. In these tasks, the aim of interactive learning is to cre-ate a classifier that is good enough to bring the correct answer(s) to the screen.For KIS tasks, a submitted result is considered as a positive example; once thecorrect result has been submitted the task is complete. For AVS tasks the pro-cess is identical, except that all videos on screen can be submitted at once usinga special button, and the process only ends once time has expired.3 Operations on Semantic Classifier RankingsTo ground the presentation, consider the two textual KIS tasks in Table 1, bothof which have a temporal component. Task T1 was solved by 6 teams during VBS2020, and was generally considered a difficult task. There are many videos withbridesmaids and brides and grooms, respectively, but in this particular video theydo not co-occur in a keyframe during the segment that was considered a solutionto the task, and hence we failed to solve this task. Task T6, on the other hand,was the only text-based task solved by all teams. The Exquisitor team solvedit efficiently during the competition by building a classifier for elevators, since(a) the elevator and the bike co-exist in the same keyframe and (b) elevatorsare rare, so the keyframe is quickly suggested for inspection. Note, however,that since there are many examples of bikes in the collection, but most of themoutdoors, building a classifier for bikes is not a productive method to solve T6.4 Khan, Jónsson, Larsen, Poulsen, Koelma, Rudinac, Worring, ZahálkaTable 1. Two example textual KIS tasks from VBS 2020.Task DescriptionT1 Seven bridesmaids in turquoise dresses walking down a street, and three stillimages of the bride and couple. The bridesmaids walk on the sidewalk towardsthe camera. The photos of the couple and bride are taken in a park.T6 Red elevator doors opening, a bike leans inside, doors closing and reopening, bikeis gone. Zoom-in on bike, zoom-out from empty elevator. The bike is silver, thetext ’ATOMZ’ is visible.Classifier Ranking Operations: The rankings obtained by two semantic clas-sifiers, C1 and C2, can be combined with a keyframe relationship operation, C1op C2, where op ∈ {∩,∪, \, −̇}. Furthermore, a temporal constraint can option-ally be added, which requires either a maximum distance between keyframes(within <frames>) or a minimum distance (after <frames>). The result of theclassifier ranking operation is a list of videos satisfying both the relationshipconstraint and optional temporal constraint. Each video is represented by a listof keyframes, annotated by the classifier(s) they appear in, and the videos areranked by an average score based on the accumulated rank of their scenes fromeach classifier and the total number of scenes.As an example, consider solving task T6 by intersection of rankings producedby semantic classifiers for bikes and elevators. A video would be returned as ananswer only if both classifiers return a scene from that video. Since the taskdescription indicates that the two elements should be close to each other, atemporal constraint of within 1, for example, would avoid videos where bikesand elevators are far apart.User Interface: Figure 2 shows the interface for classifier ranking operations.As the figure shows, the result of the merge is a list of the 10 top-rankedvideos, where each video is represented by three colour-coded keyframes. Yellowkeyframes are from C1 and blue from C2, while keyframes appearing in bothclassifiers are shown as green. The interface shows the highest ranked frame ofeach colour; if no keyframe appears in both classifiers, the third frame is thesecond highest frame from one classifier. Additionally, summary information onthe number of keyframes in the video and classifiers is shown to the left of thekeyframes.Evaluation: To evaluate the usefulness of classifier ranking operations, we at-tempted to solve the two tasks of Table 1, both by building a single classifierand by building two classifiers and intersecting their rankings. These experimentswere carried out in a calm setting, with no time limit, unlike the competitiveenvironment of VBS. Furthermore, for this evaluation, the entire task text wasconsidered. To estimate the user workload, we counted the number of interac-tions with the system, where an interaction is any action taken by the user, suchExquisitor at VBS 2021 5Fig. 2. Exquisitor’s interface for semantic classifier operations. See text for details.as labelling a keyframe as a positive example or changing to a different interfacecomponent. We chose to stop after around 75 interactions; once we reached thislimit, we considered the task to be unsolved.Table 2 summarises the results for the two tasks. Consider first T1, a difficulttask which was not solved by Exquisitor during the competition. As the tableshows, a simple intersection of the results produced by two classifiers could solvethe task. Now consider task T6, which was significantly easier. Table 2 showsthat a single classifier on ‘elevator’ is the fastest approach to solve this task, dueto the composition of the collection; this was fortunately the approach takenduring the competition. Had we chosen to focus on ‘bike’ instead, however, theresults suggest we would have failed to solve the task. Building rough classifiersfor each concept and intersecting their rankings, however, is also an efficientmethod to solve the task; since the order in which the models are built does notmatter the method is robust.Table 2. Effectiveness experiment resultsTask Models Interactions SolvedT1 ‘bridesmaid’ 76 No‘bride’ 78 No‘bridesmaid’ ∩ ‘bride’ 60 YesT6 ‘elevator’ 8 Yes‘bike’ 75 No‘elevator’ ∩ ‘bike’ 15 Yes6 Khan, Jónsson, Larsen, Poulsen, Koelma, Rudinac, Worring, Zahálka4 ConclusionsWe have outlined an extension to the Exquisitor system, supporting operationson semantic classifiers to solve VBS tasks with temporal constraints. Our pre-liminary results indicate that this new approach has significant potential, andwe look forward to testing the approach in the competitive setting.Acknowledgments: This work was supported by a PhD grant from the ITUniversity of Copenhagen and by the European Regional Development Fund(project Robotics for Industry 4.0, CZ.02.1.01/0.0/0.0/15 003/0000470).References1. Guðmundsson, G.Þ., Jónsson, B.Þ., Amsaleg, L.: A large-scale performance studyof cluster-based high-dimensional indexing. In: Proc. Int. Workshop on Very-large-scale Multimedia Corpus, Mining and Retrieval (VLS-MCM). Firenze, Italy (2010)2. Jónsson, B.Þ., Khan, O.S., Koelma, D., Rudinac, S., Worring, M., Zahálka, J.:Exquisitor at the Video Browser Showdown 2020. In: Proc. MultiMedia Modeling(MMM). Daejeon, South Korea (2020)3. Khan, O.S., Jónsson, B.Þ., Rudinac, S., Zahálka, J., Ragnarsdóttir, H., Þorleiks-dóttir, Þ., Guðmundsson, G.Þ., Amsaleg, L., Worring, M.: Interactive learning formultimedia at large. In: Proc. European Conference on IR Research (ECIR) (2020)4. Kratochvíl, M., Veselý, P., Mejzlík, F., Lokoč, J.: SOM-Hunter: Video browsingwith relevance-to-SOM feedback loop. In: Proc. MultiMedia Modeling (MMM).Daejeon, South Korea (2020)5. Lokoč, J., Kovalčík, G., Müncher, B., Schöffmann, K., Bailer, W., Gasser, R.,Vrochidis, S., Nguyen, P.A., Rujikietgumjorn, S., Barthel, K.U.: Interactive searchor sequential browsing? A detailed analysis of the Video Browser Showdown 2018.ACM TOMM 15(1) (2019)6. Lokoč, J., Kovalčík, G., Souček, T.: VIRET at Video Browser Showdown 2020. In:Proc. MultiMedia Modeling (MMM). Daejeon, South Korea (2020)7. Nguyen, P.A., Wu, J., Ngo, C.W., Francis, D., Huet, B.: VIREO @ Video BrowserShowdown 2020. In: Proc. MultiMedia Modeling (MMM). Daejeon, South Korea(2020)8. Ragnarsdóttir, H., Þorleiksdóttir, Þ., Khan, O.S., Jónsson, B.Þ., Guðmundsson,G.Þ., Zahálka, J., Rudinac, S., Amsaleg, L., Worring, M.: Exquisitor: Breaking theinteraction barrier for exploration of 100 million images. In: Proc. ACM Multime-dia. Nice, France (2019)9. Rossetto, L., Schuldt, H., Awad, G., Butt, A.A.: V3C - a research video collection.In: Proc. MultiMedia Modeling (MMM). Thessaloniki, Greece (2019)10. Sauter, L., Parian, M.A., Gasser, R., Heller, S., Rossetto, L., Schuldt, H.: Com-bining boolean and multimedia retrieval in vitrivr for large-scale video search. In:Proc. MultiMedia Modeling (MMM). Daejeon, South Korea (2020)11. Zahálka, J., Rudinac, S., Jónsson, B.Þ., Koelma, D.C., Worring, M.: Blackthorn:Large-scale interactive multimodal learning. IEEE TMM 20(3) (2018)

Exquisitor at the Video Browser Showdown 2021: Relationships Between Semantic Classifiers

https://pure.itu.dk/portal/files/86338815/Exquisitor_VBS_2021.pdf

Exquisitor at the Video Browser Showdown 2021: Relationships Between Semantic Classifiers

Abstract

Similar works

Full text

Available Versions

The IT University of Copenhagen's Repository