With an increasing amount of video data in our daily life, the need for content-based search in videos increases as well. Though a lot of research has been spent on video retrieval tools and methods which allow for automatic search in videos through content-based queries, still the performance of automatic video retrieval is far from optimal. In this tutorial we discussed (i) proposed solutions for improved video content navigation, (ii) typical interaction of content-based querying features, and (iii) advanced video content visualization methods. Moreover, we discussed interactive video search systems and ways to evaluate their performance

Hopfgartner, Frank

Schoeffmann, Klaus

English

Enlighten

Interactive Video SearchKlaus SchoeffmannKlagenfurt UniversityUniversitätsstr. 65-67,A-9020 Klagenfurt, Austriaks@itec.aau.atFrank HopfgartnerUniversity of Glasgow11 University Gardens, Glasgow G128QH, UKfrank.hopfgartner@glasgow.ac.ukABSTRACTWith an increasing amount of video data in our daily life,the need for content-based search in videos increases as well.Though a lot of research has been spent on video retrievaltools and methods which allow for automatic search in videosthrough content-based queries, still the performance of auto-matic video retrieval is far from optimal. In this tutorial wediscussed (i) proposed solutions for improved video contentnavigation, (ii) typical interaction of content-based query-ing features, and (iii) advanced video content visualizationmethods. Moreover, we discussed interactive video searchsystems and ways to evaluate their performance.Categories and Subject DescriptorsH.4 [Information Systems Applications]: MiscellaneousGeneral TermsExperimentationKeywordsInteractive Video Retrieval; Search; Tutorial1. INTRODUCTIONOver the last two decades, there has been a lot of re-search on content-based video retrieval to solve the task offinding the proper scene of interest in a large video archive.However, there are still well-known limitations with videoretrieval applications. First, there is the usability gap. Auser is often not able to express his/her needs and thoughtsthrough text, a challenge that is already apparent for animage (“An image is worth a thousand words.”) but muchworse for a video segment consisting of many images. Ad-ditionally, the issue of polysemy is a challenging problemin text-based search, which can only be partially solvedthrough relevance feedback. Similarly, an automatic re-trieval tool cannot easily determine the user-dependent rel-ative importance (i.e., weight) of a query term and hencePermission to make digital or hard copies of part or all of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contactthe Owner/Author. Copyright is held by the owner/author(s).MM’15, October 26–30, 2015, Brisbane, AustraliaACM 978-1-4503-3459-4/15/10.DOI: http://dx.doi.org/10.1145/2733373.2807417.often returns too many irrelevant results. Finally, there isanother challenging problem that affects the content-basedindexing phase, namely the semantic gap, which is the dis-crepancy between the semantics a user can derive from animage and the information a computer can extract from pixelvalues [15]. The semantic gap seriously limits the achievableperformance of visual information retrieval tools.There is consensus in the multimedia community thatcontent-based retrieval approaches should have a strongerfocus on the human behind the retrieval application [4, 16].Instead of making small improvements in the field of content-based indexing and retrieval, video search tools should aimat more intense integration of the user into the search pro-cess, focusing on interactive video search [2, 3, 7, 8] ratherthan automatic querying. Interactive video search followsthis idea of intense user integration with advanced contentinteraction and provides a powerful alternative to the com-mon video retrieval approach [12]. It is known as the processof interactive video content exploration with the help of con-tent navigation [9], summarization [1], on-demand querying[14], and browsing of querying results or filtered content [5].In contrast to typical video retrieval, such interactive videosearch tools give more control to the user and provide flexi-ble search and browsing features instead of focusing on thequery-and-browse-results approach. Hence, even if the per-formance of content analysis is not optimal, there is a chancethat a smart user could compensate shortcomings throughclever use of available features.Interactive video search tools support directed and undi-rected search scenarios. In the first scenario, users have aclear information need and want to find a specific targetsegment in the video (e.g., the weather forecast in a newsshow); such a search is also known as known-item search ortarget search. In the second scenario, users have no concretesearch goal but want to explore the content in order to learnor find something interesting (e.g., a violent scene in surveil-lance videos); such a search scenario is known as exploratorysearch. Users need exploratory search when they cannot suf-ficiently describe the desired content, e.g., through a queryin a video retrieval tool, or when they just want to browseinstead of search (i.e., without a concrete desired scene).Over the years many tools for interactive video search (of-ten known as video browsing or video exploration tools) havebeen proposed in the literature (see [12, 13] for a survey),and it has been shown that these tools can effectively helpusers find desired content in videos (e.g., [1, 14]). Some ofthese tools combine sophisticated content analysis methodscontrolled by the user for their personal needs [10]. Someothers provide rather simple content navigation features butgive the users more interactivity to allow them to effectivelytake advantage of their knowledge about the content and thecontent structure [11]. Interestingly, it has been shown thattools of the latter kind can even outperform tools of the firstkind for some search tasks [10].2. LEARNING OBJECTIVEThis tutorial gave a broad overview of solutions in twoareas: (1) video retrieval, and (2) interactive video search.We discussed state-of-the-art approaches, but also the cur-rent shortcomings and outlined important research tasks forthe future of multimedia search and retrieval. Our discus-sion also covered tools and methods that originate fromthe Human-Computer-Interaction (HCI) community, whichmany people in the multimedia community might not beaware of yet. The tutorial provided the basis to elevate re-search on video retrieval to the next level.Target audience were researchers and practitioners in thefield of multimedia analysis and information access. Theparticipants learned best practice on how to interact withmultimedia content and should be able to apply this knowl-edge in their own project upon completion of the tutorial.Following an introduction of the subject matter, we high-lighted the need for interactive video search and outlinedwell-known issues and current research areas. Then, we pre-sented different graphical user interfaces that are designed toaddress these challenges. We started by presenting commer-cial and academic video search systems that represent thestate-of-the-art in video search. In order to illustrate thelimitations of video search, we began with an introductionto video content analysis. In particular, we focused on thesegmentation of videos into different units of retrieval (i.e.,video shots and semantically coherent scenes), argued forthe selection of appropriate keyframes and explained how toindex these video materials. Then, we overviewed methodsfor video content presentation, namely abstraction, summa-rization, and visualization of video content. After that, weintroduced different methods to interact with video content.We showcased the advantages of browsing and explorationof video content, methods to navigate through the content,querying and sketching interfaces. Following this session,the participants learned how to evaluate the performance ofvideo search engines. We introduced the de-facto standardevaluation protocol that is applied for scientific performanceevaluation. Furthermore, we introduced popular Academicevaluation campaigns, namely the Known-Item search taskpromoted by TRECVid and the Video Browser Showdown.As an outlook in upcoming research challenges, we intro-duced a special case of multimedia search, namely access tovisual lifelogs [6]. With the ever increasing amount of wear-able devices that are released nowadays, people have theopportunity to capture vast amount of heterogeneous data(e.g., images, sensor readings, videos) that depict aspectsof their personal lives. In the last part of the tutorial, weintroduced approaches to access and display these “lifelogs”.AcknowledgmentsThis work was funded by the Austrian Federal Ministryfor Transport, Innovation and Technology (bmvit), AustrianScience Fund (FWF): TRP 273-N15, and the ERDF.3. REFERENCES[1] B. Adams, S. Greenhill, and S. Venkatesh. Towards avideo browser for the digital native. In ICMEW’12,pages 127–132, July 2012.[2] M. Christel, C. Huang, N. Moraveji, andN. Papernick. Exploiting multiple modalities forinteractive video retrieval. In ICASSP’04, volume 3,pages iii–1032, May 2004.[3] M. G. Christel and R. Yan. Merging storyboardstrategies and automatic retrieval for improvinginteractive video search. In CIVR’07, pages 486–493,New York, NY, USA, 2007. ACM.[4] R. Datta, D. Joshi, J. Li, and J. Z. Wang. Imageretrieval: Ideas, influences, and trends of the new age.ACM Computing Surveys, 40(2):5:1–5:60, May 2008.[5] A. Girgensohn, F. Shipman, and L. Wilcox. Adaptiveclustering and interactive visualizations to support theselection of video clips. In ICMR ’11, pages 34:1–34:8,New York, NY, USA, 2011. ACM.[6] C. Gurrin, A. F. Smeaton, and A. R. Doherty.Lifelogging: Personal big data. Foundations andTrends in Information Retrieval, 8(1):1–125, 2014.[7] F. Hopfgartner. Understanding Video Retrieval. VDMVerlag, 2007.[8] F. Hopfgartner, J. Urban, R. Villa, and J. M. Jose.Simulated testing of an adaptive multimediainformation retrieval system. In CBMI’07, pages328–335, 2007.[9] J. Huber, J. Steimle, and M. Mu¨hlha¨user. Towardmore efficient user interfaces for mobile videobrowsing: An in-depth exploration of the design space.In MM’10, pages 341–350, New York, NY, USA, 2010.[10] K. Schoeffmann, D. Ahlstro¨m, W. Bailer,C. Cobaˆrzan, F. Hopfgartner, K. McGuinness,C. Gurrin, C. Frisson, D.-D. Le, M. Del Fabro, H. Bai,and W. Weiss. The video browser showdown: a liveevaluation of interactive video search tools. IJMIR,3(2):113–127, 2014.[11] K. Schoeffmann and L. Boeszoermenyi. Videobrowsing using interactive navigation summaries. InCBMI’09, pages 243–248, Chania, Crete, June 2009.[12] K. Schoeffmann, F. Hopfgartner, O. Marques,L. Boeszoermenyi, and J. M. Jose. Video browsinginterfaces and applications: a review. SPIE Reviews,1(1):1–35(018004), 2010.[13] K. Schoeffmann, J. Huber, and M. A. Hudelist. Videointeraction tools: A survey of recent work. ACMComputing Surveys, pages 1–36, 2015. Accepted forpublication.[14] K. Schoeffmann, M. Taschwer, and L. Boeszoermenyi.The video explorer: A tool for navigation andsearching within a single video based on fast contentanalysis. In MMSys’10, pages 247–258. ACM, 2010.[15] A. W. M. Smeulders, M. Worring, S. Santini,A. Gupta, and R. Jain. Content-based image retrievalat the end of the early years. TPAMI,22(12):1349–1380, Dec 2000.[16] M. Worring, P. Sajda, S. Santini, D. A. Shamma,A. F. Smeaton, and Q. Yang. Where is the user inmultimedia retrieval? IEEE MultiMedia, 19(4):6–10,2012.

Interactive Video Search

http://eprints.gla.ac.uk/108221/7/108221.pdf

Interactive Video Search

Abstract

Similar works

Full text

Available Versions

Enlighten