We describe Dublin City University (DCU)'s participation in the Hyperlinking sub-task of the MediaEval 2012 Search and Hyperlinking Task. Our strategy involves combining textual metadata, automatic speech recognition (ASR) transcripts, and visual content analysis to create anchor summaries for each video segment available for linking. Two categories of fusion strategy, score-based and rank-based methods, were used to combine scores from different modalities to produce potential inter-item links

Chen, Shu

Jones, Gareth J.F.

O'Connor, Noel E.

English

DCU Online Research Access Service

DCU Linking Runs at MediaEval 2012: Search andHyperlinking TaskShu ChenCLARITY: Centre for SensorWeb Technologies / CNGLDublin City UniversityDublin 9, Irelandshu.chen4@mail.dcu.ieGareth J. F. JonesCNGL, School of ComputingDublin City UniversityDublin 9, Irelandgjones@computing.dcu.ieNoel E. O’ConnorCLARITY: Centre for SensorWeb TechnologiesDublin City UniversityDublin 9, IrelandNoel.OConnor@dcu.ieABSTRACTWe describe Dublin City University (DCU)’s participation inthe Hyperlinking sub-task of the MediaEval 2012 Search andHyperlinking Task. Our strategy involves combining textualmetadata, automatic speech recognition (ASR) transcripts,and visual content analysis to create anchor summaries foreach video segment available for linking. Two categories offusion strategy, score-based and rank-based methods, wereused to combine scores from different modalities to producepotential inter-item links.KeywordsHyperlinking, multimedia search, data fusion, informationretrieval1. INTRODUCTIONSearch and Hyperlinking was offered as a Brave New Taskat MediaEval 2012 as an incubator for emerging researchchallenges. This paper presents details of DCU’s partici-pation in the multimedia hyperlinking task. We describeour automatic multimedia hyperlinking construction strat-egy based on fusion of metadata, automatic speech recog-nition (ASR) transcripts and visual content. The reminderof this paper presents the overall system design, data fusionstrategy and the results of the evaluation experiment withanalysis of the results achieved.The paper is organized as following: Section 2 describesprevious research on multimedia content analysis and datafusion evaluation, Section 3 describes our automatic hyper-link generation strategy, including multimedia content ex-traction and data fusion between different modalities, Sec-tion 4 gives our experimental results, and Section 5 con-cludes the paper.2. RELATEDWORKThere has been extensive research in multimedia contentanalysis in recent years. Work of particular relevance tothe MediaEval 2012 Hyperlinking task includes the VideoGoogle system based on content-based visual search in whichthe content is described at the level of keyframes [6]. Thevisual features of each keyframe were represented using theCopyright is held by the author/owner(s).MediaEval 2012 Workshop, October 4-5, 2012, Pisa, Italyscale-invariant feature transform (SIFT). All extracted vi-sual features were then transformed into visual keywords byapplying a k-means clustering algorithm based on the ex-tracted SIFT descriptors. Finally, a tf-idf algorithm calcu-lated a document score to produce the ranked output results.The AXES project is similarly a content-based video searchengine [1] which combines text data with visual content toimplement a multimedia retrieval system.Data fusion is an effective method to combine results frommultiple information retrieval systems. A significant chal-lenge for data fusion is score normalization to achieve themost effective combination. A number of methods are eval-uated in [8] and [7], which provide detailed guidelines un-derpinning the design of our data fusion strategy.3. DESCRIPTION OF LINK TASKThe dataset for the Linking Task is described in [2]. Atotal of 30 queries were provided where each one is associ-ated with an anchor segment in the video. The objective ofthe task was to link each anchor segment to related videomaterial in the test collection. To identify a linked segment,we extracted textual information from metadata and auto-matic speech recognition (ASR) transcripts for the anchorsegment as a summary. Moreover, SIFT visual features wereextracted for keyframes for each video shot.3.1 Text ProcessingShots extracted automatically from the videos were usedas our units for linking. For each shot we created a summaryusing both the video metadata, including title and shortdescription, and ASR transcripts within the correspondingtime interval. ASR transcripts were provided by LIUM [5],where one-best hypotheses was used, and LIMSI [3]. ApacheLucene is used to index and search the text information 1.3.2 Visual Content AnalysisVisual content extraction followed the strategy introducedin [6]. We used VLFeat (version 0.9.14) to implement theSIFT algorithm to extract the visual features of each keyframe.A total of 26,0931 keyframes in total were provided with thedataset. We selected 9,034 of them as training set, whichwere randomly picked from the keyframe set of each video.After extracting SIFT descriptors from the training set, ak-means algorithm clustered the descriptor vectors to createvisual words. The number of cluster centres, which experi-mentally should not be over 1000, as a limitation of Matlab1http://lucene.apache.org/processing capacity, was set to 500. The weight vector ofeach keyframe was calculated based on visual words andits own SIFT descriptor. We used ffmpeg to extract thekeyframe at the media time of the anchor segment intervalas a query image. Finally, a tf-idf algorithm was used tocalculate the score for ranking of potential link videos.3.3 Data fusionData fusion was used to calculate final scores based onthe results from text processing and visual content analysis.We used linear combination data fusion to combine themtogether. Our fusion investigation and experiment involvedtwo fusion approaches: score-based and rank-based.The data fusion algorithm first normalizes scores from dif-ferent modalities for all multimedia documents. For score-based data fusion, we used the following formula for normal-ization:scorenormalized =si − sminsmax − smin (1)smin and smax are used to denote the minimal and maximalscore over all multimedia documents respectively, while siis the raw score of document at rank i. The normalizationformula in rank-based method follows the binary logistic re-gression model introduced in [7]:scorenormalized =11 + e−a−b ln ri(2)Here ri is the rank of document i in different modalities. aand b are two parameters whose details are described in [4].In our experiment, we provide a simple solution on settingthe value of coefficients as a = 1, and b = −1.The linear combination fusion formula was the same forboth the score-based method and rank-based method, asfollows:sf (di) = w1 · st(di) + w2 · sv(di) (3)In equation 3, st(di) means the normalized score of docu-ment d at rank i from text processing result, while sv(di)means the normalization result of visual content analysis.The coefficients w1 and w2 are both set as 0.5, meaning thattext and visual content will equally influence fusion result.4. EVALUATION RESULTSTable 1 shows the details of our 6 submitted runs, andTable 2 shows mean average precision (MAP) value of eachrun. Our best result of 0.276 was obtained for run 2. Wecan observe that: (1) the LIUM transcripts perform betterthan the LIMSI transcripts, (2) rank-based fusion achievesbetter results. The reason is only top 10 videos are involvedin evaluation, and a rank-based fusion strategy based onnon-linear formula gives a much higher score to the videosranked relatively higher.5. CONCLUSIONSThis paper presented details of DCU’s participation inthe Linking task of MediaEval 2012. We used two differentfeatures to identify anchors to produce potential inter-itemlinks: textual content from metadata and ASR transcripts,and content-based visual features. Furthermore, two fusionmethods, rank-based and score-based were investigated. Ac-cording to our evaluation results, LIUM transcripts give bet-ter results than LIMSI transcripts and the rank-based fusionRun ID Features Used Acronymrun1 LIMSI, metadata MSI Trun2 LIUM, metadata UM Trun3 LIMSI, metadata, visual, score MSI T V Srun4 LIUM, metadata, visual, score UM T V Srun5 LIMSI, metadata visual, rank MSI T V Rrun6 LIUM, metadata visual, rank UM T V RTable 1: Run DetailsRUN MAPMSI T 0.197UM T 0.276MSI T V S 0.176UM T V S 0.223MSI T V R 0.187UM T V R 0.234Table 2: Evaluation Results (MAP)strategy performs better than score-based fusion. In our fu-ture work, we will focus on how to utilize existing computervision algorithms to improve link generation performance.We will seek to improve current visual content analysis pro-gram based on SIFT algorithm. Moreover, we believe theresult of text indexing can be improved by involving namedentity recognition process.6. ACKNOWLEDGMENTSThis work is funded by the European Commission’s Sev-enth Framework Programme (FP7) as part of the AXESproject (ICT-269980).7. REFERENCES[1] S. Chen, K. McGuinness, R. Aly, N. O’Connor, andF. de Jong. The AXES-lite video search engine. InProceedings of WIAMIS 2012, pages 1–4, May 2012.[2] M. Eskevich, G. J. F. Jones, S. Chen, R. Aly,R. Ordelman, and M. Larson. Search and HyperlinkingTask at MediaEval 2012. In MediaEval 2012 Workshop,2012.[3] L. Lamel and J.-L. Gauvain. Speech processing foraudio indexing. In Advances in Natural LanguageProcessing (LNCS 5221), pages 4–15. Springer-Verlag,2008.[4] A. Le Calve´ and J. Savoy. Database merging strategybased on logistic regression. Information ProcessingManagement, 36(3):341–359, 2000.[5] A. Rousseau, F. Bougares, P. Del´l ↪eglise, H. Schwenk,and Y. Est´l´lv. LIUM’s systems for the IWSLT 2011Speech Translation Tasks. In Proceedings of IWSLT2011I, 2011.[6] J. Sivic and A. Zisserman. Video Google: a textretrieval approach to object matching in videos. InProceedings of ICCV 2003, pages 1470–1477, 2003.[7] S. Wu. Linear combination of component results ininformation retrieval. Data Knowledge Engineering,71(1):114–126, 2012.[8] S. Wu, F. Crestani, and Y. Bi. Evaluating scorenormalization methods in data fusion. In Proceedings ofAIRS 2006, pages 642–648. Springer-Verlag, 2006.

DCU linking runs at MediaEval 2012: search and hyperlinking task

Abstract

Similar works

Full text

Available Versions

DCU Online Research Access Service