The Search and Hyperlinking Task was one of the Brave New Tasks at MediaEval 2012. The Task consisted of two subtasks which focused on search and linking in retrieval from a collection of semi-professional video content. These tasks followed up on research carried out within the MediaEval 2011 Rich Speech Retrieval (RSR) Task and the VideoCLEF 2009 Linking Task

Aly, Robin

Chen, Shu

Eskevich, Maria

Jones, Gareth J.F.

Larson, Martha

Ordelman, Roeland

English

Name not available

Search and Hyperlinking Task at MediaEval 2012Maria Eskevich, GarethJ.F. Jones, Shu ChenDublin City University, Ireland{meskevich,gjones}@computing.dcu.ieshu.chen4@mail.dcu.ieRobin Aly,Roeland OrdelmanUniversity of Twente,The Netherlands{r.aly, ordelman}@ewi.utwente.nlMartha LarsonDelft University of TechnologyThe Netherlandsm.a.larson@tudelft.nlABSTRACTThe Search and Hyperlinking Task was one of the Brave NewTasks at MediaEval 2012. The Task consisted of two sub-tasks which focused on search and linking in retrieval froma collection of semi-professional video content. These tasksfollowed up on research carried out within the MediaEval2011 Rich Speech Retrieval (RSR) Task and the VideoCLEF2009 Linking Task.Categories and Subject DescriptorsH.3.1 [Information Storage and Retrieval]: ContentAnalysis and Indexing—Indexing methodsGeneral TermsMeasurement, Performance1. INTRODUCTIONThe increasing amount of digital multimedia content avail-able is inspiring potential new types of user experience withthe data. The Search and Hyperlinking Task at MediaEval2012 envisioned the following scenario: a user is searchingfor a known segment in a video collection. On occasion theuser may find that the information in the segment may notbe sufficient to address their information need or they maywish to watch other related video segments. The Search andHyperlinking Task thus required Search for a known relevantsegment and Hyperlinking of related video segments.This paper describes the experimental data set providedto task participants and details of the two subtasks.2. EXPERIMENTAL DATASETThe dataset for both subtasks was the blip10000 collec-tion of semi-professional videos crawled from the Internetvideo sharing platform Blip.tv. The collection containedthe videos and user generated textual metadata availablefor each video. Participants were also provided with thefollowing additional data: two automatic speech recogni-tion (ASR) transcripts (containing the 1-best output; lat-tices, and confusion networks), automatically identified shotboundary points with extracted key frames for each shot,and automatic face and visual concept detection results.2.1 blip10000The blip10000 dataset was created by the PetaMedia NoE.Dataset creation involved collecting 15,449 Creative Com-Copyright is held by the author/owner(s).MediaEval 2012 Workshop, October 4-5, 2012, Pisa, Italymons videos from blip.tv, along with corresponding userprovided metadata and associated Twitter social networkdata (i.e., who tweeted what about a particular video).The final version of the dataset contained 14,838 videos,or episodes, comprising a total of ca. 3,260 hours of data.These episodes were separated into development and testsets, containing 5,288 videos (having a runtime of 1,135hours) and 9,550 videos (having a runtime of 2,125 hours),respectively. The proportion of videos in the developmentand test set is ca. 1:3. Compared to the original dataset,the separation between development and test set is more bal-anced, enabling the direct application of both retrieval andclassification approaches to address the task. These episodeswere taken from 2,349 different shows.2.2 Speech recognition transcriptsAudio was extracted from all videos using a combinationof the ffmpeg and sox software (sample rate = 16,000Hz,number of channels = 1). Due to the absence of the audiosignal for some videos or corruption in the files this extrac-tion resulted in 5,237 files for the development set and 7,215for the test set.Two sets of ASR transcripts were provided for this databy LIMSI/Vocapia Research1 and LIUM Research team2.The data is predominantly English, but there are alsosmall numbers of Dutch, French and Spanish shows present.LIMSI/Vocapia:.Firstly LIMSI/Vocapia system [5] used a language identifi-cation detector (LID) to automatically identify the languagespoken in the whole video along with a language confidencescore (lconf), however the LID results were not manuallychecked.Each file with a language identification score equal orgreater than 0.8 was transcribed in the detected language.The remaining files were transcribed twice, with the de-tected language as well as with the English system. Theaverage word confidence scores (tconf) were compared. Thetranscription with the higher score was chosen. For files inlanguages that do not have a transcription system, no tran-scripts were provided. This resulted in 5,238 and 7,216 filesfor development and test sets respectively.LIUM:.The LIUM sytem [9] is based on the CMU Sphinx project,and was developed to participate in the evaluation campaign1http://www.vocapia.com/25http://www-lium.univ-lemans.fr/en/content/language-and-speech-technology-lstof the International Workshop on Spoken Language Trans-lation 2011. LIUM provided an English transcription foreach audio file successfully processed, that is 5,084 from thedevelopment set and 6,879 from the test set. These resultsconsist of: (i) one-best hypotheses in NIST CTM format,(ii) word lattices in SLF (HTK) format, following a 4-gramtopology, and (iii) confusion networks, in an ATT FSM-likeformat.2.3 Video cluesIn addition to spoken content, visual descriptions of videocontent can potentially help for searching and hyperlinking.We therefore provided the participants with shot bound-aries, and the outputs of concept-based and face-based anal-ysis of the videos.For each episode, the shot boundaries were created by TUBerlin [4]. For each shot segment, with a keyframe extractedfrom the middle each shot. In total, this dataset includes ap-proximately 420,000 shots/keyframes with an average shotlength of about 30 seconds.Concept-based descriptors based on a list of 589 conceptsselected by extracting keywords from the metadata of videos.We used the on-the-fly video detector Visor, which was kindlyprovided by the University of Oxford [1]. To make the con-fidence scores of a detector comparable over multiple detec-tors, we performed a logistic regression to bring the scoresinto the interval [0:1]. We set the linear regression parame-ters to the expected value over 374 detectors on the internetarchive collection used in TRECVid 2010 [7].The appearance of faces in videos can also be helpful in-formation for search and hyperlinking. To encourage partic-ipants to use this information we provided face detectionresults, which were kindly provided by INRIA [2]. Thedata contained for possibly multiple bounding boxes in akeyframe, a confidence score stating the likelihood that thisbounding box contains a face. Additionally, the data alsocontained the visual similarity between the faces in eachbounding box.3. SEARCH SUBTASKThe Search subtask was an extension of the Rich SpeechRetrieval (RSR) Task at MediaEval 2011 [6]. Participantswere provided with a set of queries for a known-item searchtask with 30 text queries each for the development and testsets. The queries were collected via crowdsourcing usingthe Amazon MTurk platform3. Participants were requiredto make one submission for the 1-best output of each ASRtranscript version. Additionally, participants could deliverup to two other runs for each ASR transcript version.3.1 EvaluationWe used three metrics in order to evaluate task results:mean reciprocal rank (MRR), mean generalized average pre-cision (mGAP) and mean average segment precision (MASP).MRR assesses the ranking of the relevant units. mGAP [8]awards runs that not only find the relevant items earlier inthe ranked output list, but also are closer to the jump-inpoint of the relevant content. MASP [3] takes into accountthe ranking of the results and the length of both relevantand irrelevant segments that need to be listened to beforereaching the relevant item.3http://www.mturk.com/4. LINKING SUBTASKThis subtask used the ground truth of the Search subtaskas anchor videos from which links to other video should beformed. Participants were required to return a ranked listof video segments which were potentially relevant to the in-formation in this video segment (independent of the initialtextual query). Additionally they were allowed to use theirown output of the Search subtask as the video anchors forthe Linking subtask.4.1 EvaluationThere was no ground truth created for this task before-hand. Top 10 ranks of the submitted result runs of all partic-ipanta were evaluated using the Mechanical Turk platform,and the ground truth based on the manual assessment ofa combination of participants results was released for fur-ther error analysis. Once ground truth was collected, meanaverage precision (MAP) was used to compare the results.5. ACKNOWLEDGEMENTSThis work was funded by a grant under the Science Foun-dation Ireland Research Frontiers Programme 2008 GrantNo: 08/RFP/CMS1677, and by the funding from the Euro-pean Commission’s 7th Framework Programme (FP7) underAXES ICT-269980 and CUbRIK ICT-287704.6. REFERENCES[1] K. Chatfield, V. Lempitsky, A. Vedaldi, andA. Zisserman. The devil is in the details: an evaluationof recent feature encoding method. In Proceedings ofBMVC 2011, 2011.[2] R. G. Cinbis, J. Verbeek, and C. Schmid. UnsupervisedMetric Learning for Face Identification in TV Video. InProceedings of ICCV 2011, Barcelona, Spain, 2011.[3] M. Eskevich, W. Magdy, and G. J. F. Jones. Newmetrics for meaningful evaluation of informallystructured speech retrieval. In Proceedings of ECIR2012, pages 170–181, 2012.[4] P. Kelm, S. Schmiedeke, and T. Sikora. Feature-basedvideo key frame extraction for low quality videosequences. In Proceedings of WIAMIS 2009, pages25–28, 2009.[5] L. Lamel and J.-L. Gauvain. Speech processing foraudio indexing. In Advances in Natural LanguageProcessing (LNCS 5221), pages 4–15. Springer, 2008.[6] M. Larson, M. Eskevich, R. Ordelman, C. Kofler,S. Schmiedeke, and G. J. F. Jones. Overview ofMediaeval 2011 Rich Speech Retrieval task and genretagging task. In Proceedings of the MediaEval 2011Workshop.[7] P. Over, G. Awad, J. Fiscus, B. Antonishek, M. Michel,A. Smeaton, W. Kraaij, and G. Que´not. Trecvid2010–an overview of the goals, tasks, data, evaluationmechanisms, and metrics. 2011.[8] P. Pecina, P. Hoffmannova, G. J. F. Jones, Y. Zhang,and D. W. Oard. Overview of the CLEF 2007cross-language speech retrieval track. In Proceedings ofCLEF 2007, pages 674–686. Springer, 2007.[9] A. Rousseau, F. Bougares, P. Dele´glise, H. Schwenk,and Y. Este`v. LIUM’s systems for the IWSLT 2011Speech Translation Tasks. In Proceedings of IWSLT2011, San Francisco, USA, 2011.

Search and hyperlinking task at MediaEval 2012

DCU Online Research Access Service

Feature-based video key frame extraction for low quality video sequences.

LIUM's systems for the IWSLT 2011 Speech Translation Tasks.

New metrics for meaningful evaluation of informally structured speech retrieval.

Overview of Mediaeval

Overview of the CLEF

Speech processing for audio indexing.

The devil is in the details: an evaluation of recent feature encoding method.

Trecvid 2010{an overview of the goals, tasks, data, evaluation mechanisms, and metrics.

Unsupervised Metric Learning for Face Identi in TV Video.

The Search and Hyperlinking Task was one of the Brave New Tasks at MediaEval 2012. The Task consisted of two sub- tasks which focused on search and linking in retrieval from a collection of semi-professional video content. These tasks followed up on research carried out within the MediaEval 2011 Rich Speech Retrieval (RSR) Task and the VideoCLEF 2009 Linking Task

University of Twente Research Information

Search and Hyperlinking Task at MediaEval 2012Maria Eskevich, GarethJ.F. Jones, Shu ChenDublin City University, Ireland{meskevich,gjones}@computing.dcu.ieshu.chen4@mail.dcu.ieRobin Aly,Roeland OrdelmanUniversity of Twente,The Netherlands{r.aly, ordelman}@ewi.utwente.nlMartha LarsonDelft University of TechnologyThe Netherlandsm.a.larson@tudelft.nlABSTRACTThe Search and Hyperlinking Task was one of the Brave NewTasks at MediaEval 2012. The Task consisted of two sub-tasks which focused on search and linking in retrieval froma collection of semi-professional video content. These tasksfollowed up on research carried out within the MediaEval2011 Rich Speech Retrieval (RSR) Task and the VideoCLEF2009 Linking Task.Categories and Subject DescriptorsH.3.1 [Information Storage and Retrieval]: ContentAnalysis and Indexing—Indexing methodsGeneral TermsMeasurement, Performance1. INTRODUCTIONThe increasing amount of digital multimedia content avail-able is inspiring potential new types of user experience withthe data. The Search and Hyperlinking Task at MediaEval2012 envisioned the following scenario: a user is searchingfor a known segment in a video collection. On occasion theuser may find that the information in the segment may notbe sufficient to address their information need or they maywish to watch other related video segments. The Search andHyperlinking Task thus required Search for a known relevantsegment and Hyperlinking of related video segments.This paper describes the experimental data set providedto task participants and details of the two subtasks.2. EXPERIMENTAL DATASETThe dataset for both subtasks was the blip10000 collec-tion of semi-professional videos crawled from the Internetvideo sharing platform Blip.tv. The collection containedthe videos and user generated textual metadata availablefor each video. Participants were also provided with thefollowing additional data: two automatic speech recogni-tion (ASR) transcripts (containing the 1-best output; lat-tices, and confusion networks), automatically identified shotboundary points with extracted key frames for each shot,and automatic face and visual concept detection results.2.1 blip10000The blip10000 dataset was created by the PetaMedia NoE.Dataset creation involved collecting 15,449 Creative Com-Copyright is held by the author/owner(s).MediaEval 2012 Workshop, October 4-5, 2012, Pisa, Italymons videos from blip.tv, along with corresponding userprovided metadata and associated Twitter social networkdata (i.e., who tweeted what about a particular video).The final version of the dataset contained 14,838 videos,or episodes, comprising a total of ca. 3,260 hours of data.These episodes were separated into development and testsets, containing 5,288 videos (having a runtime of 1,135hours) and 9,550 videos (having a runtime of 2,125 hours),respectively. The proportion of videos in the developmentand test set is ca. 1:3. Compared to the original dataset,the separation between development and test set is more bal-anced, enabling the direct application of both retrieval andclassification approaches to address the task. These episodeswere taken from 2,349 different shows.2.2 Speech recognition transcriptsAudio was extracted from all videos using a combinationof the ffmpeg and sox software (sample rate = 16,000Hz,number of channels = 1). Due to the absence of the audiosignal for some videos or corruption in the files this extrac-tion resulted in 5,237 files for the development set and 7,215for the test set.Two sets of ASR transcripts were provided for this databy LIMSI/Vocapia Research1 and LIUM Research team2.The data is predominantly English, but there are alsosmall numbers of Dutch, French and Spanish shows present.LIMSI/Vocapia:.Firstly LIMSI/Vocapia system [5] used a language identifi-cation detector (LID) to automatically identify the languagespoken in the whole video along with a language confidencescore (lconf), however the LID results were not manuallychecked.Each file with a language identification score equal orgreater than 0.8 was transcribed in the detected language.The remaining files were transcribed twice, with the de-tected language as well as with the English system. Theaverage word confidence scores (tconf) were compared. Thetranscription with the higher score was chosen. For files inlanguages that do not have a transcription system, no tran-scripts were provided. This resulted in 5,238 and 7,216 filesfor development and test sets respectively.LIUM:.The LIUM sytem [9] is based on the CMU Sphinx project,and was developed to participate in the evaluation campaign1http://www.vocapia.com/25http://www-lium.univ-lemans.fr/en/content/language-and-speech-technology-lstof the International Workshop on Spoken Language Trans-lation 2011. LIUM provided an English transcription foreach audio file successfully processed, that is 5,084 from thedevelopment set and 6,879 from the test set. These resultsconsist of: (i) one-best hypotheses in NIST CTM format,(ii) word lattices in SLF (HTK) format, following a 4-gramtopology, and (iii) confusion networks, in an ATT FSM-likeformat.2.3 Video cluesIn addition to spoken content, visual descriptions of videocontent can potentially help for searching and hyperlinking.We therefore provided the participants with shot bound-aries, and the outputs of concept-based and face-based anal-ysis of the videos.For each episode, the shot boundaries were created by TUBerlin [4]. For each shot segment, with a keyframe extractedfrom the middle each shot. In total, this dataset includes ap-proximately 420,000 shots/keyframes with an average shotlength of about 30 seconds.Concept-based descriptors based on a list of 589 conceptsselected by extracting keywords from the metadata of videos.We used the on-the-fly video detector Visor, which was kindlyprovided by the University of Oxford [1]. To make the con-fidence scores of a detector comparable over multiple detec-tors, we performed a logistic regression to bring the scoresinto the interval [0:1]. We set the linear regression parame-ters to the expected value over 374 detectors on the internetarchive collection used in TRECVid 2010 [7].The appearance of faces in videos can also be helpful in-formation for search and hyperlinking. To encourage partic-ipants to use this information we provided face detectionresults, which were kindly provided by INRIA [2]. Thedata contained for possibly multiple bounding boxes in akeyframe, a confidence score stating the likelihood that thisbounding box contains a face. Additionally, the data alsocontained the visual similarity between the faces in eachbounding box.3. SEARCH SUBTASKThe Search subtask was an extension of the Rich SpeechRetrieval (RSR) Task at MediaEval 2011 [6]. Participantswere provided with a set of queries for a known-item searchtask with 30 text queries each for the development and testsets. The queries were collected via crowdsourcing usingthe Amazon MTurk platform3. Participants were requiredto make one submission for the 1-best output of each ASRtranscript version. Additionally, participants could deliverup to two other runs for each ASR transcript version.3.1 EvaluationWe used three metrics in order to evaluate task results:mean reciprocal rank (MRR), mean generalized average pre-cision (mGAP) and mean average segment precision (MASP).MRR assesses the ranking of the relevant units. mGAP [8]awards runs that not only find the relevant items earlier inthe ranked output list, but also are closer to the jump-inpoint of the relevant content. MASP [3] takes into accountthe ranking of the results and the length of both relevantand irrelevant segments that need to be listened to beforereaching the relevant item.3http://www.mturk.com/4. LINKING SUBTASKThis subtask used the ground truth of the Search subtaskas anchor videos from which links to other video should beformed. Participants were required to return a ranked listof video segments which were potentially relevant to the in-formation in this video segment (independent of the initialtextual query). Additionally they were allowed to use theirown output of the Search subtask as the video anchors forthe Linking subtask.4.1 EvaluationThere was no ground truth created for this task before-hand. Top 10 ranks of the submitted result runs of all partic-ipanta were evaluated using the Mechanical Turk platform,and the ground truth based on the manual assessment ofa combination of participants results was released for fur-ther error analysis. Once ground truth was collected, meanaverage precision (MAP) was used to compare the results.5. ACKNOWLEDGEMENTSThis work was funded by a grant under the Science Foun-dation Ireland Research Frontiers Programme 2008 GrantNo: 08/RFP/CMS1677, and by the funding from the Euro-pean Commission’s 7th Framework Programme (FP7) underAXES ICT-269980 and CUbRIK ICT-287704.6. REFERENCES[1] K. Chatfield, V. Lempitsky, A. Vedaldi, andA. Zisserman. The devil is in the details: an evaluationof recent feature encoding method. In Proceedings ofBMVC 2011, 2011.[2] R. G. Cinbis, J. Verbeek, and C. Schmid. UnsupervisedMetric Learning for Face Identification in TV Video. InProceedings of ICCV 2011, Barcelona, Spain, 2011.[3] M. Eskevich, W. Magdy, and G. J. F. Jones. Newmetrics for meaningful evaluation of informallystructured speech retrieval. In Proceedings of ECIR2012, pages 170–181, 2012.[4] P. Kelm, S. Schmiedeke, and T. Sikora. Feature-basedvideo key frame extraction for low quality videosequences. In Proceedings of WIAMIS 2009, pages25–28, 2009.[5] L. Lamel and J.-L. Gauvain. Speech processing foraudio indexing. In Advances in Natural LanguageProcessing (LNCS 5221), pages 4–15. Springer, 2008.[6] M. Larson, M. Eskevich, R. Ordelman, C. Kofler,S. Schmiedeke, and G. J. F. Jones. Overview ofMediaeval 2011 Rich Speech Retrieval task and genretagging task. In Proceedings of the MediaEval 2011Workshop.[7] P. Over, G. Awad, J. Fiscus, B. Antonishek, M. Michel,A. Smeaton, W. Kraaij, and G. Quénot. Trecvid2010–an overview of the goals, tasks, data, evaluationmechanisms, and metrics. 2011.[8] P. Pecina, P. Hoffmannova, G. J. F. Jones, Y. Zhang,and D. W. Oard. Overview of the CLEF 2007cross-language speech retrieval track. In Proceedings ofCLEF 2007, pages 674–686. Springer, 2007.[9] A. Rousseau, F. Bougares, P. Deléglise, H. Schwenk,and Y. Estèv. LIUM’s systems for the IWSLT 2011Speech Translation Tasks. In Proceedings of IWSLT2011, San Francisco, USA, 2011.

Search and Hyperlinking Task at MediaEval 2012

Ordelman, Roeland J.F.

NARCIS 

https://ris.utwente.nl/ws/files/5341549/mediaeval2012_submission_14.pdf

Search and hyperlinking task at MediaEval 2012

Abstract

Similar works

Full text

Available Versions

Name not available

DCU Online Research Access Service

University of Twente Research Information

NARCIS