We describe our runs and results for the fixed label Wild

Wild Web Tagging Task at MediaEval 2010. Our experiments indicate that including all words in the ASR transcripts of the document set results in better labeling accuracy than restricting the index to only words with recognition confidence above a fixed level. Additionally our results

show that tagging accuracy can be improved by incorporating additional metadata describing the documents where it is available

Gyarmati, Ágnes

Jones, Gareth J.F.

English

We describe our runs and results for the fixed label Wild
Wild Web Tagging Task at MediaEval 2010. Our experiments indicate that including all words in the ASR transcripts of the document set results in better labeling accuracy than restricting the index to only words with recognition confidence above a fixed level. Additionally our results
show that tagging accuracy can be improved by incorporating additional metadata describing the documents where it is available

Irish Universities

DCU at MediaEval 2010 – Tagging task WildWildWeb

DCU Online Research Access Service

DCU at MediaEval 2010 – Tagging Task WildWildWebÁgnes GyarmatiCentre for Digital Video ProcessingDublin City UniversityDublin 9, Irelandagyarmati@computing.dcu.ieGareth J. F. JonesCentre for Digital Video ProcessingDublin City UniversityDublin 9, Irelandgjones@computing.dcu.ieABSTRACTWe describe our runs and results for the fixed label WildWild Web Tagging Task at MediaEval 2010. Our experi-ments indicate that including all words in the ASR tran-scripts of the document set results in better labeling accu-racy than restricting the index to only words with recogni-tion confidence above a fixed level. Additionally our resultsshow that tagging accuracy can be improved by incorporat-ing additional metadata describing the documents where itis available.Categories and Subject DescriptorsH.3 [Information Storage and Retrieval]: H.3.1 Con-tent Analysis and Indexing; H.3.3 Information Search andRetrieval; H.3.4 Systems and Software; H.3.7 Digital Li-braries; I.2 [Artificial Intelligence]: I.2.7 Natural Lan-guage Processing—Speech recognition and synthesisGeneral TermsMeasurement, ExperimentationKeywordsinformation retrieval, automatic speech recognition, text tag-ging1. INTRODUCTIONThe Tagging Task “Wild Wild Web” held as part of Me-diaEval 2010 required participants to automatically predicttag annotations for a set of internet videos from blip.tv.The videos are accompanied by automatic speech recogni-tion (ASR) generated transcripts provided by Vecsys Re-search1 and LIMSI-CNRS2 and various other metadata (e.g.titles, descriptions, twitter messages related to the videos).The topic, genre, quality and language of the videos vary,hence the name of the task ‘ ‘Wild Wild Web.” Tags orig-inally assigned manually by the uploaders are used as thebasis of the evaluation of the automatic tagging carried outin this task. Since there is no tagging convention in usageat blip.tv, there may be inconsistencies in the assignment1http://www.vecsysresearch.fr/2http://www.limsi.fr/Copyright is held by the author/owner(s).MediaEval 2010 Workshop, October 24, 2010, Pisa, Italyof these manual tags, potentially making it challenging toemulate the behaviour of manual taggers.Two flavours of “Wild Wild Web” tagging task were of-fered at MediaEval 2010, with participants being allowedto submit up to 5 runs based on variants of their taggingapproach: Closed-set tagging: a list of possible tags wasprovided with the data, and the task was to automaticallypredict assignment of these on the test video data. Open-set tagging: by disregarding the tag list provided, the taskwas to select and assign tags to videos without vocabularyconstraint.Tagging is usually considered as a categorization task, butit can be approached as an information retrieval (IR) task.Adopting the IR approach has been shown to have potentialfor tasks such as this, for example at VideoCLEF 2009 [2].In addition to words in the most likely transcript fromthe ASR transcript, each word hypothesis included a confi-dence score which was available for use. One might expectwords with higher confidence scores to lead to more reliableor accurate assignment of tags. In related work Zechnerand Waibel demonstrated positive effects using confidenceinformation in summarization [4], however in IR using thisinformation has shown minimal impact on effectiveness [3].2. SYSTEM DESCRIPTIONFor our participation in MediaEval 2010 we decided totackle the “closed set” variant of the task. We approachedtagging as an IR task, where the documents to be retrievedwere the videos (i.e. their transcripts, optionally combinedwith some metadata: titles and descriptions). Queries wereformed using the closed set of possible tags.We used the Indri model of the open source Lemur Toolkit3for indexing and retrieving. English texts were stemmed us-ing Lemur’s own built-in stemmer, while Dutch, French andSpanish texts were stemmed using Oleander’s4 implemen-tation of Snowball’s5 Porter stemmer algorithm for variouslanguages. We used stopword lists provided by Snowball forall languages involved in this task. No preprocessing (stem-ming or stopping) was done if the language was intentionallyconsidered as “unknown”.The same basic searching method was applied for eachrun. Our four submitted runs used the transcripts incor-porating varied thresholds on the word confidence scoresextracted from the transcripts and did not distinguish be-3http://www.lemurproject.org/4http://sourceforge.net/projects/porterstemmers/5http://snowball.tartarus.org/tween the document languages. Additional runs incorpo-rated metadata fields in the indexed collection and indexedthe documents differently according to their language.3. RUNCONFIGURATIONSANDRESULTS3.1 Submitted RunsWe submitted four runs for the MediaEval 2010 “WildWild Web” fixed vocabulary tagging task. These made nodistinction between the languages of the transcripts, andthey were all combined into a single search index. Sincethe documents’ languages were considered as “unknown”, nostemming and stopping was performed on either the tran-scripts or the query labels. The difference between the sub-mitted runs was the text used for indexing: whether all thewords of the transcripts were used (Run 1), or only wordswith a confidence score better than a fixed threshold of 70%,80% and 90% for Run 2, Run 3, and Run 4 respectively.Each tag was assigned to a maximum of 20 videos, by se-lecting the top 20 relevant items returned by the IR system.Table 1 shows the Mean Average Precision (MAP) valuesfor our submitted runs. While the differences between thevalues are small, there is a noticeable tendency of decreasingperformance as the confidence threshold is set higher.Run MAPRun 1 0.155Run 2 0.145Run 3 0.139Run 4 0.129Table 1: Official results (MAP values)as “unknown”language ASR +metadataall words 0.169 0.273conf.sc > 70% 0.158 0.269conf.sc > 80% 0.151 0.266conf.sc > 90% 0.141 0.264as distinctlanguages ASR +metadataall words 0.162 0.254conf.sc > 80% 0.150 0.253Table 2: Unofficial results (MAP values)3.2 Additional RunsWe repeated the “official” runs but now with no limit onthe number of returned documents (i.e. videos) to which thelabels were assigned, hence the slight changes in performancecompared to the official runs. We also performed retrievalon an indexed collection that included metadata (title anddescriptions fields) information as well. All other details(confidence scores, lack of preprocessing) were the same, re-sults are shown in the upper half of Table 2. Combiningmetadata with the transcript texts can be seen to improvethe performance, as one would expect.We also investigated indexing the transcripts separatelyfor each language. For each collection a language specificstop word list was used and stemming applied, optionallytexts were also combined with metadata. Stopping andstemming was similarly applied to query labels. Note thatsince these were not labelled by language, all labels wereused to search each collection with the stopping and stem-ming matching the transcript language applied. If cate-gorised by language, labels might be used multilingually, ei-ther with the same meaning (e.g. international words, propernames), or with a different meaning (the same word formcoincidentally existing in several languages). Search wasperformed in each language with the same conditions asbefore (confidence scores, metadata). The four results foreach query label were then merged into single file sortedby matching score. The results for the language-sensitiveruns are shown in the lower half of Table 2. These runs areslightly outperformed by runs considering language as un-known. The reason for this is not immediately clear and willbe the subject of further investigation.4. CONCLUSIONS AND FURTHERWORKAll our official and unofficial runs seem to confirm thattaking confidence scores into account does not necessarilyhelp in IR, a slight decrease in performance is noticeablehere. Recognising and dealing with language differences isalso an issue that needs further investigations. In furtherwork we also plan to explore the use of existing methodsfor determining different cut off points for individual tagssuch as [1], and also to examine other possible techniquesto do this, as some tags will be associated with many itemsand others with only a small number of them. Also we planto extend use of this IR based tagging method to the taskto open-set tagging. This will necessitate development of asystem component to automatically identify potential tags,as well as their assignment to the items to be annotated.5. ACKNOWLEDGEMENTSThis work is funded by a grant under the Science Founda-tion Ireland Research Frontiers Programme 2008 Grant No:08/RFP/CMS1677.6. REFERENCES[1] J. Ku¨rsten and M. Eibl. Video classification as IR task:Experiments and observations. In C. Peters, B. Caputo,J. Gonzalo, G. J. F. Jones, J. Kalpathy-Cramer,H. Mu¨ller, and T. Tsikrika, editors, MultilingualInformation Access Evaluation II. MultimediaExperiments, volume 6242 of Lecture Notes inComputer Science, pages 377–384. Springer Berlin /Heidelberg, 2010. 10.1007/978-3-642-15751-6 49.[2] M. Larson, E. Newman, and G. J. F. Jones. Overviewof videoclef 2009: New perspectives on speech-basedmultimedia content enrichment. In C. Peters,B. Caputo, J. Gonzalo, G. J. F. Jones,J. Kalpathy-Cramer, H. Mu¨ller, and T. Tsikrika,editors, Multilingual Information Access Evaluation II.Multimedia Experiments, volume 6242 of Lecture Notesin Computer Science, pages 354–368. Springer Berlin /Heidelberg, 2010. 10.1007/978-3-642-15751-6 46.[3] M. Sanderson and X. M. Shou. Search of spokendocuments retrieves well recognized transcripts. InECIR, pages 505–516, 2007.[4] K. Zechern and A. Waibel. Minimizing word error ratein textual summaries of spoken language. InProceedings of NAACL-ANLP-2000, 2000.

Minimizing word error rate in textual summaries of spoken language.

Overview of videoclef 2009: New perspectives on speech-based multimedia content enrichment.

Search of spoken documents retrieves well recognized transcripts.

Video classi as IR task: Experiments and observations.

ABSTRACT We describe our runs and results for the fixed label Wild Wild Web Tagging Task at MediaEval 2010. Our experiments indicate that including all words in the ASR transcripts of the document set results in better labeling accuracy than restricting the index to only words with recognition confidence above a fixed level. Additionally our results show that tagging accuracy can be improved by incorporating additional metadata describing the documents where it is available

Ágnes Gyarmati

Gareth J F Jones

CiteSeerX

DCU at MediaEval 2010—Tagging Task Wild Wild Web

We describe our runs and results for the fixed label Wild\ud
Wild Web Tagging Task at MediaEval 2010. Our experiments indicate that including all words in the ASR transcripts of the document set results in better labeling accuracy than restricting the index to only words with recognition confidence above a fixed level. Additionally our results\ud
show that tagging accuracy can be improved by incorporating additional metadata describing the documents where it is available

Name not available

http://doras.dcu.ie/16451/1/DCU_at_MediaEval_2010_%E2%80%93_Tagging_Task_WildWildWeb.pdf

DCU at MediaEval 2010 – Tagging task WildWildWeb

Abstract

Similar works

Full text

Available Versions

Irish Universities

DCU Online Research Access Service

CiteSeerX

Name not available