We describe Dublin City University (DCU)'s participation in the VideoCLEF 2009 Linking Task. Two approaches were implemented using the Lemur information retrieval toolkit. Both approaches rst extracted a search query from the transcriptions of the Dutch TV broadcasts. One method rst performed search on a Dutch Wikipedia archive, then followed links to corresponding pages in the English Wikipedia. The other method rst translated the extracted query using machine translation and then searched the English Wikipedia collection directly. We found that using the original Dutch transcription query for searching the

Dutch Wikipedia yielded better results

Gyarmati, Ágnes

Jones, Gareth J.F.

English

DCU Online Research Access Service

When to Cross Over? Cross-Language LinkingUsing Wikipedia for VideoCLEF 2009A´gnes Gyarmati and Gareth J. F. JonesCentre for Digital Video ProcessingDublin City University, Dublin 9, Ireland,{agyarmati,gjones}@computing.dcu.ie,Abstract. We describe Dublin City University (DCU)’s participation inthe VideoCLEF 2009 Linking Task. Two approaches were implementedusing the Lemur information retrieval toolkit. Both approaches first ex-tracted a search query from the transcriptions of the Dutch TV broad-casts. One method first performed search on a Dutch Wikipedia archive,then followed links to corresponding pages in the English Wikipedia. Theother method first translated the extracted query using machine trans-lation and then searched the English Wikipedia collection directly. Wefound that using the original Dutch transcription query for searching theDutch Wikipedia yielded better results.1 IntroductionThe VideoCLEF Linking Task involved locating content related to sections ofan automated speech recognition (ASR) transcription cross-lingually. Elementsof a Dutch ASR transcription were to be linked to related pages in an EnglishWikipedia collection [1]. We submitted four runs by implementing two differ-ent approaches to solve the task. Because of the difference between the sourcelanguage (Dutch) and the target language (English), a switch between the lan-guages is required at some point in the system. Our two approaches differed inthe switching method.One approach performed the search in a Dutch Wikipedia archive with theexact words (either stemmed or not) and then returned the corresponding linkspointing to the English Wikipedia pages. The other one first performed an auto-matic machine translation of the Dutch query into English, the translated querywas then used to search the English Wikipedia archive directly.2 System DescriptionFor our experiments we used the Wikipedia dump dated May 30th 2009 for theEnglish archive, and the dump dated May 31st 2009 for the Dutch Wikipediacollection. In a simple preprocessing phase, we eliminated some information ir-relevant to the task, e.g. information about users, comments, links to other2languages we did not need. For indexing and retrieving, we used the Indri modelof the open source Lemur Toolkit [2]. English texts were stemmed using Lemur’sbuilt-in stemmer, while Dutch texts were stemmed using Oleander’s implemen-tation [5] of Snowball’s Dutch stemmer algorithm [6]. We used stopword listsprovided by Snowball for both languages.Queries were formed based on sequences of words extracted from the ASRtranscripts using the word timing information in the transcript file. For eachof the anchors defined by the task, the transcript was searched from the an-chor starting point until the given end point, and the word sequence betweenthese boundaries extracted as the query. These sequences were used directlyas queries for retrieval from the Dutch collection. The Dutch Wikipedia’s linkspointing to the corresponding articles of the English version were returned as thesolution for each anchor point in the transcript. For the other approach querieswere translated automatically from Dutch to English using the query translationcomponent developed for the Multimatch project [3]. This translation tool com-bines the WorldLingo machine translation engine augmented with a bilingualdictionary from the cultural heritage domain automatically extracted from themultilingual Wikipedia. The translated query was used to search the EnglishWikipedia archive.3 Run ConfigurationsHere we describe the four runs we submitted to the Linking Task, plus an addi-tional one performed subsequently.1. Dutch The Dutch Wikipedia was indexed without stemming or stopping.Retrieval was performed on the Dutch collection, returning the relevant linksfrom the English collection.2. Dutch stemmed Identical to Run 1, except that the Dutch Wikipedia textis stemmed and stopped as described in Sect. 2.3. English This run represents the second approach with stop word removaland stemming applied to the English documents and queries. The translatedquery was applied to the indexed English Wikipedia.4. Dutch with blind relevance feedback This run is almost identical toRun 1, with a difference in parameter setting for Lemur to perform blindrelevance feedback. Lemur/Indri uses a relevance model for query expansion,for details see [4]. The first 10 retrieved documents were assumed relevantand queries were expanded by 5 terms.5. English (referred to as 3′) This is an amended version of Run 3, with the dif-ference of an improved preprocessing phase applied to the English Wikipedia,disregarding irrelevant pages as described in Sect. 4.4 ResultsThe Linking Task was assessed by the organisers as a known item task. The topranked relevant link for each anchor is referred to as a primary link, and all otherrelevant links identified by the assessors as secondary links [1].3Table 1. Scores for Related LinksRun Recall (prim) MRR (prim) MRR (sec)Run 1 0.267 0.182 0.268Run 2 0.267 0.182 0.275Run 3 0.079 0.056 0.090Run 4 0.230 0.144 0.190Run 3′ 0.230 0.171 –Table 1 shows Recall and Mean Reciprocal Rank (MRR) for primary links,and MRR values for secondary links. Recall cannot be calculated for secondarylinks due to the lack of an exhaustive identification of secondary links. Table 1also includes Run 3′ evaluated automatically using the same set of primary linksas in the official evaluation. Secondary links have been omitted as we could notprovide the required additional manual case-by-case evaluation by the assessors.Runs 1 and 2 achieved the highest scores. Although they do yield slightlydifferent output, the decision of whether to stem and stop text does not alter theresults statistically, in the matter of primary links, while stemming and stopping(Run 2) did improve results a little in finding secondary links. Run 4 using blindrelevance feedback to expand the queries was not effective here. Setting theoptimal parameters for this process would require further experimentation, andeither this or alternative expansion methods may produce better results.The main problem of retrieving from the Dutch collection lies in the dif-ferences between the English and the Dutch versions of Wikipedia. Althoughthe English site contains a significantly larger number of articles, there are ar-ticles that have no equivalent pages cross-lingually, due to different structuringor cultural differences. Systems 1, 2 and 4 might (and in fact did) come up withrelevant links at some points which were lost when looking for a direct link toan English page. Thus a weak point of our approach is that some hits fromthe Dutch Wikipedia might get lost in the English output due to the lack ofan equivalent English article. In the extreme case, our system might return nooutput at all if none of the hits for a given anchor are linked to any page in theEnglish Wikipedia.Run 3 performed significantly worse. This might be due to two aspects ofthe switch to the English collection. First, the query text was translated au-tomatically from Dutch to English, which in itself carries a risk of translationerrors due to misinterpretation of the query or weaknesses in the translationdictionaries. While the MultiMatch translation tool has a vocabulary expandedto include many concepts from the domain of cultural heritage, there are manyspecialist concepts in the ASR transcription which are not included in its trans-lation vocabulary. Approximately 3.5% of Dutch words were left untranslated (inaddition to names). Some of these turned out to be important expressions, e.g.rariteitenkabinet ’cabinet of curiosities’, which were in fact successfully retrievedby the systems for Run 1 and 2 (although ranked lower than desired).4The other main problem we encountered in Run 3 lay in the English Wikipediaand our limited experience concerning its structure. The downloadable dump in-cludes a large number of pages that look like useful articles, but are in fact not.These articles include old articles set for deletion and meta-articles containingdiscussion of an existing, previous or future article. We were not aware of thesearticles during the initial development phase, but this had a significant impacton our results, about 18.5 % of the links returned in Run 3 proved to be in-valid articles. Run 3′ reflects results where the English Wikipedia archive hasbeen cleaned up to remove these irrelevant pages prior to indexing. As shownin Table 1, this cleanup produces a significant improvement in performance. Asimilar cleanup applied to the Dutch collection would produce a new ranking ofDutch documents. However, very few of the Dutch pages which would be deletedin cleanup are actually retrieved or have a link to English pages, and thus anychanges in the Dutch archive will have no noticeable effect on evaluation of theoverall system output.5 ConclusionsIn this paper we have outlined the two approaches used in our submissions tothe Linking Task at VideoCLEF 2009. We found using the source language forretrieval to be more effective than switching to the target language in an earlyphase. This result may be different if translation of the query for the secondmethod were to be improved. Both methods could be expected to benefit fromthe ongoing development of Wikipedia collections.AcknowledgementsThis work is funded by a grant under the Science Foundation Ireland ResearchFrontiers Programme 2008. We are grateful to Eamonn Newman for assistancewith the MultiMatch translation tool.References1. Larson, M., Newman, E. and Jones, G. J. F.: Overview of VideoCLEF 2009: NewPerspectives on Speech-based Multimedia Content Enrichment. In: Multilingual In-formation Access Evaluation Vol. II Multimedia Experiments. Lecture Notes in Com-puter Science, Springer (2010)2. The Lemur Toolkit. http://www.lemurproject.org/3. Jones, G. J. F., Fantino, F., Newman, E., Zhang, Y.: Domain-Specific Query Trans-lation for Multilingual Information Access Using Machine Translation AugmentedWith Dictionaries Mined From Wikipedia, In Proceedings of the 2nd InternationalWorkshop on Cross Lingual Information Access - Addressing the Information Needof Multilingual Societies (CLIA-2008), Hyderabad, India, pages 34–41, (2008)4. Metzler, Don. Indri Retrieval Model Overview.http://ciir.cs.umass.edu/˜metzler/indriretmodel.html5. Oleander Stemming Library. http://sourceforge.net/projects/porterstemmers/6. Snowball. http://snowball.tartarus.org/

Domain-Speci Query Translation for Multilingual Information Access Using Machine Translation Augmented With Dictionaries Mined From Wikipedia,

Indri Retrieval Model Overview.

Overview of VideoCLEF 2009: New Perspectives on Speech-based Multimedia Content Enrichment. In: Multilingual Information Access Evaluation Vol. II Multimedia Experiments. Lecture Notes in Computer Science,

The Lemur Toolkit.

When to cross Over? Cross-language linking using Wikipedia for VideoCLEF 2009

We describe Dublin City University (DCU)\u27s participation in the VideoCLEF 2009 Linking Task. Two approaches were implemented using the Lemur information retrieval toolkit. Both approaches rst extracted a search query from the transcriptions of the Dutch TV broadcasts. One method rst performed search on a Dutch Wikipedia archive, then followed links to corresponding pages in the English Wikipedia. The other method rst translated the extracted query using machine translation and then searched the English Wikipedia collection directly. We found that using the original Dutch transcription query for searching the
Dutch Wikipedia yielded better results

Irish Universities

We describe Dublin City University (DCU)'s participation in the VideoCLEF 2009 Linking Task. Two approaches were implemented using the Lemur information retrieval toolkit. Both approaches rst extracted a search query from the transcriptions of the Dutch TV broadcasts. One method rst performed search on a Dutch Wikipedia archive, then followed links to corresponding pages in the English Wikipedia. The other method rst translated the extracted query using machine translation and then searched the English Wikipedia collection directly. We found that using the original Dutch transcription query for searching the\ud
Dutch Wikipedia yielded better results

Name not available

http://doras.dcu.ie/15365/1/gyarmati-DCU_VideoCLEF2009.pdf

When to cross Over? Cross-language linking using Wikipedia for VideoCLEF 2009

Abstract

Similar works

Full text

Available Versions

DCU Online Research Access Service

Irish Universities

Name not available