Cardoso, Nuno

Costa, Miguel

Silva, Mário J.

English

Repositório Comum

The XLDB Group at CLEF 2004Nuno Cardoso, Mário J. Silva, and Miguel CostaGrupo XLDB - Departamento de Informática,Faculdade de Ciências da Universidade de Lisboa{ncardoso, mjs, mcosta} at xldb.di.fc.ul.ptAbstract. This paper describes the participation of the XLDB Group in theCLEF monolingual ad hoc task for Portuguese. We present tumba!, a Portuguesesearch engine and describe its architecture and the underlying assumptions. Wediscuss the way we used tumba! in CLEF, providing details on our runs and ourexperiments with ranking algorithms.1 IntroductionIn 2004, for the first time, CLEF included Portuguese document collections for mono-lingual & bilingual ad hoc retrieval and question answering tasks. This collection [14]was based on news of several categories taken from Publico [13], a Portuguese newspa-per, and compiled by Linguateca [7]. This year, the XLDB Group, from the Universityof Lisbon, made its debut in CLEF.This paper is organized as follows: in Section 2, we introduce the XLDB Group. InSection 3, we describe tumba!, our IR system, and the modifications we made to it tohandle the CLEF 2004 data set. Section 4 describes our official runs with the algorithmsimplemented for CLEF 2004, and Section 5 presents our results. Section 6 summarizesthe conclusions we drew from this first participation in CLEF.2 The XLDB GroupThe XLDB Group is a research unit of LaSIGE (Large Scale Information Systems Lab-oratory) at FCUL - Faculdade de Ciências da Universidade de Lisboa. We study datamanagement systems for data analysis, information integration and user access to largequantities of complex data from heterogeneous platforms. Current research lines spanWeb search, mobile data access, temporal web data management and bioinformatics.The XLDB Group is involved in several projects and activities. One of our mainprojects is tumba! [8, 15], a Portuguese Web search engine. tumba! is described in Sec-tion 3.Since January 2004, the XLDB Group hosts a node of Linguateca, a distributedlanguage resource center for Portuguese [6].The participation of the XLDB Group in the monolingual task for Portuguese withthe tumba! search engine was motivated by two main reasons:C. Peters et al. (Eds.): CLEF 2004, LNCS 3491, pp. 245–252, 2005.c©Springer-Verlag Berlin Heidelberg 2005246 N. Cardoso, M.J. Silva, and M. Costa1. Although we had previous experiences in evaluation contests, namely in the bio-texttask of the KDD Cup 02 [4] and in the BioCreative workshop [5], this was our firstopportunity to evaluate tumba! jointly with other IR systems, with the advantage ofthe evaluation being conducted on a Portuguese collection.2. Although we were aware that our system was out of its natural environment, theWeb, we could take the opportunity to tune the indexing and ranking engines oftumba!, by submitting our results using different ranking configurations and thenanalyzing the results.3 tumba! in the Monolingual Task3.1 Overview of tumba!The tumba! search engine has been specifically designed to archive and provide searchservices to a Web community formed by those interested in subjects related to Portu-gal and the Portuguese people [8]. tumba! has been offered as a public service sinceNovember 2002.tumba is mainly written in Java and built on open-source software: the Linux oper-ating system. It has an index of over 3.5 million Web documents and a daily traffic ofup to 20,000 queries per day. Its response time is less than 0.5 seconds for 95% of therequests. It is also a platform for PhD and MSc research projects at our university.WEB USERSused on CLEFCrawlers WebRepositoryRankingSystemPresentationEngineIndexingSystemFig. 1. tumba’s architectureThe architecture of tumba! is similar to that of global search engines and adoptsmany of the algorithms used by them [1]. However, its configuration data is much richerin its domain of specialisation. tumba! has a better knowledge of the location and orga-nization of Portuguese Web sites (both in qualitative and quantitative terms) [15].The data flows from the Web to the user through a pipeline of the following tumba!sub-systems (See Figure 1):Crawlers: collect documents from the Web, given an initial URL list. They parse andextract URLs from each document, and use these to collect new documents. Thesesteps are performed recursively until a stop condition is met [10].Web Repository: The Web data collected by the crawlers is stored in Versus, a repos-itory of Web documents and associated meta-data [9].The XLDB Group at CLEF 2004 247Indexing system: the indexing system Sidra creates indexes over the documents in theWeb Repository [3], so that when a query is received, Sidra uses the indexes builtto find the documents that match that query.Ranking system: computes, for each document d returned by the indexing system, asimilarity value between d and the submitted query using a set of heuristics. It thensorts the documents by these similarities.Presentation Engine: formats the result sets received from the ranking engine for theuser’s access platforms such as Web browsers, PDA devices or WAP phones.3.2 The Portuguese Monolingual TaskThe previous editions of CLEF showed that the top performing groups for the monolin-gual ad hoc tasks were systems that performed robust stemming, well-known weight-ing schemes (BM25, Lnu.ltn or Berkeley ranking) and blind feedback or query ex-pansion [12]. tumba’s system does not have a stemmer and a blind feedback or queryexpansion system, and the term weighting scheme is tuned for Web searches. However,we decided not to make any changes to the architecture of tumba! for this evaluation.We wanted to evaluate tumba!’s performance with its current components, so that wecould have a baseline for comparison on future CLEF tasks. Nonetheless, we felt thatour participation in CLEF would provide us with valuable ideas to optimize our searchengine results, and resources to evaluate our system performance.One of the difficulties we encountered in the CLEF monolingual task was relatedto the SGML-format used on the collection of Portuguese documents. The documentshave tags for associated metadata like author, category and date of publication. Thecontents are in plain text, with no additional tags. tumba! was not conceived to workwith document collections organized like this. Its ranking system was developed toprofit from annotations extracted from the Web documents, such as:– Information obtained from the Web graph, like links and anchor text, which are avaluable resource to find related pages that might interest the user;– Documents’ structural elements like titles and headings, which provide valuableinformation on the subject of the document.We used the same alghorithms as those designed for the Web in CLEF, despite thedifferent search context. The lack of this kind of “light semantic” annotation in thecollection was a major handicap for the tumba! system, since the only semantic in-formation we managed to extract from the documents was the title of the news. Ourheuristic for extracting document titles consisted in finding paragraphs in the collectionwith a maximum of 15 terms and ending with no punctuation.We disabled the query-independent ranking calculations and most of the emphasisranking augmenters of the Indexing and Ranking system, since there was no informa-tion of this type on the collection.tumba’s Crawlers and Presentation Engine were not used for the CLEF Portuguesemonolingual ad hoc task. We loaded the document collection directly into the WebRepository, bypassing the system’s crawlers. The collection was then indexed by theSidra Indexing system. Queries were sent directly to Sidra, bypassing the PresentationEngine, and the matching documents were then ranked according to some heuristics tocompute document relevance.248 N. Cardoso, M.J. Silva, and M. Costa4 RunsIn the monolingual ad hoc task, the number of runs that could be submitted was limitedto 4.4.1 Manual Run (XLDBTumba01)Since this was the first time that CLEF used Portuguese collections in an evaluationcampaign, this task didn’t have previous relevant judgements and training collections.In order to have a prior evaluation of tumba!, we created our own baseline against whichwe could compare our runs to measure how much we were improving our system.For each one of the 50 given topics, we created several different queries related to thetopic and we used them to retrieve documents matching the query terms. Then, the re-turned results were manually examined by two doctoral students, with some IR systemusage experience but unfamiliar with the tumba! system, and classified the documentsas relevant or irrelevant according to the topic criteria. This was time-consuming work,which consumed most of the time for this task.After that, we compiled a list of the relevant documents and submitted it to CLEF asour run XLDBTumba01, to measure the offset of our baseline compared with the CLEFsolutions.When the relevant judgements were released by CLEF, we observed that we hadmany errors in our manual experiment; from incorrect topic interpretation to bad queryformulation. In the end, this was the run that had the worst performance. Yet, this runclearly showed us how difficult it is to formulate queries that correctly match an infor-mation need.4.2 Flat Ranking Run (XLDBTumba02)For subsequent runs, we chose among the different queries used to create the XLDB-Tumba01 run to select which 50 queries would be used on the remaining runs. Notethat we didn’t use more than one query per topic, neither did we do any kind of queryexpansion.This run was produced by submitting the 50 queries directly to the Sidra Indexingand Ranking system, configured to perform an exact matching (flat-ranking algorithm),returning only the documents that match all the query terms.We see this run as our automatic baseline run, and we were anticipating that theother runs would improve precision and recall compared to this run. Yet, this run out-performed all the other runs.4.3 Distances Run (XLDBTumba05)This run was generated using the following ranking algorithm:– distMinTerms(d,q) - uses the minimum distances between any pair of query terms qin documents d, minDist, to increase the ranking of documents whose query termsare closer in the document. For distances above 10, the function gives similarity 0to the document. If all query terms are adjacent on a document, their minDist valueequals 1.The XLDB Group at CLEF 2004 249distMinTerms(d,q) =⎧⎨⎩1 minDist = 11− minDist−19 1 < minDist < 100 minDist ≥ 10This function indeed improved the results accordingly to our own evaluation, as thequeries with more than one term we used for the topic tend to be adjacent.4.4 Distances + Titles Run (XLDBTumba04)This run was generated by using two ranking algorithms in Sidra:– distMinTerms(d,q)– termsInTitle(d,q) - this is a similarity function between the terms in the title of eachdocument d, denoted T, and the query terms in a query q, denoted Q.termsInTitle(d,q) = |T ∩Q|max(|T |, |Q|)This run evaluated the importance of the title in the document ranking, and turnedout as the one with the worst performance in our self-evaluation. This was probablycaused by the heuristic used to extract titles from the documents, which was a verynaive approach and may have mislead the ranking engine. The tumba! search enginegives great importance to title texts, as many people search named entities on searchengines and these are usually clearly stated in the titles.5 ResultsFor a prior evaluation of our automatic runs, we compared the results with the manualrun XLDBTumba01. We used precision@1, precision@3, precision@10, recall and F-Measure (β = 1) metrics in our self-evaluation. The results are summarized inTable 1.The results obtained in CLEF are presented on Table 2 and Figure 2. The aver-age precision (non-interpolated) for all relevant documents and the R-Precision (preci-sion after R documents retrieved) are the measures presented by the trec_eval program.[2, 11]Table 1. Automatic Submitted Runs, compared to the Manual Run XLDBTumba01Run Description Precision@ Recall F-Measure1 3 10XLDBTumba02 flat ranking 53.2% 47.2% 40.6% 89.6% 44.4%XLDBTumba05 Distances 46.8% 53.5% 44.9% 89.6% 44.4%XLDBTumba04 Distances & Titles 48.9% 45.0% 41.1% 89.6% 44.4%250 N. Cardoso, M.J. Silva, and M. CostaTable 2. XLDB official runs evaluated by CLEFRun Manual Run Flat ranking Distances Distances + titles(XLDBTumba01) XLDBTumba02 XLDBTumba05 XLDBTumba04Nr. Docs Retrieved 209 2350 2350 2350Nr. Relevant Docs 678 678 678 678Relevant Docs Retrieved 79 168 168 168Overall Precision 37,8% 7,1% 7,1% 7,1%Overall Recall 11,6% 24,8% 24,8% 24,8%Average Precision 21,84% 28,10% 25,13% 27,75%R-Precision 22,41% 26,28% 26,73% 27,26%The XLDBTumba02, XLDBTumba05 and XLDBTumba04 runs have the same over-all precision and recall values, because we used the same queries which retrieved thesame documents, differing only in the order in which the documents were submitted foreach topic.Fig. 2. Recall-Precision Values for our runs, according to CLEF results6 ConclusionWe used the Web search engine tumba! in the CLEF 2004 monolingual task for Por-tuguese. Our main objective was to test, compare, and improve the quality of tumba’sresults, and gather ideas on how to do it. However, the enviroment that we work on, theWeb, is different from the flat and small collection of document texts that we used inthe CLEF task.As we didn’t have a baseline of relevant judgements, we manually annotated rele-vant and non relevant documents for the 50 topics. We found that this task is not easy.It is time consuming and requires experienced human annotators to review hundredsof documents, compare the results and eliminate erroneous judgements. The other runsThe XLDB Group at CLEF 2004 251submitted used combinations of two algorithms used on the tumba! ranking engine. Wedid our own evaluation with several metrics based on our own relevance judgements,and submitted 4 runs for CLEF evaluation. We presented both evaluations in this paper.tumba! does not perform stemming or query expansion and relies heavily on de-tecting the presence of query terms in document titles and URLs. As these were notavailable for this evaluation, the performance of tumba! was below average when com-pared to other systems.During the creation of the XLDBTumba01 run and while analysing our results to-gether with the CLEF relevant judgements, we realized that in many cases, a simplequery could not retrieve all the relevant documents. Take for instance, topic #204,looking for documents concerning avalanche victims. In the Portuguese monolingualtask, this topic had 7 relevant judgements, which contained the relevant words of the’avalanche’ noun and the ’morrer’ verb (to die) / ’morte’ (death) family shown inTable 3.Table 3. Relevant words in the relevant documents of the topic 204Word Rel #1 Rel #2 Rel #3 Rel #4 Rel #5 Rel #6 Rel #7avalanche x x xavalanches x x xavalancha x x xmortos x x x xmortas xmorte x x x x xmorreu xmorreram x xmorrido xmata xWe can see that it would be impossible on a system like tumba! to achieve a goodrecall value with a query containing ’avalanche’ ’morte’ terms only. This is a situationthat is not uncommon and systems must be able to deal with it. We intend to extend ourWeb search system to provide much better results in situations where the documentsare not rich in HTML features, such as hyperlinks and meta-tags. tumba! is effectivein named-page finding tasks, in particular when these have appropriate titles and havemultiple links, but needs to become more effective in supporting other queries as well.AcknowledgementsWe should like to thank to Bruno Martins and Daniel Gomes for making changes totumba! components for the CLEF task and their valuable comments, Marcírio Chavesand Lauro Nakayama for their manual retrieval and judgement of the documents for theXLDBTumba01 run, and Diana Santos and Luís Costa for their valuable suggestionsand comments.252 N. Cardoso, M.J. Silva, and M. CostaThis work was financed by the Portuguese Fundação para a Ciência e Tecnologiathrough grant POSI / PLP / 43931 / 2001 (Linguateca) and by grant POSI / SRI / 40193/ 2001 (XMLBase Project).References1. Arvind Arasu, Junghoo Cho, Hector Garcia-Molina, Andreas Paepcke, and Sriram Ragha-van. Searching the Web. j-TOIT, 1(1):2–43, August 2001. http://www.acm.org/pubs/contents/journals/toit.2. Martin Braschler and Carol Peters. CLEF 2002 Methodology and Metrics, Advances inCross-Language Information Retrieval: Results of the CLEF 2002 Evaluation Campaign.Lecture Notes in Computer Science, 2758, Spring 2003.3. Miguel Costa and Mário J. Silva. Sidra: a Flexible Distributed Indexing and Ranking Archi-tecture for Web Search. In Proceedings of the VIII Conference on Software Engineering andDatabases JISBD 2003, Alicante, Spain, November 2003.4. Francisco Couto, Bruno Martins, Mário J. Silva, and P. Coutinho. Classifying BiomedicalArticles using Web Resources: application to KDD Cup 02. DI/FCUL TR 03–24, Depart-ment of Informatics, University of Lisbon, July 2003.5. Francisco Couto, Mário Silva, and P. Coutinho. Finding Genomic Ontology Terms in Textusing Information Content. In Critical Assessment of Information Extraction systems inBiology (BioCreative), Granada, Spain, March 2004. BMC Bioinformatics Journal (acceptedfor publication).6. Pólo XLDB da Linguateca. http://xldb.di.fc.ul.pt/linguateca/.7. Linguateca Distributed Resource Center for the Portuguese Language. http://www.linguateca.pt.8. Tumba! Portuguese Web Search Engine. http://www.tumba.pt.9. Daniel Gomes, João P. Campos, and Mário J. Silva. Versus: a Web Repository. In WDAS -Workshop on Distributed Data and Structures 2002, Paris, France, March 2002.10. Daniel Gomes and Mário J. Silva. Tarântula - Sistema de Recolha de Documentos da Web.In CRC’01 - 4a Conferência de Redes de Computadores, November 2001. (in Portuguese).11. Notes on TREC Eval. http://ir.iit.edu/~dagr/cs529/files/project_files/trec_eval_desc.htm.12. Carol Peters and Martin Braschler. Cross-Language Evaluation Forum: Objectives, Results,Achievements. Information Retrieval, 7(1/2):7–31, January/April 2004.13. Público. http://www.publico.pt.14. Diana Santos and Paulo Rocha. CHAVE: Topics and Questions on the Portuguese Participa-tion in CLEF. This volume, 2004.15. Mário J. Silva. The Case for a Portuguese Web Search Engine. In Proceedings of theIADIS International Conference WWW/Internet 2003, ICWI 2003, pages 411–418, Algarve,Portugal, 5-8 Novembro 2003. IADIS.

The XLDB Group at CLEF 2004

https://comum.rcaap.pt/bitstream/10400.26/95/1/200501cardoso04clefadhocSpringer.pdf

The XLDB Group at CLEF 2004

Abstract

Similar works

Full text

Available Versions

Repositório Comum