International audienceAt the TREC9 conference, we presented a new adaptive filtering system called RELIEFS. This system which is based on the idea of resonance, combines for each term t, the relative frequency of relevance knowing t and the relative frequency of t kwowing relevance. On the basis of other experiments, several changes have been made. We improved our threshold adaption, we slightly changed our relevance evaluation function and we gave up the use of conjunctions and thesaurus. The system is now focusing more exclusively on the combination of both reverse frequencies that we believe to represent the fundamental aspects of relevance estimation. This year we used the system in its new version and we tested it on the Reuters corpus. Focusing on the combination of the two frequencies, we varied their relative importance. The results show globally a good effectiveness especially when both frequencies are balanced

Brouard, Christophe

Hal - Université Grenoble Alpes

CLIPS at TREC-11: Experiments in FilteringChristophe BrouardTo cite this version:Christophe Brouard. CLIPS at TREC-11: Experiments in Filtering. 11th Text REtrievalConference, 2002, Unknown, 2002. <hal-00954140>HAL Id: hal-00954140https://hal.inria.fr/hal-00954140Submitted on 28 Feb 2014HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestine´e au de´poˆt et a` la diffusion de documentsscientifiques de niveau recherche, publie´s ou non,e´manant des e´tablissements d’enseignement et derecherche franc¸ais ou e´trangers, des laboratoirespublics ou prive´s.CLIPS  at TREC-11: Experiments in FilteringChristophe BrouardLaboratoire CLIPSEquipe MRIMBP. 5338041 Grenoble cedex 9, FranceChristophe.Brouard@imag.frAbstractAt the TREC9 conference, we presented a new adaptive filtering system calledRELIEFS.  This system which is based on the idea of resonance, combines for each term t, therelative frequency of relevance knowing t and the relative frequency of t kwowing relevance. Onthe basis of other experiments, several changes have been made. We improved our thresholdadaption, we slightly changed our relevance evaluation function and we gave up the use ofconjunctions and thesaurus. The system is now focusing more exclusively on the combination ofboth reverse frequencies that we believe to represent the fundamental aspects of relevanceestimation. This year we used the system in its new version and we tested it on the Reuterscorpus. Focusing on the combination of the two frequencies, we varied their relative importance.The results show globally a good effectiveness especially when both frequencies are balanced.1. IntroductionIn a logical approach, it has been found that the evaluation of relevance of a document Dto a query Q can be based on the evaluation of the implication from document to query D->Q(van Rijsbergen, 1986). Some authors also emphasized the role of the reverse implication (Q->D)(Nie, 1988). These two entailment relations can also be found in most probabilistic models,formulated as P(Q/D) (Probability of Q knowing D) and P(D/Q) (see Crestani, Lalmas, vanRijsbergen & Campbell, 1998, for an overview of Probabilistic models).If one considers a document as a set of terms, and a query as a specification of what weare looking for, the entailment D->Q may be decomposed to the judgment of "if the term ispresent then the document is relevant" for each term of the document. So, if we use a set ofdocuments with the relevant judgments associated, the entailment D->Q can be decomposed in aset of relative frequencies of relevance knowing term for each term of the document. Similarly,the reverse entailment relation can be decomposed in a set of relative frequencies of termknowing relevance. From a pragmatic point of view, the use of the first frequency is easilyunderstood since it allows to favour the terms which predict relevance (when the term is presentthen the document in which it occurs is relevant). The consideration of the reverse frequenciesmay be justified by the fact that it allows to favour terms which have been met many times inrelevant documents. Since they appear relatively frequently, these terms are likely to be usefullfor the next evaluations, and we can think that their presence in relevant documents is not acoincidence (comparing to rare terms which occured once or twice).Figure 1 – Relationships between terms and relevanceWe believe that both entailment relations capture the essence of relevance. As we notedbefore they are found in the most IR models but not always in a obvious way. With the systemRELIEFS we try to isolate them and to use them in a simple way. So, for each term ti , wepropose to compute the frequency of relevance knowing term F(R / t i) and the reverse frequencyF(t i / R) (Figure 1).2. System descriptionThe RELIEFS document processing can be decomposed in three steps:1. Selection of N terms from the document,2. Estimation of the document's relevance,3. Revision of term's relative frequencies.2.1 Step 1 : Selection of N document termsAll the document terms are compared with the terms which have been extracted from thequery, from the document examples given for learning and from the documents which have beenpreviously selected. They are sorted by the value the product F(R / t i)* F(t i / R). If less than Nterms  can be selected in this way (in our experiments we choose N=30), this selection iscompleted by the document terms in their lecture order.2.2 Step 2 : Relevance estimationConsidering the terms which appear in the document, the score of the document iscomputed as follows :F(R / ti )* F(t i / R)i =1N∑F(R/ t j )* F(t j / R)j =1N∑relevancetermrelevance knowing term term knowing relevanceF(R / t i) F(t i / R)where i are the indices of the N best terms (according to the product F(R / t) * F(t / R) ))which are present in the document and where j are the indices of the N best terms (present or notpresent in the document). In RELIEFS, the relevance of a document can be interpreted as a kindof resonance in a network (Figure 2) in which each relative frequency corresponds to a weightedconnection and each term corresponds to a node (Brouard & Nie, 2003). A relevance node is builtfor each query.relevanceterm6term1 term2 term3 term5term4Figure 2 - Relevance evaluation in RELIEFS of a document in which term1 and term4 occur2.3 Step 3 : Relative frequencies updatingIf the evaluation of step2 is larger than a defined threshold the document is selected andthe relevance feedback takes place. Given the relevant judgment, the N terms selected in the firststep are submitted to an updating process on their relative frequencies.2.4 Threshold adaptionThe queries are very different from each other. Therefore, a different threshold isdetermined for each query. The value of the threshold is determined empirically. The initialthreshold is computed on the basis of the score of the three first relevant documents. We used thefollowing strategy to adjust the value:- If a selected document is irrelevant, the system is considered to be too tolerant. Thethreshold is increased.- If a document (that we do not know if it is relevant or not) is not selected, the system isconsidered to be too strict. Then the threshold is decreasedThe increase scale is set to be higher than the decrease scale because there are much moreunselected documents than the selected ones. The decrease of threshold in the second case is dueto the fact that we do not want the system to remain silent for a too long period. This allows togradually correct an initial threshold that is fixed too high. In both cases, we considered differentcriteria for the modification of the threshold, including:- The number of irrelevant documents that are selected consecutively: The higher thisnumber, the larger the increase scale and the smaller the decrease scale.- The number of consecutive relevant document: The higher this number, the larger thedecrease scale. This change only concerns the decrease case.- The number of documents considered: The larger this number, the lower the changescales. The intuition behind this criterion is that we would make larger changes at the beginningof the filtering. When a certain number of documents have been treated, the system shouldstabilize.We also considered a more global criterion on the probability of relevance. Each score isassociated with a frequency of relevance and we try to adjust the threshold on the score for whichthe probability is 0.33 (this selection criterion is optimal for the utility measure).3. Experiments3.1 PreprocessingWe used the Porter’s stemmer to get the root form of every term and we removed thestopwords.3.2 Details of processingAll parameters (which mainly concern the threshold tuning) were tuned on the Oshumedcorpus (TREC9). For each document of the Reuters corpus, we only considered the title and thetext fields. For updating frequencies, the unjudged documents were considered as irrelevant. Apseudo-relevance feedback has been applied when the scores were smaller than a definedthreshold. In this case, the document is considered as irrelevant.3.3 Varying the relative importance of the entailment relationsIn our previous tests, both entailment relations are considered with the same importance:they are multiplied. Our question is, should they play equal role in the estimation or should onefactor be more important than another? In order to answer this question, we vary the importanceof one of the implication by replacing the weight F(R / t i) by F(R / t i)ρ . The factor ρ  changes therelative importance of this relation. The relevance estimate is defined as follows:F(R / ti )* F(t i / R)ρi =1N∑F(R/ t j )* F(t j / R)ρj =1N∑3.4 ResultsGlobally, it turns out that the best results are obtained when the two frequencies arebalanced ( ρ  in [1.0,1.4]) even if we can observe that good results for ρ =1.6 and ρ =1.7(Table 1). This is consistent with the previous observations we obtained with the Oshumedcorpus.ρ Utility ρ Utility ρ Utility ρ Utility0.0 -0.26 0.5 10.35 1.0 16.68 1.5 14.010.1 2.65 0.6 12.25 1.1 17.25 1.6 16.820.2 4.92 0.7 12.42 1.2 17.84 1.7 17.270.3 8.01 0.8 13.22 1.3 17.68 1.8 13.190.4 9.41 0.9 15.44 1.4 16.95 1.9 12.03Table 1 : The impact of ρ  on the utility.We submitted the best run ( ρ =1.2) for comparison on utility criteria. The comparison isfavourable since about 80% of the scores are above or equal to the median. Like the othersystems we obtained better results for the first 50 queries than for the 50 last ones which are builtdifferently.In conclusion, results suggest that the two reverses frequencies are sufficient clues toestimate the relevance. Furthermore, the fact that the best results are obtained when the twoaspects are balanced should indicate that they are both necessary and equally important.ReferencesBrouard, C., & Nie, J.-Y. (2000, 11/00). The system RELIEFS: a new approach for informationfiltering. Text Retrieval Conference (TREC-9), Washington D.C.Brouard, C., & Nie, J.Y. (2003). Relevance as Resonance : A new Theoretical Perspective and aPractical Utilisation in Information Filtering. Information Processing & Management. Revisionunder review.Crestani, F., Lalmas, M., C.J, V. R., & Campbell, I. (1998). Is This DocumentRelevant?...Probably: A Survey of Probabilistic Models in Information Retrieval. ACMComputing Surveys, 30(4), 528-552.Nie, J. Y. (1988). An outline of a general model for information retrieval. Proceedings of the 11thAnnual ACM Conference on Research and Development in Information Retrieval, Grenoble.van Rijsbergen, C. J. (1986). A non-classical logic for information retrieval. The ComputerJournal, 29(6), 481-485.

CLIPS at TREC-11: Experiments in Filtering

Abstract

Similar works

Full text

Available Versions

Hal - Université Grenoble Alpes