For the first year of the TREC Microblog Track the CLARITY group concentrated on a number of areas, investigating the underlying term weighting scheme for ranking tweets, incorporating query expansion to introduce new terms into the query, as well as introducing an element of temporal re-weighting based on the temporal distribution of assumed relevant microblogs

Ferguson, Paul

Lanagan, James

McCarthy, Kevin

O'Hare, Neil

Phelan, Owen

Smeaton, Alan F.

Smyth, Barry

English

Name not available

CLARITY at the TREC 2011 Microblog TrackPaul Ferguson, Neil O’Hare, James Lanagan, Alan F. SmeatonCLARITY: Centre for Sensor Web Technologies,Dublin City University.Owen Phelan, Kevin McCarthy, Barry SmythCLARITY: Centre for Sensor Web Technologies,University College Dublin.AbstractFor the first year of the TREC Microblog Track the CLARITY group concentrated on anumber of areas, investigating the underlying term weighting scheme for ranking tweets,incorporating query expansion to introduce new terms into the query, as well as introducingan element of temporal re-weighting based on the temporal distribution of assumed relevantmicroblogs.1 IntroductionThe introduction of the Microblog Track at TREC provides an opportunity to develop newresearch on a large corpus of microblog data (Tweets2011 collection). The CLARITY grouptook this opportunity to investigate a number of approaches to the retrieval of microblogdata, which exhibits diﬀerent characteristics than text traditional corpora.In Section 2 we discuss the our baseline approach for ranking microblogs in response toa user query. Section 3 outlines our method for providing query expansion, while Section 4describes our approach for re-weighting results based on their temporal proximity to a setof assumed relevant microblogs. Our experiments are described in Section 5 and we drawconclusions in Section 6.2 Tweet SortingTo provide a term-based document sorting baseline we used the Okapi BM25 model [1],which has been used extensively within the TREC community on a variety of corpora. Thebase formula that we used is as follows:Scorebm25(q, d) =￿t￿qlog￿N − dft + 0.5dft + 0.5￿× (k1 + 1)tftk1((1− b) + b dlavdl ) + tft(1)Here tft represents the within document term frequency, dl is the document length and avdlis the average document length over the collection. Also, dft is the number of times theterm t occurs within the collection although, in order to comply with the TREC Microblogrequirements for not using future data, this value was dynamically calculated based on thequery time of the topic: eﬀectively finding the number of occurrences of the term t from thestart time of the collection until the time the query was issued.Due to the short nature of microblogs we choose to set the BM25 parameters to eﬀectivelyuse a binary term weighting scheme within the BM25 model and, to discount documentnormalisation, to oﬀer no penalty against longer microblogs. This can be achieved with1the BM25 model by setting the parameters k1 and b to 0. This simplifies Equation 1 sothat only the inverse document frequency (IDF) component of the equation is considered,leaving us with the following:Scorebm25−idf (q, d) =￿t￿q,dlog￿N − dft + 0.5dft + 0.5￿(2)One of the drawbacks of simplifying the BM25 to that shown in Equation 2 is that thereis a likelihood that a large number of results will have tied scores. We choose to resolvethese ties was based on the recency of the microblogs, so that for any tied scores the mostrecent microblogs would appear first in the ranked list.3 Query ExpansionSince microblogs are very short documents and the query topics for this task are short, thequery terms may not always be present in a relevant document. For this reason, we usestandard pseudo relevance feedback query expansion techniques to add new terms to thequery, in an attempt to create a better representation of the topic.Since our baseline approach ranks solely be the IDF component of the BM25 formuladescribed in Equation 2, we can calculate the maximum possible ranking score for anydocument as the sum of the IDF scores for all of the query terms: any document containingall the query terms will have this maximum score. Rather than using the standard approachto pseudo relevance feedback and assuming that the top N microblogs are relevan, whetherthe value ofN is chosen in advance, instead we assume relevance on the ratio of a microblog’sscore to this maximum possible score:Scoreratio =Scorebm25−idf (q, d)max(Scorebm25−idf (q))(3)Note that the maximum score in this formula refers to the maximum possible score, and notthe score if the highest ranked document. If Scoreratio is greater than a threshold value,λ, then a document is assumed to be relevant. In the absence of any ground truth queryrelevance (qrel) data to tune this parameter, the value chosen was, by necessity, somewhatarbitrary. For the experiments described here, a value of λ = 0.7 was used, meaning adocument’s relevance score needed to be more than 70% of the maximum possible scoreto be assumed relevant. This approach will cause N , the number of assumed relevantdocuments, to vary from topic to topic, and some topics will have no assumed relevantdocuments.Once we have chosen a set of assumed relevant documents, we use the Robertson Selec-tion Value [2] to rank the terms in this pseudo relevant set:rsv =rN× rw (4)where r is the number of relevant document the term occurs in, rw is the IDF weightingof the term, and N is the number of assumed relevant microblogs. After ranking candidateterms by their rsv score, we add the top X terms to the query, and then use Equation 2 torank microblogs by this new query. Again, we arbitrarily choose a value for X, setting it to4.4 Temporal Re-weightingIn order to test the hypothesis that the time a microblog was published is an importantfactor in determining its relevance to a given query, we use pseudo relevance feedback tomodel the temporal distribution of a topic. We use the approach described in the previous2section to create pseudo relevance judgements for a topic. If there are 2 or more assumedrelevant microblogs for a topic, we make the initial assumption that the oldest of theserepresents the start time for a topic, and that the most recent of these represents the endtime for the topic. We then expand this temporal extent for the topic as follows:rangeinit = end timeinit − start timeinit (5)start timeextented = start timeinit − rangeα(6)end timeextended = end timeinit +rangeα(7)rangeextended = end timeextended − start timeextended (8)Equation 5 calculates the temporal range of the topic as the amount of time between thefirst and last relevant microblog. We then expand this temporal range, with the extentof the expansion determined by the α parameter: a value of α = ∞ would result in noexpansion of a topic’s temporal range, while a value of 2 would double the the temporalrange of a topic (i.e. a 50% expansion on either side).If a microblog falls outside of this temporal range, we re-weight microblogs based on this‘temporal centre’ of the topic:Scoretemporal = Scorebm25−idf (q, d)× temporal weight (9)where the temporal weight is calculated as follows:decay factor =rangeextendedα(10)temporal weight =decay factordistancetemp(11)The distancetemp is the temporal distance of a microblog from the start/end of thetopic’s temporal range. The decay factor controls how quickly a microblog’s score decaysas it’s distance from the temporal centre of a query increases, and is controlled by the sameα parameter used in Equations 6 and 7, with a larger parameter value enforcing a moresevere temporal decay.5 ExperimentsFor our four oﬃcial submissions to the TREC Microblog Track we submitted the followingruns: CLARITY1, CLARITY2, CLARITY3, CLARITY4, which we will now describe inmore detail.CLARITY1: this provides us with a baseline run, and does not use future data orexternal evidence. The run uses the BM25 sorting approach described in Equation 2 inSection 2 and, as described previously, tied scores are resolved based on the recency of thetweet, so that the most recent tweet appears higher in the ranked list.CLARITY2: in addition to the baseline sorting provided by CLARITY1, the CLAR-ITY2 run adds query expansion with pseudo relevance feedback as described in Section 3.The number of assumed relevant documents will vary from topic to topic, with the relevancethreshold(λ) set to 70%. The number of query expansion terms is fixed at 4. This run doesnot use future data or external evidence.CLARITY3: the CLARITY3 run uses the same approach as the previous CLARITY2run, however in addition it uses a language classification step in an attempt to filter non-English tweets (which are considered to be non-relevant). For this language classifier we3used a Language Detection Library1 which we use to filter out non-English tweets, prior toquery expansion. Once again this run does not include any future data. Due due to theinclusion of the language detection component, however, we consider this to use externalevidence.CLARITY4: this builds upon the CLARITY3 run, but also adds a temporal re-weighting element (as described in Section 4) which downweights tweets that are far fromthe temporal centre of the assumed set of relevant tweets. For this run we set the temporaldecay factor, α, to 2. Similar to CLARITY3 this run uses external data (due to the inclusionof the language classifier). As with all of our runs, no future data is used.It is worth noting that for all runs, having produced a final ranked list (as described foreach run above), finally we truncated each list to the top 30 most relevant tweets beforethese were re-ranked in reverse chronological order.5.1 AnalysisFor this year’s Microblog track precision at 30 (P30) was chosen to be the main evaluationmeasure, and so for each run we report performance based on P30. Figures 1 shows theresults for all relevant and 2 shows the results for highly relevant microblogs. For comparisonpurposes, we include a number of other baselines provided by TREC. The “best” run is apseudo run that we created by taking the union of the top performing of all submitted runsfor each topic. The “median” is also a pseudo run, this time comprising the median runs foreach topic. Finally the “disjunctive baseline” consists of a single run which was generatedusing Lucene2, selecting the most recent 1000 tweets that contain any of the query terms.We can see that the baseline run CLARITY1 outperforms all other runs. For the bothall relevant and highly relevant conditions, using query expansion harms performance. TheCLARITY3 run, however, is an improvement over CLARITY2, showing a minor benefitfrom using language filtering. Comparing CLARITY4 with CLARITY3, we can see thattemporal re-weighting harms performance for the all relevant condition. For the highlyrelevant condition, however, temporal re-weighting gives a small improvement, suggestingthat temporal re-weighting approach may be beneficial, and is worth further investigation.!"!#$"!#%"!#&"!#'"!#("!#)"!#*"!""#$%"%&'()#*+,-.#+,-./01$"+,-./01%"+,-./01&"+,-./01'"23456789:;"<=4;>37;"?;@3=7"<;4A"Figure 1: P30 scores for our oﬃcial runs compared with best and median results from allparticipants as well as a TREC supplied disjunctive baseline (on all relevant results).Figures 3 and 4 show the P30 performance on a topic by topic basis for all relevantand highly relevant tweets respectively. We can see that overall CLARITY1 performs best,although on certain queries the other runs gain higher scores, in particular for the highlyrelevant condition. For a number of queries the runs CLARITY2, CLARITY3 and CLAR-ITY4 perform similarly or identically to the baseline. This is due of our pseudo relevance1http://code.google.com/p/language-detection2http://lucene.apache.org/java/docs/index.html4!"!#!$"!#%"!#%$"!#&"!#&$"!#'"!"#$%&'()%)*+,-'./012'()*+,-.%"()*+,-.&"()*+,-.'"()*+,-./"0123456789":;29<159"=9>1;5":92?"Figure 2: P30 scores for our oﬃcial runs compared with best and median results from allparticipants as well as a TREC supplied disjunctive baseline (on highly relevant results).feedback method, which will result in a diﬀerent number of documents being assumed rel-evant for each topic. If no documents are assumed relevant for a given topic, then queryexpansion and temporal re-weighting will not be used, meaning there will be no diﬀerencefrom the baseline. This approach could be considered reasonably conservative and limitsthe influence of query expansion and temporal re-weighting, although without further inves-tigation it is not clear whether an increased influence for these would increase or decreasethe overall performance.TopicsP300.00.20.40.60.80.00.20.40.60.81.00.00.20.40.60.81.00.00.20.40.60.81.0CLARITY1CLARITY2CLARITY3CLARITY4Figure 3: P30 performance for each run, on a topic by topic basis for all relevant results.5TopicsP300.00.10.20.30.40.50.00.10.20.30.40.00.10.20.30.40.00.10.20.30.4CLARITY1CLARITY2CLARITY3CLARITY4Figure 4: P30 performance for each run, on a topic by topic basis for highly relevant results.6 ConclusionsIn our experiments we investigated a number of diﬀerent approaches for microblog retrieval.None of our runs used future data, while 2 of our runs use external data in the form of anexternal language detection library. Our baseline run CLARITY1 focused on the underlyingterm weightings by ignoring term frequency and document length information, in an attemptto produce a retrieval approach suitable to microblog data. This seems to have been quitesuccessful, achieving high P30 scores for both the all relevant and highly relevant tweets. Theincorporation of query expansion proved to be useful for certain queries, although overall itperformed worse than the baseline run. Finally, the use of temporal re-weighting improvesperformance slightly for the highly relevant condition (CLARITY4 vs CLARITY3). Sincetemporal re-weighting was incorporated alongside with our query expansion component, itcannot be compared directly to the baseline, although we expect that if it was combinedwith the baseline without query expansion then it may lead to an improvement.Acknowledgments.This work is supported by Science Foundation Ireland under grant 07/CE/I114.References[1] K. S. Jones, S. Walker, and S. E. Robertson. A probabilistic model of informationretrieval: development and comparative experiments. In Information Processing andManagement, pages 779–840, 2000.[2] S. E. Robertson. On term selection for query expansion. J. Doc., 46:359–364, December1990.6

CLARITY at the TREC 2011 microblog track

O\u27Hare, Neil

Irish Universities

DCU Online Research Access Service

A probabilistic model of information retrieval: development and comparative experiments.

On term selection for query expansion.

http://doras.dcu.ie/16789/1/clarity_trec2011.pdf

CLARITY at the TREC 2011 microblog track

Abstract

Similar works

Full text

Available Versions

Name not available

Irish Universities

DCU Online Research Access Service