This paper describes the details of our participation in expert search task of the TREC 2007 Enterprise track. The presented study demonstrates the predicting potential of the expertise evidence that can be found outside of
the organization. We discovered that combining the ranking built solely on the Enterprise data with the Global Web
based ranking may produce significant increases in performance. However, our main goal was to explore whether
this result can be further improved by using various quality measures to distinguish among web result items. While,
indeed, it was beneficial to use some of these measures, especially those measuring relevance of URL strings and titles,
it stayed unclear whether they are decisively important

Aly, R.B.N.

Hiemstra, D.

Serdyukov, P.

English

This paper describes the details of our participation in expert search task of the TREC 2007 Enterprise track. The presented study demonstrates the predicting potential of the expertise evidence that can be found outside of the organization. We discovered that combining the ranking built solely on the Enterprise data with the Global Web based ranking may produce significant increases in performance. However, our main goal was to explore whether this result can be further improved by using various quality measures to distinguish among web result items. While, indeed, it was beneficial to use some of these measures, especially those measuring relevance of URL strings and titles, it stayed unclear whether they are decisively important

Serdyukov, Pavel

Hiemstra, Djoerd

Aly, Robin

NARCIS 

Using the Global Web as an Expertise Evidence Source

University of Twente Research Information

University of Twente at the TREC 2008 Enterprise Track:Using the Global Web as an expertise evidence sourcePavel Serdyukov, Robin Aly, Djoerd HiemstraUniversity of TwentePO Box 217, 7500 AEEnschede, The Netherlands{serdyukovpv, alyr, hiemstra}@cs.utwente.nlABSTRACTThis paper describes the details of our participation in ex-pert search task of the TREC 2007 Enterprise track.1. INTRODUCTIONThis is the fourth (and the last) year of TREC 2007 En-terprise Track and the second year the University of Twente(Database group) submitted runs for the expert finding task.In the methods that were used to produce these runs, wemostly rely on the predicting potential of those expertiseevidence sources that are publicly available on the GlobalWeb, but not hosted at the website of the organization understudy (CSIRO). This paper describes the follow-up studiescomplimentary to our recent research [8] that demonstratedhow taking the web factor seriously significantly improvesthe performance of expert finding in the enterprise.2. EXPERTISE EVIDENCE ACQUISITIONFROM THE GLOBAL WEBOne could imagine an expert finder that is equipped witha web crawler focusing on retrieval of employee-specific in-formation from the Web. Such a spider would provide uswith a plenty of information about how the organization ispositioned at the world or regional markets, how influentialand wide-spread its organizational knowledge. However, incase when an expert finder should be made cheap but good,the enterprise may rely on powerful mediators between peo-ple and the Web: leading search engines and their publicsearch APIs.In our latest studies [8] we found that extracting topic-and person-specific information with Yahoo! and GoogleSearch APIs is a universal way to expand the search scope ofexpert finders. We used as many expertise evidence sourcesas possible to finally aggregate ranks from several source-specific rankings per each candidate. We relied on the hy-pothesis that real experts should be popular not only locally,in the enterprise, but also in the other web spaces availablefor search: news, blogs, academic libraries etc. We extractedexpertise evidence from search engines by issuing queries foreach candidate containing:• the quoted full person name: e.g. “tj higgins”,• the name of the organization: csiro,• query terms without any quotes: e.g. genetic modifi-cation,• the directive prohibiting the search at the organiza-tional web site: -inurl:csiro.au.Adding the organization’s name was important for theresolution of an employee’s name, the clause restricting thesearch to URLs that do not contain the domain of the or-ganization separated organizational data from the rest ofavailable information (one could also enlist all organizationaldomains, each in separate -inurl clause). As the second stepof acquiring the evidence of a certain type, we send the queryto a web search service and regard the number of returnedresults as a measure of personal expertness. Due to the lim-its of the Search Engine API technology we used, we had torestrict the number of persons for which we extracted globalexpertise evidence: it was unrealistic and unnecessary to is-sue thousands of queries containing each person for eachquery provided by a user. So, making an initial expert find-ing run on enterprise data was a requirement. As a resultof that run, we used 100 most promising candidate experts(actually, the maximum number of candidates per query al-lowed for a single TREC submission) for the further analysis.Apart from the ranking built on fully indexed organizationaldata, we built rankings using 6 different sources of expertiseevidence from the Global Web: Global Web Search, Re-gional Web Search, Document-specific Web search, NewsSearch (all via Yahoo! Web search API), Blogs Search andBooks Search (via Google Blog and Book Search APIs).Our experiments demonstrated a substantial increase in per-formance when we used combinations of up to three rank-ings. The best combination was comprised of the Enterprise,Global Web and News based rankings.Despite that the main idea was to combine various rank-ings, we used obviously naive measure of expertness. Thatis why in the present work, we focus on combination of onlytwo rankings, Enterprise and Global Web based, but use var-ious measures of quality of web results returned by YahooGlobal Web Search API in response to the above describedqueries. Some of statistics per URL (the domain size andthe number of inlinks) are still extracted by means of GoogleWeb Search API, since it has no limit on the number ofqueries per user IP and it was a decisive factor to completeour experiments in time.3. MEASURING THE QUALITY OF A WEBSEARCH RESULTAfter all the majority of expert finding approaches is basedon measuring the quality of a person-specific result set re-turned by the search engine in response to a query. Person-specific means that it contains only those documents thathave at least one mention of the certain person and its qual-ity may be represented by various features: the number ofdocuments it contains or the sum of their relevance proba-bilities. A result set returned by a typical web search engineconsists of a list of result items described by their URLs, ti-tles and summaries (snippets). To measure the overall qual-ity of a web search result, we should aggregate calculatedquality measures for all or top-k result items:Expertise(e) =∑Item∈WebResultSetQuality(Item) (1)Certainly, downloading web pages using URLs of web re-sult items for the deeper analysis of web result quality maylead to the better perfomance, but in our experiments werestrict ourselves to quality measures calcualted just fromthe search result pages or using such page statistics thatcan be quickly acquired from a search engine without down-loading the full content of a page. All measures that weconsidered in this paper could be classified into two types:query-dependent and query-independent.3.1 Query independent quality measuresIn our experiments we focused on four kinds of query in-dependent quality measures of a result item (web page).3.1.1 URL lengthPrevious studies indicated that URL length is inverselyproportional to the usefulness of the page it refers to [5, 3].We apply simple quality measure based on this assumption:Quality(Item) = 1/√Length(ItemURL). The URL lengthis expressed in levels: the number of backslashes in the URLafter its domain part. It should be mentioned that express-ing the URL length in symbols performed much worse in ourpreliminary experiments.3.1.2 Inlinks for domainAnother quality estimate we used is an approximation ofthe result item’s authority. Since it was impossible to calcu-late sophisticated web graph centrality measures and sincepages themselves are not often linked by pages outside oftheir domain, we used a simple inlink authority measurefor the domain of the result item, considering that in manyother authority measures (e.g. Pagerank) this value anywaypropagates to all pages hosted at the result item’s domain:Quality(Item) = Inlinks(Domain(Item)). The authorityestimate was acquired using the link: clause plus the do-main name to query Google Web Search API that returnedthe number of pages citing the given domain.3.1.3 Domain sizeWe also supposed that the importance of the domain whichhosts the returned result page should also be expressed by itssize: Quality(Item) =√Size(Domain(Item)) The mainintuition was that large domains usually become so only dueto the time and money spent on their maintenance what inturn demonstrates their respectability. The size estimatewas acquired using site: clause plus the domain name toquery Google Web Search API that returned the number ofpages indexed by Google at the given domain.3.1.4 FreshnessWe supposed that a page’s last date of modification showshow much trust we should put in expertise evidence foundin it. Obviously, the freshness of expertise evidence implic-itly indicates the freshness of candidate’s expert knowledge.In our preliminary experiments it appeared that consider-ing only those results that where at least once modified (orcreated) after 2006 was better than just treating all of themequally useful:Quality(Item) ={1, Y ear(Item) ≥ 20060, Y ear(Item) < 20063.2 Query dependent quality measuresThe state-of-the-art methods, including one that we useto get Enterprise based ranking [2], often rank candidates bythe sum of relevance probabilities of pages that contain theirmentions. Since it is very time- and broadband-consumingto download all pages in the result list in order to measuretheir relevance, we use a very simple measure of an Item’s(URL, Title or Summary) relevance which we sum over theresult list:Quality(Item) =N(q, q ∈ Item ∧ q ∈ Q)N(q, q ∈ Q) (2)what is the number of query terms q appearing in the resultItem divided by the number of terms in the query Q. Sinceit is hard to tokenize URLs, we just search for a query termas for a substring in this case.4. RANK AGGREGATIONThe problem of rank aggregation is well known in researchon metasearch [6]. Since our task may be viewed as peoplemetasearch, we adopt solutions from that area. In our previ-ous experiments with different rank aggregation methods wefound that the simplest approach is also the best performing[8]. To get the final score we just summed the negatives ofranks for a person from each source to sort them in descend-ing order:Expertise(e) =K∑i=1−Ranki(e) (3)This approach is often referred as Borda count [1]. In ourprevious work we just sorted all candidates’ expertise esti-mates for each evidence source to get their source-specificranks. In this work we assigned these ranks more smoothly.First, we considered that all candidates with zero expertiseestimates are always assigned with the lowest negative rankpossible in the system (-100 in our experiments, since wealways start by taking top-100 candidates from the Enter-prise based ranking). Second, we assigned equal ranks tothe candidates with equal expertise estimates, since beforethey were given arbitrary ranks by the sorting algorithm.5. EXPERIMENTSThe CERC collection was indexed by Lucene retrieval en-gine using Snowball stemmer at the text parsing stage. Forthe purpose of finding candidate experts, we extracted allemail addresses from the collection with csiro.au domainand firstname.lastname-like first part. We also had a list ofemail addresses to be banned which were not personal, butorganizational addresses (e.g. publishing.photos@csiro.au).After all, we had 3500 candidate experts in total. Later, inorder to find an association between a candidate and a doc-ument, we searched for the candidate’s full email address orRanking MAP MRR P@5Enterprise 0.362 0.508 0.220Enterprise +WebNumOfResults 0.485 0.627 0.256WebURLLenInLevels 0.386 0.532 0.216WebInlinksForDomain 0.477 0.632 0.252WebSizeForDomain 0.477 0.604 0.248WebAfter2006 0.491 0.620 0.256WebRelevURL 0.501 0.650 0.26WebRelevTitle 0.488 0.634 0.26WebRelevSummary 0.485 0.627 0.252Table 1: The performance of TREC 2007 queriesfull name in the document’s text. For each TREC title querywe retrieved 50 documents (using a language model basedretrieval model [2]) that contained at least one candidateexpert mentioned. Then we analyzed these documents withthe state-of-the-art Enterprise based expert finding method(Balog’s candidate-centric Model 2 described in [2] and usedin our previous experiments with web expertise evidence [8]).Finally, we considered only top 100 candidates from the En-terprise based ranking and built Web based rankings onlyfor those.The results analysis is based on calculating popular IRperformance measures also used in official TREC evalua-tions: Mean Average Precision (MAP), precision at top5 ranked candidate experts (P@5) and Mean ReciprocalRank (MRR). We analyzed the performance of the Enter-prise based ranking combined with one of the following rank-ings:• WebNumOfResults: based on the number of webresult items returned,• WebURLLenInLevels: based on the sum of URLLength based quality estimates for web result items,• WebInlinksForDomain: based on the sum of inlinksof domains of web result items,• WebSizeForDomain: based on the sum of sizes ofdomains of web result items,• WebAfter2006: based on the number of web resultitems modified or created after 2006,• WebRelevURL: based on the sum of URL relevanceprobabilities for web result items,• WebRelevTitle: based on the sum of title relevanceprobabilities for web result items,• WebRelevSummary: based on the sum of summaryrelevance probabilities for web result items,Our initial intention was to improve the combination ofthe Enterprise and the Enterprise+WebNumOfResultsrankings that we regarded as our baseline (see Table 1).Only the WebURLLenInLevels ranking showed signifi-cantly degraded performance, the others were equally orbetter performing. Three rankings appeared to have slightlybetter performance in combination with the Enterpriserankings: WebAfter2006, WebRelevTitle, WebRele-vURL. We also tried to further combine different rankingsfrom the above list. However, we did not succeed to beat theWebRelevURL’s ranking performance with any of thesecombinations.Ranking MAP MRR P@5Enterprise +WebNumOfResults 0.371 0.740 0.469WebAfter2006 0.370 0.743 0.458WebRelevURL 0.373 0.765 0.487WebRelevTitle 0.371 0.754 0.480Table 2: The performance of TREC 2008 queriesWe finally submitted combinations of the Enterprise rank-ing with WebNumOfResults, WebAfter2006, WebRe-levTitle, and WebRelevURL rankings as runs to TREC2008 (see Table 2). The only difference with experimentswith TREC 2007 queries is that we used our own infiniterandom walk based expert finding method [9] to build theEnterprise ranking. In this case all methods were equallyeffective according to MAP measure, but according to MRRand P@5 measures, considering relevance of URLs was in-deed beneficial.6. RELATED WORKThe usefullness of query-independent document qualitymeasures for expert finding was recently studied. MacDon-ald et. al. [7] reported a bit different findings for the enter-prise data only (e.g. all inlinks are only from pages of thesame domain): they used similar expert finding method asa baseline and using Inlinks and URL length improved itsMAP by a few percents. Similar document quality measuresfor document retrieval task can be found in some groups’ re-ports on TREC Enterprise Track 2007 [12, 4, 11]. Measur-ing the quality of web result set to predict users’ satisfactionwith a search engine was just proposed by White et. al. [10].7. CONCLUSIONSThe presented study demonstrates the predicting poten-tial of the expertise evidence that can be found outside ofthe organization. We discovered that combining the rank-ing built solely on the Enterprise data with the Global Webbased ranking may produce significant increases in perfor-mance. However, our main goal was to explore whetherthis result can be further improved by using various qual-ity measures to distinguish among web result items. While,indeed, it was beneficial to use some of these measures, espe-cially those measuring relevance of URL strings and titles,it stayed unclear whether they are decisively important.There still stays a number of parallel directions to follow.First, various normalization and smoothing techniques couldbe applied to the URL quality measures we used. How-ever, it seems more promising to apply machine learningmechanisms to find out which quality features of a web re-sult item are the most important and how to combine theminto a powerful expertise prediction model. Other sources ofweb expertise evidence besides Global Web should also notbe overlooked: blog features (e.g. number of subscribers)when using Blog search based evidence or publication fea-tures (e.g. publisher’s authority or a citation index) whenusing academic search services.8. ACKNOWLEDGEMENTSWe want to sincerely thank Henning Rode for the helpwith the collection preprocessing and series of exciting dis-cussions.9. REFERENCES[1] J. A. Aslam and M. Montague. Models formetasearch. In SIGIR ’01: Proceedings of the 24thannual international ACM SIGIR conference onResearch and development in information retrieval,pages 276–284, New York, NY, USA, 2001. ACM.[2] K. Balog, L. Azzopardi, and M. de Rijke. Formalmodels for expert finding in enterprise corpora. InSIGIR ’06, pages 43–50, 2006.[3] N. Craswell, S. Robertson, H. Zaragoza, andM. Taylor. Relevance weighting for query independentevidence. In SIGIR ’05: Proceedings of the 28thannual international ACM SIGIR conference onResearch and development in information retrieval,pages 416–423, New York, NY, USA, 2005. ACM.[4] H. Duan, Q. Zhou, Z. Lu, O. Jin, S. Bao, Y. Cao, andY. Yu. Research on enterprise track of trec 2007 atsjtu apex lab. In Proceeddings of the 15th TextREtrieval Conference (TREC 2007), 2007.[5] W. Kraaij, T. Westerveld, and D. Hiemstra. Theimportance of prior probabilities for entry page search.In SIGIR ’02: Proceedings of the 25th annualinternational ACM SIGIR conference on Research anddevelopment in information retrieval, pages 27–34,New York, NY, USA, 2002. ACM.[6] Y.-T. Liu, T.-Y. Liu, T. Qin, Z.-M. Ma, and H. Li.Supervised rank aggregation. In WWW ’07:Proceedings of the 16th international conference onWorld Wide Web, pages 481–490, New York, NY,USA, 2007. ACM.[7] C. Macdonald, D. Hannah., and I. Ounis. High qualityexpertise evidence for expert search. In Proceedings of30th European Conference on Information Retrieval(ECIR08), 2008.[8] P. Serdyukov and D. Hiemstra. Being omnipresent tobe almighty: The importance of the global webevidence for organizational expert finding. InFCHER’08: Proceedings of the SIGIR’08 Workshopon Future Challenges in Expertise Retrieval, 2008.[9] P. Serdyukov, H. Rode, and D. Hiemstra. Modelingmulti-step relevance propagation for expert finding. InCIKM ’08, Napa Valley, USA, 2008.[10] R. W. White, M. Richardson, M. Bilenko, and A. P.Heath. Enhancing web search by promoting multiplesearch engine use. In SIGIR ’08: Proceedings of the31st annual international ACM SIGIR conference onResearch and development in information retrieval,pages 43–50, New York, NY, USA, 2008. ACM.[11] M. Wu, F. Scholer, M. Shokouhi, S. Puglisi, andH. Ali. Rmit university at the trec 2007 enterprisetrack. In Proceeddings of the 15th Text REtrievalConference (TREC 2007), 2007.[12] J. Zhu, D. Song, and S. Rger. The open university attrec 2007 enterprise track. In Proceeddings of the 15thText REtrieval Conference (TREC 2007), 2007.

Using the Global Web as an Expertise Evidence Source

Abstract

Similar works

Full text

Available Versions

NARCIS

University of Twente Research Information