This paper presents Strathclyde iSchool's (SiS) participation in the Technological Assisted Reviews in Empirical Medicine Task. For the ranking task, we explored two ways in which assistance to reviewers could be provided during the assessment process: (i) topic models, where we use Latent Dirichlet Allocation to identify topics within the set of retrieved documents, ranking documents by the topic most likely to be relevant and (ii) relevance feedback, where we use Rocchio's algorithm to update the query model for subsequent rounds of interaction. A third approach combines the topic and relevance feedback to quickly identify the relevant abstracts. For the thresholding task, we apply a score threshold, and exclude documents which did not exceed the threshold given BM25

Azzopardi, Leif

Georgiadis, Georgios

Kalphov, Vassil

University of Strathclyde Institutional Repository

Kalphov, Vassil and Georgiadis, Georgios and Azzopardi, Leif (2017) SiS at CLEF 2017 eHealth tar task. CEUR Workshop Proceedings, 1866. pp. 1-5. ISSN 1613-0073 , This version is available at https://strathprints.strath.ac.uk/62718/Strathprints is  designed  to  allow  users  to  access  the  research  output  of  the  University  of Strathclyde. Unless otherwise explicitly stated on the manuscript, Copyright © and Moral Rights for the papers on this site are retained by the individual authors and/or other copyright owners. Please check the manuscript for details of any other licences that may have been applied. You may  not  engage  in  further  distribution  of  the  material  for  any  profitmaking  activities  or  any commercial gain. You may freely distribute both the url (https://strathprints.strath.ac.uk/) and the content of this paper for research or private study, educational, or not-for-profit purposes without prior permission or charge. Any correspondence concerning this service should be sent to the Strathprints administrator: strathprints@strath.ac.ukThe Strathprints institutional repository (https://strathprints.strath.ac.uk) is a digital archive of University of Strathclyde research outputs. It has been developed to disseminate open access research outputs, expose data about those outputs, and enable the management and persistent access to Strathclyde's intellectual output.SiS at CLEF 2017 eHealth TAR TaskVassil Kalphov1, Georgios Georgiadis1, Leif Azzopardi1vassil.kalphov.2013@uni.strath.ac.uk,georgios.georgiadis.2013@uni.strath.ac.uk, andLeif.Azzopardi@strath.ac.ukUniversity of Strathclyde, Glasgow, UKAbstract. This paper presents Strathclyde iSchool’s (SiS) participationin the Technological Assisted Reviews in Empirical Medicine Task. Forthe ranking task, we explored two ways in which assistance to review-ers could be provided during the assessment process: (i) topic models,where we use Latent Dirichlet Allocation to identify topics within the setof retrieved documents, ranking documents by the topic most likely to berelevant and (ii) relevance feedback, where we use Rocchio’s algorithm toupdate the query model for subsequent rounds of interaction. A third ap-proach combines the topic and relevance feedback to quickly identify therelevant abstracts. For the thresholding task, we apply a score threshold,and exclude documents which did not exceed the threshold given BM25.1 IntroductionCLEF 2017 introduced a new eHealth retrieval problem - that of providing tech-nological assistance to reviewers of systematic reviews - where the goals of thetask were to explore how Information Retrieval techniques could be used to: (i)identify relevant material more quickly in the ranking challenge and (ii) identifywhen reviewers could stop processing documents in the thresholding challenge [3,2]. During the review process, reviewers will routinely examine hundreds to thou-sands of abstracts to decided if the document (and evidence it contains) couldbe included in the systematic review that they are conducting [5]. Once theyhave identified a subset of abstracts, which are potentially relevant, they exam-ine the document’s contents to decide whether the document should be includedor excluded. The track focused on the first part, identifying potential relevantdocuments during the, so called, screening phase.In this work we considered two different approaches - one which uses topicmodelling and the other which uses relevance feedback. In selecting these ap-proaches we thought that such techniques could be used in the following way.For topic modelling, we envisaged that the download abstracts could be seman-tically clustered - and the different clusters could be presented to the reviewer -the reviewer could then start the review process by selecting a cluster that theyfelt was most likely to contain the relevant documents. Since we did not haverecourse to reviewers, we explored a number of different ways to automaticallyselect the best cluster. For relevance feedback, we envisaged that as the reviewerstarts to examine documents, the query could be updated to bring back the nextmost relevant documents, so that they would quickly find all the relevant mate-rial as soon as possible. Obviously, if the aim is to reduce the workload of thereviewers, then we need to be able to select a point where the reviewer can stopassessing documents - however, this runs the risk of losing relevant documents.To this end, we explore various heuristics to select the threshold such that weminimize effort and maximise recall (but ideally obtain total recall).2 Experimental Set-upData: Given the list of topic descriptions the PubMed IDs were extracted, anda scripted fetched the Abstract and associated Metadata from the PubMed API.From the topics, we extracted the title for each topic and use that as the query.Indexing and Retrieval System: We used Lucene 6.2 to create a separateindex for each of the topic (where stop words were removed, no stemming wasapplied). A Lucene Document was created where the following fields were index:Title, Abstract, Author, and Publication Name. The baseline retrieval algorithmwe employed was fielded BM25 with standard parameters settings i.e. b = 0.75,and equal weights between fields (denoted as BM25).Relevance Feedback: We implemented Rocchio’s Algorithm [4] in Lucene -where feedback was used to provide relevance information. In each round offeedback, 30 documents were examined, and the query model updated, to pro-vide a re-ranking of the subsequent documents. This was performed on the first10%, 20%, etc of documents associated with the topic. Here, we only reportthe 30% runs (AL30) as these generally performed the best and little change inperformance was observed on the training set using more feedback.Topic Modelling: We used MALLET toolkit, and thus Latent Dirichlet Allo-cation [1] to semantically cluster the documents within each topic. We set thenumber of latent topics to 5, and α =. To rank the documents we selected one ofthe latent topics zi, and ordered the documents by the probability p(zi|d|). In anattempt to select the cluster that provides the best ranking, we took the BM25ranking from above, and use the top 100 ranked documents as pseudo-relevancefeedback. Then we ranked each latent topic (given the ordering by p(zi|d|)) andselect the one with the highest overlap with BM25 (TMBM).Combined: Since our topic modelling approaches do not use feedback, we de-cided to see whether we could start the process of relevance feedback using thetopic modelling run, and refine the query model accordingly. Thus, we selectedthe best performing method TMAL, given the training data, and used this asstarting point for the active learning. Again we considered obtained feedback forthe first 10%,20% and 30% of the documents per topic.Thresholded Runs To create thresholded runs, we took the BM25 run andapplied a simple score based threshold. Using Lucene the scores for a querygiven BM25 are from 1.0 or greater, so we used thresholds 1.0, 1.5, 2.0 and2.5. This led to a reasonable reduction in the number of documents withoutsacrificing much recall.3 Results and DiscussionTables 1 and 3 report the performance on the Ranking Task, while Tables 2and 4 report the performance on the Threshold Task. Our best performing runon the Ranking Task, in terms of normalized area under the gain curve, wasAL30, which used 30% of the documents as feedback. From our results, it isclear that the topic modelling approach, as we have employed it, has not leadto significantly better improvements over the BM25 baseline. However, wheninspecting the individual topic based ranking, the most probably topic, was notalways the best performing topic. So we will direct more research into topicselection. This is because, when formulating queries, reviewers could use topicmodelling to understand the space of documents retrieved, and then refine theirquery further - and thus make savings a priori rather than have to trawl throughhundreds and thousands of documents. Another factor that could significantlyimprove our results is that for these initial runs we used the title as the query,as opposed to the boolean query provided within the topics. It is quite possiblethat the more complex and verbose boolean queries could lead to a better initialranking - and so when used in conjunction with relevance feedback the relevantitems could be found sooner. We leave these directions for further work.Table 1: Results on Training Data for Ranking TaskRun Random BM25 AL30 TMBM TMAL30NumRels 2494 2494 2494 2494 2494NumFeed 0 0 44939 0 44940RelsFound 2494 2494 2494 2494 2494AP 0.044 0.171 0.273 0.15 0.24MinRel 6947 5174 3210 4221 2835WSS100 0.05 0.25 0.33 0.30 0.37Area 0.51 0.83 0.91 0.80 0.90NCG10 0.07 0.30 0.41 0.30 0.41NCG20 0.18 0.51 0.73 0.57 0.73NCG30 0.29 0.67 0.86 0.74 0.86TotalCost 7453 7453 11947 7453 11947TotalCostUniform 7453 7453 11947 7453 11947TotalCostWeighted 7453 7453 11947 7453 11947loss er 0.38 0.38 0.38 0.38 0.38loss r 0.0 0.0 0.0 0.0 0.0loss e 0.38 0.38 0.38 0.38 0.38References1. Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. Journal of MachineLearning Research 3, 993–1022 (2003)Table 2: Results on Training Data for Threshold TaskRun BM25 T1 T1.5 T2 T2.5NumRels 2494 2949 2494 2494 2494NumFeed 0 0 0 0RelsFound 2494 2439 2419 2318 2187AP 0.17 0.171 0.17 0.17 0.17MinRel 5174 4531 3930 3453 3236WSS100 0.25 0.23 0.22 0.19 0.19Area 0.83 0.83 0.83 0.82 0.80NCG10 0.30 0.30 0.30 0.30 0.30NCG20 0.51 0.51 0.51 0.51 0.51NCG30 0.67 0.67 0.67 0.67 0.67TotalCost 7453 6010 5391 4700 4144TotalCostWeighted 7453 6811 6978 6911 6985TotalCostUniform 7453 6049 5482 5008 4714loss er 0.38 0.30 0.24 0.20 0.16loss r 0.0 0.0 0.001 0.006 0.016loss e 0.38 0.29 0.24 0.19 0.15Table 3: Results on Test Data for Ranking TaskRun Random BM25 AL30 TMBM TMAL30NumRels 1857 1857 1857 1857 1857NumFeed 0 0 35730 0 35432RelsFound 1857 1857 1857 1857 1857AP 0.05 0.17 0.22 0.12 0.16MinRel 3722 2851 2290 3124 2305WSS100 0.04 0.29 0.41 0.27 0.40Area 0.48 0.81 0.86 0.73 0.84NCG10 0.09 0.45 0.62 0.31 0.54NCG20 0.19 0.65 0.79 0.55 0.77NCG30 0.28 0.75 0.88 0.68 0.86TotalCost 3918 3918 6300 3918 6280TotalCostUniform 3918 3918 6300 3918 6280TotalCostWeighted 3918 3918 6300 3918 6280loss er 0.54 0.54 0.54 0.54 0.54loss r 0.00 0.00 0.00 0.00 0.00loss e 0.54 0.54 0.54 0.54 0.542. Goeuriot, L., Kelly, L., Suominen, H., Ne´ve´ol, A., Robert, A., Kanoulas, E., Spijker,R., Palotti, J., Zuccon, G.: CLEF 2017 eHealth evaluation lab overview. In: CLEF2017 - 8th Conference and Labs of the Evaluation Forum, Lecture Notes in ComputerScience (LNCS). Springer (September 2017)3. Kanoulas, E., Li, D., Azzopardi, L., Spijker, R.: echnologically assisted reviews inempirical medicine. In: CLEF 2017 Evaluation Labs and Workshop: Online WorkingNotes, CEUR-WS (2017)4. Rocchio, J.J.: Relevance feedback in information retrieval (1971)Table 4: Results on Test Data for Threshold TaskRun BM25 T1 T1.5 T2 T2.5NumRels 1857 1857 1857 1857 1857NumFeed 0 0 0 0 0RelsFound 1857 1828 1809 1784 1758AP 0.17 0.17 0.17 0.17 0.17MinRel 2851 2503 2333 2068 1877WSS100 0.29 28 0.27 0.23 0.22Area 0.81 0.81 0.80 0.80 0.79NCG10 0.45 0.45 0.45 0.45 0.45NCG20 0.65 0.65 0.65 0.65 0.65NCG30 0.75 0.75 0.75 0.75 0.75TotalCost 3918 3435 3165 2824 2536TotalCostUniform 3918 3786 3865 3748 3902TotalCostWeighted 3918 3454 3280 3117 2905loss er 0.54 0.54 0.38 0.33 0.27loss r 0.00 0.001 0.005 0.01 0.014loss e 0.54 0.54 0.38 0.32 0.265. Shemilt, Khan, Park, Thomas: Use of cost-effectiveness analysis to compare theefficiency of study identification methods in systematic reviews. Systematic reviews(2016)

SiS at CLEF 2017 eHealth tar task

Abstract

Similar works

Full text

Available Versions

University of Strathclyde Institutional Repository