DI NUNZIO, GIORGIO MARIA

MARCHESIN, STEFANO

English

Di Nunzio, Giorgio Maria

Marchesin, Stefano

Padua Research Archive

The University of Padua IMS Research Group at CENTRE@TREC 2018

Giorgio Maria Di Nunzio

Stefano Marchesin

Archivio istituzionale della ricerca - Università di Padova

The University of Padua IMS Research Group atCENTRE@TREC 2018Giorgio Maria Di Nunzio, Stefano MarchesinDepartment of Information EngineeringUniversity of Padua, Italy{giorgiomaria.dinunzio, stefano.marchesin}@unipd.itAbstract. In this paper, we present our participation in one of the tasksof the CENTRE@TREC 2018 Track: the Clinical Decision Support task.We describe the steps of the original paper we wanted to reproduce,identifying the elements of ambiguity that may affect the reproducibilityof the results.The experimental results we obtained follow a similar trend to those pre-sented in the original paper: using clinical trials’ “note” field decreasesthe retrieval performances significantly, while the pseudo-relevance feed-back approach together with query expansion achieves the best resultsacross different measures.In the experimental results we find out that the choice of the stoplistis fundamental to achieve a reasonable level of reproducibility. However,stoplist creation is not described sufficiently well in the original paper.Keywords: Precision medicine, query expansion, reproducibility1 IntroductionThe CENTRE@TREC challenges TREC participants to reproduce the resultsof the most interesting systems submitted in previous editions of TREC. Trackorganizers select these top performing experiments and each participating groupis challenged to replicate and/or reproduce one or more of the selected systemsby only using standard open source IR systems [4].In 2018, CENTRE@TREC had three tasks: replicating runs of the TREC2016 Clinical Decision Support track; replicating runs of the TREC 2013 andTREC 2014 Web track. In this paper, we focus on the first task: the retrievalof biomedical articles relevant for answering clinical questions based on patientrecords. The target document collection contains 1,251,954 full-text articles, asnapshot of the Open Access Subset of PubMed Central (PMC) from March28, 2016. The list of topics comprises 30 clinical notes with the following struc-ture: summary, description and note. The paper to reproduce combines pseudo-relevance feedback and semantic query expansion [3]. In our experiments, wewere able to partially reproduce three of the four runs described in the originalpaper.2 G.M. Di Nunzio, S. MarchesinThis paper is organized as follows. Section 2 lists elements of the originalpaper that may be ambiguous or not clear from an experimental point of view.Section 3 describes our experimental setting and the choices that we made inorder to reproduce the results. Section 4 analyzes and compares the results ofthe original paper with the results of our runs. Finally, Section 5 concludes thepaper with some general comments.2 Analysis of the MethodsBelow, we analyze the description of the model components from the originalpaper. First, a quote from the paper is reported to highlight the component.Then, a description of how we reproduced — or we tried to reproduce — thecomponent follows.The document collection for 2016 consists of 1,251,954 PMC articles inNXML format that were indexed with Solr (version 5.5.2).In our experiments, we used the latest version of Solr (version 7.5.0). One thingthat is unclear is whether the authors have used the Data Import Handler (DIH)of Solr to import and index the content. In our case, we decided to parse thecollection into a Solr XML data structure that can be used with a curl commandto update the index.Fields used for indexing were PMCID, title, abstract, body, conclusion,journal title and journal type. Indexing was performed with standard Solrsettings which include tokenization, stemming and stopword removal. Amaster stopword list was constructed which is a combination of standardEnglish stopwords and a list of stopwords constructed manually basedon most frequent terms occurring in the document collection.This is a crucial point in the description of the model. There is no field named“PMCID”. We believe that the “pmc” field of the article metadata is the correctmapping. Besides, the most problematic field is “conclusion” as there is no ex-plicit field named conclusion. Therefore, in our experiments, we used pmc, title,abstract, body, journal title and journal type as fields.Furthermore, the lack of a detailed description for the stoplist used makes thereproduction of the results extremely difficult. According to [2], “the most promi-nent effects are those of stop lists and IR models, as well as their interactions,while stemmers and n-grams play a smaller role”. Following this consideration,we used three ‘standard’ stoplists (described in the experimental section) andwe tried to find the optimal number of ‘most frequent terms’ that were manuallyadded to the list.Okapi BM25 was the similarity measure used for querying and retrievalfrom the index.IMS @ TREC PM 2018 3We used the BM25 model implemented in Solr 7.5.0. According to the Solrdocumentation, the implementation of the BM25 model for version 5.5.2 (originalpaper) and version 7.5.0 (our implementation) is the same.For query expansion, the authors of the original paper used MetaMap [1]:The MetaMap program was applied for the identification of UMLS con-cepts in topics. Since UMLS has over 100 semantic types, mappings wererestricted to only the following semantic types [...] For all the mappedconcepts, synonyms were extracted from UMLS knowledge sources whichwere thereafter used for query expansion.We used MetaMap in the same way described in the original paper,Finally, the description of the Pseudo-Relevance Feedback approach is asfollows:For a given query, Pseudo-Relevance Feedback (PRF) was implementedto collect words from titles of top k retrieved documents. Stopwordswere eliminated and the remaining words were added to the initial queryin order to generate a new query that can be reused for searching andretrieval. [...] Although, the value of k had only minor impact on retrievalperformance, k = 30 was chosen which was observed to deliver the bestresults. For the runs that leveraged on UMLS query expansion, terms inthe topic after stopword removal in combination with UMLS terms (i.e.expanded queries) were used as initial query.The paper is clear on the number of k documents to use for PRF. However,the lack of description for the stoplist used — especially regarding the manualselection of frequent medical terms — makes it hard to replicate the performancesof PRF.3 Experimental SettingWe tried to reproduce the following steps from the original paper: pre-processing,indexing and query expansion. A description of each step is presented below.3.1 Pre-processingIn order to parse the collection of documents, we created an XML file containingthe schema used by the update handler of Solr for adding documents to theindex. The structure of the file is shown in Listing 1.1, and the code used toproduce all the XML files is openly available.1<?xml version="1.0"?><add><doc>1 https://github.com/gmdn/CENTRE-TREC20184 G.M. Di Nunzio, S. Marchesin<field name="pmc">1064073</field><field name="title">Role of Notch signaling [...]</field><field name="abstract">Introduction Notch signaling has [...]</field><field name="body">Introduction Stem cells in adult tissues [...]</field><field name="journal-title">Breast Cancer Research</field></doc><doc>[...]</doc></add>Listing 1.1: Example of XML file indexed by Solr.3.2 IndexingFor indexing, we tried three different stoplists: the default Solr stoplist (33terms), the SMART stoplist (571 terms)2, the SMART stoplist together with alist of the 300 most frequent words3 in the collection . The union of the SMARTstoplist with the most frequent words produced a stoplist of 731 words. Wename the three variants in the experimental results as: Lucene, SMART, andSMART+.The stoplists can be found at the GitHub repository.43.3 Query ExpansionFor query expansion, we followed the steps described in the paper. We usedMetaMap 5 to identify UMLS concepts for each query and for each title of thefirst 30 documents retrieved. First, we restricted the mappings of UMLS conceptsto only those semantic types listed in [3]. Then, we extracted all the synonymsof the matched concepts.4 Comparison of ResultsIn this section, we analyze the results obtained with our experiments by compar-ing the impact of the three stoplists with the original runs. We replicated threeout of four runs of the original paper: MRKPrfNote, MRKSumCln, MRKUmlsSolr.In Tables 1a and 1b, we report the values of the infNDCG and infAP measuresobtained in [3], along with the performances of each of our three runs based onthe different stoplists (i.e. Lucene, SMART and SMART+). We also add theperformance of each run in terms of Rprec in Table 1c.The results we obtained are close to those reported in the original paper.Regardless of the stoplist used, we found that:2 http://members.unine.ch/jacques.savoy/clef/3 We used the Solr Terms Component to extract the most frequent terms.4 https://github.com/gmdn/CENTRE-TREC20185 https://metamap.nlm.nih.govIMS @ TREC PM 2018 5infNDCGrun original Lucene SMART SMART+MRKPrfNote 0.150 0.125 0.133 0.138MRKSumCln 0.222 0.187 0.197 0.202MRKUmlsSolr 0.226 0.190 0.205 0.199(a) infNDCG valuesinfAPrun original Lucene SMART SMART+MRKPrfNote 0.018 0.012 0.014 0.014MRKSumCln 0.027 0.021 0.023 0.024MRKUmlsSolr 0.029 0.022 0.024 0.024(b) infAP valuesRPrecrun original Lucene SMART SMART+MRKPrfNote 0.102 0.084 0.093 0.093MRKSumCln 0.156 0.144 0.147 0.147MRKUmlsSolr 0.168 0.151 0.153 0.156(c) RPrec valuesTable 1: A comparison of different performance measures among the originalruns and the reproduced runs with different stoplists.• Using the “note” field of clinical trials decreases the performance significantly(MRKPrfNote run);• The query expansion-based approach (MRKSumCln run) increases perfor-mance significantly over the plain BM25 approach (the values of the perfor-mance for plain BM25 are not presented in the original paper nor in thispaper);• The PRF approach (MRKUmlsSolr run) increases the performance over thesimple query expansion approach (MRKSumCln run).A deeper analysis shows that we were not able to match the scores of theoriginal paper. However, by increasing the size of the stoplist we were able toget closer and closer to the values reported in the original paper.In Figure 1, we show a topic by topic analysis of the difference between thereproduced run MRKUmlsSolr and the original one for the three performancemeasures. A negative value (red bars below zero) means that the original runperformed better than the reproduced run. For most of the topics, the differencein the inferred measures, infNDCG in Figure 1a and infAP in 1b, is in favourof the original run. On the other hand, for RPrec we observe a more evenlydistributed difference across topics. A paired t-test confirms that there is nostatistical significant difference between the two runs (the reproduced run andthe original one).6 G.M. Di Nunzio, S. Marchesin5 ConclusionsIn this paper, we presented the results of our participation in the CENTRE@TREC2018 Track. Our objective was the reproduciblity of a paper aiming to retrievebiomedical articles relevant for answering clinical questions based on patientrecords. In particular, we wanted to reproduce the query expansion and pseudo-relevance feedback components.We analyzed the steps of the original paper we wanted to reproduce, and wedescribed the elements of ambiguity that may affect the reproducibility of theresults. We provided a detailed list of the choices we made for each step and, inparticular, the selection of terms for the stoplist.The experimental results we obtained follow a similar trend to those of theoriginal paper: using the “note” field of clinical trials decreases the retrievalperformances in a significant way, while pseudo-relevance feedback and queryexpansion obtain the best results across different measures.We were able to get close to the performances of the best run reported in [3] byincreasing the number of terms in the stoplist. The results showed that the bestresults for some performance measures (such as the inferred NDCG and inferredAverage Precision) are more difficult to achieve compared to other measures(such as RPrec).References1. Aronson, A.R., Lang, F.M.: An overview of metamap: historical perspective andrecent advances. Journal of the American Medical Informatics Association : JAMIA17(3), 229–236 (May-Jun 2010). https://doi.org/10.1136/jamia.2009.002733,https://www.ncbi.nlm.nih.gov/pubmed/204421392. Ferro, N., Silvello, G.: Toward an anatomy of IR system component performances.JASIST 69(2), 187–200 (2018). https://doi.org/10.1002/asi.23910, https://doi.org/10.1002/asi.239103. Gurulingappa, H., Toldo, L., Schepers, C., Bauer, A., Megaro, G.: Semi-supervisedinformation retrieval system for clinical decision support. In: Proceedings of TheTwenty-Fifth Text REtrieval Conference, TREC 2016, Gaithersburg, Maryland,USA, November 15-18, 2016 (2016), http://trec.nist.gov/pubs/trec25/papers/MERCKKGAA-CL.pdf4. Soboroff, I., Ferro, N., Sakai, T.: Overview of the TREC 2018 centre trask. In: Pro-ceedings of The Twenty-Seventh Text REtrieval Conference, TREC 2018, Gaithers-burg, Maryland, USA, November 14-16, 2018 (2018), https://trec.nist.gov/IMS @ TREC PM 2018 7(a) infNDCG values(b) infAP values(c) RPrec valuesFig. 1: Topic by topic comparison of the difference of values for the MRKUmlsSolrreproduced run and the original one.

https://www.research.unipd.it/bitstream/11577/3302624/1/ims_unipd-C.pdf

The University of Padua IMS Research Group at CENTRE@TREC 2018

Abstract

Similar works

Full text

Available Versions

Padua Research Archive

Archivio istituzionale della ricerca - Università di Padova