We developed a tool that integrates the National Library of Medicine's MetaMap software with GATE, an open-source text an- alytics framework. The tool allows non-ASCII encoded documents of numerous formats to be annotated with UMLS concepts. We created a GATE pipeline to chunk cardiovascular disease guideline text into default segments (blank-line delimited), XML element content, sentences and phrases, which were sequentially submitted to MetaMap for annotation. XML element, sentence and phrase chunking allowed term extraction and mapping to be completed in around 1/3 of the time taken with de- fault chunking, although with slight loss of accuracy (F1.0s=0.94-0.99). However, phrase chunking allows more complex input to be processed in real time, which is not possible with the other approaches. We discuss the results in relation to use of MetaMap's --term processing option for generating pre- and post-coordinated mappings from composite phrases

Gooch, P.

Roudsari, A.

English

City Research Online

Gooch, P. & Roudsari, A. (2011). A tool for enhancing MetaMap performance when annotating clinical guideline documents with UMLS concepts. Paper presented at the IDAMAP Workshop at 13th Conference on Artificial Intelligence in Medicine (AIME'11), 02-07-2011 - 06-07-2011, Bled, Slovenia.City Research OnlineOriginal citation: Gooch, P. & Roudsari, A. (2011). A tool for enhancing MetaMap performance when annotating clinical guideline documents with UMLS concepts. Paper presented at the IDAMAP Workshop at 13th Conference on Artificial Intelligence in Medicine (AIME'11), 02-07-2011 - 06-07-2011, Bled, Slovenia.Permanent City Research Online URL: http://openaccess.city.ac.uk/1036/ Copyright & reuseCity  University  London has developed City  Research Online  so that  its  users  may access the research outputs of City University London's staff. Copyright © and Moral Rights for this paper are retained by the individual author(s) and/ or other copyright holders. Users may download and/ or print one  copy  of  any  article(s)  in  City  Research  Online  to  facilitate  their  private  study  or  for  non-commercial research. Users may not engage in further distribution of the material or use it for any profit-making activities or any commercial gain. All material in City Research Online is checked for eligibility for copyright before being made available in the live archive. URLs from City Research Online may be freely distributed and linked to from other web pages. Versions of researchThe version in City Research Online may differ from the final published version. Users are advised to check the Permanent City Research Online URL above for the status of the paper.EnquiriesIf you have any enquiries about any aspect of City Research Online, or if you wish to make contact  with the author(s) of this paper, please email the team at publications@city.ac.uk.A tool for enhancing MetaMap performancewhen annotating clinical guideline documentswith UMLS conceptsPhilip Gooch?1 and Abdul Roudsari121 Centre for Health Informatics, School of Informatics, City University, London, UK,2 School of Health Information Science, University of Victoria, BC, CanadaAbstract. We developed a tool that integrates the National Libraryof Medicine’s MetaMap software with GATE, an open-source text an-alytics framework. The tool allows non-ASCII encoded documents ofnumerous formats to be annotated with UMLS concepts. We created aGATE pipeline to chunk cardiovascular disease guideline text into defaultsegments (blank-line delimited), XML element content, sentences andphrases, which were sequentially submitted to MetaMap for annotation.XML element, sentence and phrase chunking allowed term extractionand mapping to be completed in around 1/3 of the time taken with de-fault chunking, although with slight loss of accuracy (F1.0s=0.94-0.99).However, phrase chunking allows more complex input to be processed inreal time, which is not possible with the other approaches. We discussthe results in relation to use of MetaMap’s --term processing option forgenerating pre- and post-coordinated mappings from composite phrases.Keywords: biomedical language processing, information extraction, can-didate term identification, terminology, concept mapping, informaticsapplications1 IntroductionThe use of natural language processing (NLP) techniques has recently beenproposed as a method for generating tailored clinical decision support (CDS)from free text[1]. Precise identification of biomedical and clinical terms in thetext is a crucial step in this process[2].Term identification typically involves three stages[2]:1. recognize the text string as a possible term (candidate term selection)2. classify the candidate term (e.g. body part, disease, physiological funtion)3. map the term to a single concept (pre-coordination) or to qualified, multipleconcepts (post-coordination) within a standardised vocabulary or ontology.? Corresponding author.2 P. Gooch and A. RoudsariOne of the key tasks in integrating a guideline-based CDS system with anelectronic medical record (EMR) is to map clinical terms contained in bothguidelines and patient notes to a common, controlled terminology[3][4]. Thiscan facilitate the provision of point of care recommendations within an EMR,and allows encoded guidelines to be shared across institutions.The Unified Medical Language System (UMLS) Metathesaurus from the Na-tional Library of Medicine (NLM)[5] comprises over 100 reference terminologies,such as HL7 v3, SNOMED-CT and LOINC, which have been adopted as interna-tional standards for patient data encoding, and form the basis of the informationmodel adopted by at least one formalised guideline model[3].The NLM’s MetaMap software[6] is considered to be the reference tool foridentifying biomedical terms in free text and mapping them to UMLS con-cepts[7]. However, MetaMap only handles unformatted, ASCII text as input,and processing complex phrases can require many hours of computation[6].2 BackgroundOur goal in this study was to develop a tool that leveraged the domain cover-age of MetaMap, while enabling documents in a variety of non-ASCII formatsto be rapidly mined for biomedical terms and mapped to UMLS source vocab-ulary concepts. We also required that the tool be easily integrated with otherNLP pipelines for other information extraction (IE) tasks, such as identifyinglinguistic patterns in clinical guidelines[8] to extract decision support rules.In this paper we present an overview of the tool with examples and brief eval-uation results of its performance, in terms of annotation accuracy and processingtime, in comparison to the standalone MetaMap application, when extractingSNOMED-CT concepts from clinical guideline documents.3 MethodsGATE[9] is an open-source text analytics framework for processing documentsin variety of formats including PDF, RTF, and XML. Using the MetaMap JavaAPI, we developed a plugin that integrates MetaMap with GATE, wherebytext is normalized and chunked into blank-line delimited segments, which aresubmitted to MetaMap server (or multiple servers for parallel processing) andthe results converted to GATE annotations and features for further processing.As MetaMap works with ASCII data, it returns term positions as byte offsetsrelative to the start of the phrase. Annotation offsets in GATE are given asencoding-dependent character offsets. To accommodate this, UTF-8 and ISO-8859-1 data in the payload is normalized to its ASCII equivalent by using thejava.text.Normalizer class[10] to 1) convert the string to its Roman equivalentplus diacritic, 2) stripping the diacritic and 3) converting the resulting string toan ASCII byte stream from which to create a new ASCII-encoded string thatforms the input to MetaMap. This process allows terms containing diacritics(e.g. Guillain-Barre´) to be correctly recognised and mapped by MetaMap.UMLS annotation of clinical guideline documents 3The plugin provides a number of features designed to optimize the processingof large documents. For example, the document’s data payload to MetaMap canbe reduced to solely:– the content of candidate term annotations (identified by an upstream pro-cess), or named elements in the original XML markup;– the first instance of each candidate term, with remaining instances co-referencedagainst the mapped term;– named attributes of each candidate term, rather than the original sourcedata. For example, an upstream process might normalize prepositional phrasessuch as ‘cancer of the lung’ → ‘lung cancer’, or verb phrases such as ‘painis severe’ → ‘severe pain’, and store the result as an attribute to the term.This allows these grammatical variants to be treated as equivalent for thepurposes of coreferencing.In this study, we treated all noun (NP), prepositional (PP) and verb phrases(VP) as candidate terms. We used GATE’s Sentence Splitter to identify sen-tences, and combined GATE’s Tokenizer and Part of Speech Tagger with JavaAnnotation Patterns Engine (JAPE) rules to identify NP, PP and VP chunks.For example, a simple noun phrase can be identified using the following rule:NP = (DT)? (VB)? (ADJ)* (NN)+ (1)where DT represents a determiner (‘a’, ‘the’, ‘those’, ‘no’); VB a verb; ADJ anadjective; NN a noun. For example, ‘raisedV B bloodNN pressureNN ’.A prepositional phrase can be identified with:PP = (NP) (IN) ((NP) (IN))* (NP) (2)where IN represents a preposition. For example,‘absenceNP ofIN pulseNP ’, ‘no evidenceNP ofIN cardiovascular diseaseNP ’.A verb phrase can be identified with:VP = (NP | PP | ADJ+) VG+ (NP | PP | ADJ+)? (3)where VG represents a verb or verb group (‘is’, ‘should be’, ‘may provide’).We configured the plugin to identify only SNOMED-CT terms, and exam-ined its performance when annotating the content of the 106kB, 11,000 wordrecommendations section of the ESC European Guidelines on CardiovascularDisease Prevention in Clinical Practice in UTF-8 XML format for different inputchunking options (processed in serial): default (blank line delimited seqments),individual XML elements (p, li, h1, h2), sentences and NP, PP and VP chunks.We ran the tests both without and with MetaMap’s --term processingoption, which treats the input as a single phrase for direct lookup into theMetathesaurus, and provides a mechanism for mapping composite phrases to asingle identifier in UMLS.GATE’s Corpus Quality Assurance tool was used to calculate recall, preci-sion and strict F -measure scores, where F -measure = 2 * precision * recall /(precision + recall).4 P. Gooch and A. Roudsari4 ResultsTable 1 shows the time taken to annotate the guideline following default chunkingcompared with annotation of individual XML elements, sentences and phrases.Table 2 shows recall and precision scores for MetaMap annotations producedfrom the different lexical units both with and without term processing. Theoutput of the default-chunked input was used as the reference standard.Table 1. Processing times for different lexical unitsDefault Element Sentence PhraseTime (s) w/o term processing 745 208 213 268Time (s) w/term processing > 105† > 105† > 105† 325† Process aborted after 3 hours with no outputTable 2. Recall/precision for Element, Sentence, Phrase (B) vs default chunking (A)Annotation Match Only A Only B Overlap Rec. Prec. F1.0sElement (w/o term processing) 4122 349 154 9 0.92 0.96 0.94Sentence (w/o term processing) 4393 27 13 9 0.99 1.00 0.99Phrase (w/o term processing) 4168 128 53 184 0.93 0.95 0.94Phrase (w/ term processing) 3889 224 118 367 0.87 0.89 0.885 DiscussionInspection of the output suggests that the slight reduction in annotation accu-racy resulting from processing by phrase arises from lone verbs not being pickedup by our phrase chunker. While identification of relevant verbs is required forextracting ‘action’ information from clinical guidelines[8], we do not require themto be mapped to UMLS concepts, so perhaps this slight reduction is acceptable.Overall, processing by sentence provided the best trade-off between speed andaccuracy.However, phrase chunking allows rapid term extraction and mapping frominput without syntactical structure, such as bulleted lists, which can requiremany hours of computation if processed directly by MetaMap[6]. Also, as shownin Table 1, by-phrase processing allows the --term processing option to beused on entire documents. This is useful as it allows composite phrases in theguideline to be mapped to their pre-coordinated terms in UMLS, rather than tomultiple terms requiring post-coordination. Some examples are shown in Table 3.In a future paper we extend these techniques to the processing of EMRdata, as a first step in automating the analysis of guideline compliance, by com-paring SNOMED-CT encoded information from discharge summaries with thatextracted from the relevant guidelines.UMLS annotation of clinical guideline documents 5Table 3. Phrase chunking mappings both without and with (bold) term processingPhrase Mappingsmanagement of risk factors Administration(C0001554); History of- risk factor(C0455624)Risk management(C0035649)foot pulses Entire foot(C1281587); Pulse(C0391850)Pedal pulse(C0232157)increases in blood pressure Increase(C0442805); Blood pressure finding(C1271104)Elevated blood pressure(C0497247)6 ConclusionWe have presented a tool and method for rapid extraction of UMLS conceptsfrom clinical guidelines. It utilizes MetaMap’s domain coverage strengths andremoves some performance bottlenecks. The tool is now available as part ofthe standard GATE distribution, and can be easily integrated into other NLPpipelines within GATE, allowing it to be customised for more specific tasks.References1. Demner-Fushman, D., Chapman, W.W., McDonald, C.J.: What can natural lan-guage processing do for clinical decision support? J Biomed Inf 42(5), 760–72 (2009)2. Krauthammer, M., Nenadic, G.: Term identification in the biomedical literature. JBiomed Inf 37, 512–526 (2004)3. Tu, S.W., Campbell, J.R. , Glasgow, J., Nyman, M.A. et al.: The SAGE GuidelineModel: Achievements and Overview. Journal of the American Medical InformaticsAssociation 14(5), 589–598 (2007)4. Peleg, M., Keren, S., Denekamp, Y.: Mapping computerized clinical guidelines toelectronic medical records: Knowledge-data ontological mapper (KDOM). Journalof Biomedical Informatics 41, 180–201 (2008)5. National Library of Medicine: UMLS Reference Manual, Bethesda, MD, NationalLibrary of Medicine (2009)6. Aronson, A.R., Lang, F.-M.: An overview of MetaMap: historical perspective andrecent advances. J Am Med Inform Assoc 17, 229–236 (2010)7. Shah, N.H. Bhatia, N. Jonquet, C. Rubin, D. et al.: Comparison of concept recog-nizers for building the Open Biomedical Annotator. BMC Bioinformatics 10(Suppl9), S14 (2009)8. Serban, R., et al.: Extraction and use of linguistic patterns for modelling medicalguidelines. Artificial Intelligence in Medicine 39(2), 137–149 (2007)9. Cunningham, H., Maynard, D., Bontcheva, K., Tablan, V.: GATE: A Frameworkand Graphical Development Environment for Robust NLP Tools and Applications.In: Proceedings of the 40th Anniversary Meeting of the Association for Computa-tional Linguistics (ACL’02), Philadelphia (2002)10. Sun Microsystems: Normalizer (Java Platform SE 6). http://java.sun.com/javase/6/docs/api/java/text/Normalizer.html

A tool for enhancing MetaMap performance when annotating clinical guideline documents with UMLS concepts

Abstract

Similar works

Full text

Available Versions

City Research Online