Di Nunzio, Giorgio Maria

English

Archivio istituzionale della ricerca - Università di Padova

Classification of ICD10 Codes with no Resourcesbut Reproducible Code.IMS Unipd at CLEF eHealth Task 1Giorgio Maria Di NunzioDept. of Information Engineering – University of Paduagiorgiomaria.dinunzio@unipd.itAbstract. In this paper, we describe the second participation of theInformation Management Systems (IMS) group at CLEF eHealth 2018Task 1. In this task, participants are required to extract causes of deathfrom multilingual death reports (French, Hungarian and Italian) andlabel them with the correct International Classification Diseases (ICD10)code. We tackled this task by focusing on the reproducible code, that wepublished last year, which produces a clean dataset that can be used toimplement more sophisticated approaches.1 IntroductionIn this paper, we report the experimental results of the second participation ofthe IMS group to the CLEF eHealth Lab [5], in particular to Task 1: “Multilin-gual Information Extraction - ICD10 coding” [2]. This task consists in automat-ically labelling death certificates written in different languages (French, Hungar-ian, and Italian) with International Classification Diseases (ICD10) codes.The main goal of our participation to the task this year was to test theeffectiveness of the reproducible code made available by [3] which builds a clas-sification system that i) converts raw data into a cleaned dataset following a‘tidyverse’ approach1, ii) implements a set of manual rules to split sentences andtranslate medical acronyms, and iii) implement a lexicon based classificationapproach [1].The contribution of our experiments to this task can be summarized as fol-lows:– A study of a reproducibility framework to explain each step of the pipelinefrom raw data to cleaned data;– An evaluation of the application of a classification system prepared for alanguage (French) and applied without any additional training or changesto the source code to two different languages (Hungarian and Italian).We submitted three official runs, one for each language and prepared a num-ber of additional unofficial runs that we will evaluate and compare in order tostudy the change in performance when adding more information in the pipeline.1 https://www.tidyverse.orgTable 1. Expressions in French or punctuation marks used to split a line of a deathcertificate.Frenchavecsurparsuite à un[e]dans un contexte deaprès“,”, “;”, “/”2 MethodIn this section, we summarize the pipeline used in [3] that has been reproducedin this work for each run.2.1 Pipeline for Data CleaningIn order to produce a clean dataset, we followed the same pipeline for dataingestion and preparation for all the experiments:– read a line of a death certificate,– split the line according to the expression listed in Table 1;– remove extra white space (leading, trailing, internal);– transform letters to lower case;– remove punctuation;– expand acronyms (if any);– correct common patterns (if any).Acronym Expansion Acronym expansion is a crucial step to normalize dataand make the death certificate clearer and more coherent with the ICD10 codes.For the French experiments, we used. the original list of 1179 acronyms preparedby a semi-automated approach by [3].We show the first ten acronym expansions in Table 2. We want to stress thefact that this particular implementation of the expansion selects, in those caseswhere there is more than once choice (for example “aa”), only the first choice.This is part of our current work in order to improve this step of the pipeline.2.2 ClassificationWe used a simple unsupervised lexicon based approach to label each (segmentof a) line of a death certificate [1]. The procedure to assign an ICD10 code thatdoes not require any training is the following:Table 2. Acronym table (fist 10 rows) used to expand acronyms.acronym expansion5-hiaa acide 5-hydroxyindolactique5-ht 5-hydroxytryptamine5-ht srotoninea1at alpha-1-antitrypsinea1at a1-antitrypsineaa aorte ascendanteaa affection actuelleaa acide aminaa antiarthrosiqueaaa anvrisme de l’aorte abdominaleTable 3. Example of classification of a line of a certificate. The definition of the ICD10labels are shown in Table 4step dataline pneumopathie infectieuse lobaire inférieure droiteterms pneumopathie, infectieuse, lobaire, inferieure, droiteICD10 scores J181 = 7, J13 = 1– for each term in the (segment of a) line, sum one for each ICD10 label thatcontains the term,– for each (segment of a) line compute the score of each ICD10 label;– group the ICD10 labels that have the maximum score;– assign the most frequent code within this group.The score of each label is the sum of the binary weights. In those cases wheretwo or more classes have the same number of entries with the maximum score,the first class in the list is assigned by default. This is another part of the pipelinethat requires more effort in order to improve the effectiveness of the classifier.In Table 3, we show an example of the first three steps, while in Table 4 thedefinition of the ICD10 codes that received the highest score.3 Experiments and ResultsWe submitted three official runs, one for each language: French, Hungarian, andItalian. The idea of these experiments was to test the effectiveness of the originalFrench ICD10 classifier on two new languages without any modification to thesource code. That is, acronym expansion and sentence splitting are done usingFrench resources. We used only the raw dataset for all the languages.Table 4. Example of definitions (translitterated) of ICD10 selected in Table 3ICD10 definitionJ13 pneumopathie franche lobaire inferieureJ181 pneumopathie commune lobaire inferieureJ181 pneumopathie infectieuse lobaire aigueJ181 pneumopathie infectieuse lobaire moyenneJ181 pneumopathie infectieuse lobaire superieureJ181 pneumopathie lobaire inferieureJ181 pneumopathie lobaire inferieure aigueJ181 pneumopathie lobaire inferieure bilateraleTable 5. Results for the official runsFrench Hungarian ItalianPre Rec F-1 Pre Rec F-1 Pre Rec F-1runs 0.6534 0.3963 0.4933 0.7609 0.7482 0.7545 0.5353 0.4844 0.5086baseline 0.3410 0.2005 0.2525 0.2425 0.1735 0.2023 0.1648 0.1723 0.1685average 0.7228 0.4102 0.5066 0.8266 0.7830 0.8025 0.8441 0.7606 0.7992median 0.7981 0.4750 0.5790 0.9221 0.8972 0.9095 0.8995 0.8239 0.86303.1 Official RunsThe results of the three experiments are shown in Table 5. The French runperformed sufficiently well, and comparable to the results presented in [3]. TheF1 measure is close to the average of the results of all the participants in thistask. This confirms that a solid clean dataset is a good starting point to build aclassifier, even a simple classifier like the one we implemented.The Hungarian and Italian results are, as we expected, worse than the averagescores (much worse for Italian). However, it seems that the Hungarian datasetwas in a sense “easier” compared to the our results of our experiments in theItalian subtask. We are going to investigate the reasons for this large differencein performance as future work. Another interesting fact is that, while for theFrench task Precision was much higher than Recall, for the Hungarian and Italiandataset these two measures seem more “balanced”. This may suggest that abetter acronym expansion and better sentence splitting may favour Precisionover Recall.3.2 Unofficial RunsAs part of current and future work, we have prepared a set of unofficial runs. Afirst set of runs study the effect of an alternative weighting scheme, tf-idf insteadof binary weighting, another set of runs (for Hungarian and Italian) explore theeffectiveness of splitting the sentence with the correct words, see Table 6, as wellas expand acronym with the appropriate language. More runs will be createdTable 6. Expressions in Hungarian and Italian or punctuation marks used to split aline of a death certificate.Hungarian Italiana contovább a causa diáltal perután a a seguito diösszefüggésben conseguentemente aután dopo“,”, “;”, “/” “,”, “;”, “/”Table 7. Results for the unofficial runs, tf-idf vs binary weightingHungarian ItalianPre Rec F-1 Pre Rec F-1official 0.7609 0.7482 0.7545 0.5353 0.4844 0.5086official + tf-idf 0.6870 0.6720 0.6794 0.3642 0.3195 0.3404binary 0.7559 0.7478 0.7519 0.5353 0.4878 0.5104binary w/o acronym exp 0.7729 0.7652 0.7690 0.5495 0.5061 0.5269with additional parameters concerning the multiple label assignment and a betteracronym expansion algorithm.At present time, we have been able to evaluate the effectiveness of somecombinations of these parameters. In particular, we tested the binary weightingapproach vs the tf-idf approach, using the original French source code (‘inap-propriate’ acronyms and sentence splitting), results are shown in the first twolines of Table 7. These results confirms that for Hungarian and Italian the binaryweighting approach performs better than tf-idf (the only language that showedsome improvement in this task with the tf-idf weights was English [3])Then, we performed an experiment with binary weights and a ‘correct’ sen-tence splitting (see Table 6) with or without the French acronym expansion.Results are shown in the last two rows of Table 7. The fact that we used a lan-guage specific sentence splitting did not produce any significant change in theperformance of the classifier. This is probably due to the fact that the Hungarianand Italian death certificates are much more structured (from a language stand-point) than French ones. For example, we could rarely find complex sentenceswith words or terms listed in Table 6 in the Italian certificates. It seems thatpunctuation marks work sufficiently well for these two languages. Moreover, byremoving the French acronym expansion, we obtained a slight improvement dueto the fact that we removed the noise introduced by a module in the pipeline (theacronym expansion). In this case, results are better in terms of both Precisionand Recall compared to the official runs.4 Final remarks and Future WorkThe aim of our second participation to the CLEF eHealth Task 1 was to testthe reproducibility of the source code of the lexicon based classifier that wasimplemented the previous year. The performance of the French run was goodand we consider to use it as a baseline to build a new and improved classifier.The application of this classifier to two different language gave interesting results:the results of the Hungarian run was surprisingly high and close to the averageof the results of the participant. However, the high value of the median of F1(close to 90%) suggests that this subtask may be easier than the French one. Forthe Italian run, we obtained a worse performance the reasons of which we willinvestigate in a failure analysis.As current and future work, we are studying– the adaptation of the pipeline to the two new languages (better split sentenceand acronym expansion [4]);– the possibility to include multiple acronym expansions;– how to assign multiple labels to the same line (when scores are tied).References1. Jacob Eisenstein. Unsupervised learning for lexicon-based classification. In Proceed-ings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9,2017, San Francisco, California, USA., pages 3188–3194, 2017.2. A. Névéol, Robert A., F. Grippo, C. Morgand, C. Orsi, L. Pelikán, L. Ramadier,G. Rey, and P. Zweigenbaum. Clef ehealth 2018 multilingual information extractiontask overview: Icd10 coding of death certificates in french, hungarian and italian. InCLEF 2018 Evaluation Labs and Workshop: Online Working Notes. CEUR-WS.org,September 2018.3. Giorgio Maria Di Nunzio, Federica Beghini, Federica Vezzani, and Geneviève Hen-rot. A lexicon based approach to classification of ICD10 codes. IMS unipd at CLEFehealth task 1. In Working Notes of CLEF 2017 - Conference and Labs of theEvaluation Forum, Dublin, Ireland, September 11-14, 2017., 2017.4. Borbála Siklósi and Attila Novák. Detection and expansion of abbreviations in hun-garian clinical notes. In Félix Castro, Alexander Gelbukh, and Miguel González, ed-itors, Advances in Artificial Intelligence and Its Applications, pages 318–328, Berlin,Heidelberg, 2013. Springer Berlin Heidelberg.5. Hanna Suominen, Liadh Kelly, Lorraine Goeuriot, Evangelos Kanoulas, Leif Az-zopardi, Rene Spijker, Dan Li, Aurélie Névéol, Lionel Ramadier, Aude Robert,Guido Zuccon, and Joao Palotti, editors. Overview of the CLEF eHealth Evalu-ation Lab 2018. CLEF 2018 - 8th Conference and Labs of the Evaluation Forum,Lecture Notes in Computer Science. Springer, September 2018.

Classification of ICD10 codes with no resources but reproducible code. IMS Unipd at CLEF eHealth Task 1

Padua Research Archive

https://www.research.unipd.it/bitstream/11577/3302621/1/paper_180.pdf

Classification of ICD10 codes with no resources but reproducible code. IMS Unipd at CLEF eHealth Task 1

Abstract

Similar works

Full text

Available Versions

Archivio istituzionale della ricerca - Università di Padova

Padua Research Archive