Juric, Damir

Moramarco, Francesco

Reiter, Ehud

Savkov, Aleksandar

English

arXiv

Aberdeen University Research

Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval), pages 56–61Online, April 19, 2021. ©2021 Association for Computational Linguistics56Towards objectively evaluating the qualityof generated medical summariesFrancesco Moramarco∗Babylon Health / London, UKUniversity of Aberdeen / Aberdeen, UKfrancesco.moramarco†Damir Juric*Babylon Health / London, UKdamir.juric†Aleksandar SavkovBabylon Health / London, UKsasho.savkov†Ehud ReiterUniversity of Aberdeen / Aberdeen, UKe.reiter@abdn.ac.uk†@babylonhealth.comAbstractWe propose a method for evaluating the qual-ity of generated text by asking evaluators tocount facts, and computing precision, recall, f-score, and accuracy from the raw counts. Webelieve this approach leads to a more objectiveand easier to reproduce evaluation. We applythis to the task of medical report summarisa-tion, where measuring objective quality and ac-curacy is of paramount importance.1 IntroductionNatural Language Generation in the medical do-main is notoriously hard because of the sensitivityof the content and the potential harm of hallucina-tions and inaccurate statements (Kryscinski et al.,2020; Falke et al., 2019). This informs the humanevaluation of NLG systems, selecting accuracy andoverall quality of the generated text as the mostvaluable aspects to be evaluated.In this paper we carry out a human evaluationof the quality of medical summaries of ClinicalReports generated by state of the art (SOTA) textsummarisation models.Our contributions are: (i) a re-purposed paralleldataset of medical reports and summary descrip-tions for training and evaluating, (ii) an approachfor a more objective human evaluation using counts,and (iii) a human evaluation conducted on thisdataset using the approach proposed.2 Related WorkA recent study by Celikyilmaz et al. (2020) givesa comprehensive view on different approaches totext summary evaluation. While many of these can∗Equal contributionbe wholly or partly translated between differentdomains, the medical domain remains particularlyproblematic due to the sensitive nature of its data.Moen et al. (2014) and Moen et al. (2016) try toestablish if there is a correlation between automaticand human evaluations of clinical summaries. A4-point and 2-point Likert scales are used for thehuman evaluation. In Goldstein et al. (2017) theauthors generate free-text summary letters fromthe data of 31 different patients and compare themto the respective original physician-composed dis-charge letters, measuring relative completeness,quantifying missed data items, readability, andfunctional performance.Closest to our approach is the Pyramid methodby Nenkova et al. (2007), which defines seman-tically motivated, sub-sentential units (SummaryContent Units) for annotators to extract in each ref-erence summary. SCUs are weighed according tohow often they appear in the multiple referencesand then compared with the SCUs extracted inthe hypothesis to compute precision, recall, andf-score.3 DataThe MTSamples dataset comprises 5,000 samplemedical transcription reports from a wide varietyof specialities uploaded to a community platformwebsite1. The dataset has been used in past medicalNLP research (Chen et al., 2011; Lewis et al., 2011;Soysal et al., 2017) including as a Kaggle dataset2.There are 40 medical specialties in the dataset,such as ‘Surgery’, ‘Consult - History and Phy.’,and ‘Cardiovascular / Pulmonary’. Each specialty1https://mtsamples.com2https://www.kaggle.com/tboyle10/medicaltranscriptions57contains a number of sample reports ranging from6 to 1103.The reports are free text with headings, whichchange according to the specialty. However, all re-ports also have a description field, which is a goodapproximation of a summary of the report. Thelength of each report varies greatly according tothe specialty, with an average of 589 words for thebody of the report, and 21 words for the description.Figure 1 shows an example of MTSamples reports,inclusive of description.Figure 1: An MTSamples clinical report of specialty‘Diets and Nutritions’. Note the reference Descriptionat the bottom.Given the brevity of some descriptions, wediscard reports with descriptions shorter than 12words and consider a dataset of 3242 reports. Byexamining the dataset, we note that descriptionsare mostly extractive in nature, meaning they arephrases or entire sentences taken from the report.To quantify this we compute n-gram overlap withRouge-1 (unigram) and Rouge-L (longest commonn-gram) (Lin, 2004) precision scores, which are0.989 and 0.939 respectively.We split the dataset into 2 576 reports for train-ing (80%), 323 for development (10%) and 343for testing (10%). We perform the split separatelyfor each medical specialty to ensure they are ade-quately represented and then aggregate the data.The dataset, models, and evaluation results canbe found on Github3.4 Experimental SetupFor our experiment, we consider one baseline andthree SOTA automatic summarisation models (ex-tractive, abstractive, and fine-tuned on our trainingset respectively). More specifically:• Lead–3 — this is our baseline. FollowingZhang et al. (2018), this model selects the firstthree sentences of the clinical report as thedescription;• Bert–Ext — the unsupervised extractivemodel by Miller (2019) 4;• Pegasus–CNN — an abstractive model byZhang et al. (2019) trained on the CNN/Dailymail dataset and used as is;• Bart–Med — an abstractive model by Lewiset al. (2020), which we fine-tune on our MT-Samples training set.We generate descriptions with these 4 modelsusing the entire clinical report text as input.5 Human Evaluation ProtocolWe select 10 clinical reports and summary descrip-tions from our MTSamples test set. Our subjectsare three general practice physicians. They are em-ployed at Babylon Health and have experience inAI research evaluation. The task is implementedwith the Heartex Annotation Platform5, which letsresearchers define tasks in an XML language andspecify the number of annotators. It then generateseach individual task and collates the results.The task involves (i) reading the clinical report,(ii) reading the reference description (supplied bythe dataset, see Figure 1), (iii) then evaluating 4generated descriptions by answering 5 questions(for a total of 40 generated descriptions). We askthe evaluators to count the “medical facts” in eachgenerated description and to compare them againstthose in the reference. Initially, we considered list-ing the types of facts to be extracted, as done byThomson and Reiter (2020), but the sheer diversityin the structure and content across the specialties3https://github.com/babylonhealth/medical-note-summarisation4https://pypi.org/project/bert-extractive-summarizer/5https://www.heartex.ai/58in our dataset made this approach impractical. In-stead, we give evaluators instructions containingtwo examples and ask them to extrapolate a processfor fact extraction. Figure 2 shows the instructionswe give them.The evaluation consists of reading a clinicalreport and a number of short descriptions,then quantifying how many “medical facts”were correctly reported. We understand thatthe definition of a “medical fact” is vague,and so it’s up to your interpretation. As anexample, in the following description:2-year-old female who comes in forjust rechecking her weight, herbreathing status, and her diet.There are (arguably) 4 facts:• 2 year old female• coming to recheck her weight• coming to recheck her breathing status• coming to recheck her dietHere’s another example:The patient had a syncopal episodelast night. She did not have anyresidual deficit. She had a headacheat that time. She denies chest painsor palpitations.Here there are (arguably) 5 facts:• had a syncopal episode last night• no residual deficit• headache• no chest pains• no palpitationsFigure 2: Instructions to evaluators.The evaluators are asked to read the clinical re-port (as shown in Figure 1), then to analyse thereference description by reporting the number offacts. To aid them in the task, they can optionallyselect the facts in the text using an in-built Hear-tex feature. Next, they are shown four generateddescriptions (one per model) and asked to countfacts and answer 5 questions. Figure 3 shows thereference, generated descriptions, and questions fora given task, and gives an example annotation fromone of the evaluators. When answering question 3(How many facts in G are correct?) they refer tothe clinical report as a ground truth.Based on this set of questions, we gather thefollowing raw counts:Figure 3: A completed task. Real Description is thereference.• R: facts in the reference description• G: facts in the generated description• R&G: facts in common• C: correct facts in the generated descriptionWe use these raw counts to compute four derivedmetrics:• Precision, calculated as R&GG• Recall, calculated as R&GR• F-Score, calculated as 2 · Precision·RecallPrecision+Recall• Accuracy, calculated as CGFor Coherence, we take Chen et al. (2020) andJuraska et al. (2019) definition: “whether the gen-erated text is grammatically correct and fluent, re-59Model Metric Eval 1 Eval 2 Eval 3 AvgLead-3Precision 0.42 0.43 0.46 0.44Recall 0.64 0.60 0.73 0.66F-Score 0.49 0.45 0.51 0.48Accuracy 1.0 1.0 1.0 1.0Coherence 0.95 0.95 0.90 0.93Bert-ExtPrecision 0.58 0.48 0.48 0.51Recall 0.62 0.61 0.60 0.61F-Score 0.59 0.52 0.51 0.54Accuracy 1.0 1.0 1.0 1.0Coherence 1.0 0.95 1.0 0.98Pegasus-CNN Precision 0.29 0.36 0.31 0.32Recall 0.43 0.50 0.50 0.47F-Score 0.34 0.40 0.36 0.37Accuracy 0.97 0.97 0.98 0.97Coherence 1.0 1.0 0.95 0.98Bart-MedPrecision 0.65 0.58 0.55 0.59Recall 1.0 0.96 0.97 0.98F-score 0.77 0.70 0.68 0.72Accuracy 1.0 1.0 1.0 1.0Coherence 1.0 0.75 0.95 0.90Table 1: Derived metrics for each model and each eval-uator, aggregated across tasks.gardless of factual correctness” and ask evaluatorsto choose between three options (Coherent, MinorErrors, Major Errors) and convert these into contin-uous numbers with Coherent = 1.0, Minor Errors =0.5, and Major Errors = 0.0.6 Results and DiscussionTable 1 shows the results for all derived metrics,calculated on the raw counts from the evaluators.Expectedly, Bart-Med, the model trained on theMTSamples training set, scores highest in all met-rics (except Coherence).Interestingly, all four models score almost-perfect accuracy, meaning they don’t hallucinatemedical facts. This is not a surprise for Lead-3 andBert-Ext, which are extractive in nature. As forPegasus-CNN and Bart-Med, while the models areabstractive, we notice they tend to mostly select andcopy phrases or entire sentences from the sourcereport. The only hallucination the evaluators foundis a numerical error, reported by Pegasus-CNN inthe following generated description:Patient’s weight has dropped from 201pounds to 201 pounds. She has lost atotal of 24 pounds in the past month.Whereas, the source report states:Metric E1–E2–E3 E1–E2 E1–E3 E2–E3RawCounts R facts 0.25 0.44 0.27 0.01G facts 0.33 0.50 0.26 0.12G&R facts 0.55 0.74 0.50 0.40G acc facts 0.34 0.51 0.27 0.13Coherence 0.40 0.14 0.56 0.49Der.Metrics Precision 0.87 0.84 0.88 0.88Recall 0.90 0.93 0.89 0.89F-Score 0.89 0.88 0.91 0.87Accuracy 0.87 0.79 0.96 0.84Table 2: Krippendorff Alpha for each metric, where Ris reference, G the generated description, G acc factsthe count of accurate facts in the generated description,E1-E2-E3 the agreement of all three evaluators, and Ex-Ey the agreement between Evaluator x and Evaluator y.Weight today is 201 pounds, which isdown 3 pounds in the past month. Shehas lost a total of 24 pounds.6.1 AgreementTo validate the human evaluation task, we computeinter-annotator agreement for each derived metric,as well as on the raw counts. We use KrippendorffAlpha (Hayes and Krippendorff, 2007) as we aredealing with continuous values. Table 2 includesoverall agreement and a breakdown for each pairof evaluators.Looking at the E1-E2-E3 column, we note aclear divide between the low agreement on rawcounts and the high agreement on the derived met-rics. We investigate this by comparing the factsselected by each annotator and notice a degree ofvariability in the level of granularity they employed.Consider the description:An 83-year-old diabetic female presentstoday stating that she would like dia-betic foot care.Table 3 shows the facts selected by the threeevaluators.We compute pairwise agreement in Table 2 andnotice that two of the evaluators (E1 and E2) sharea similar (more granular) approach to fact selection,whereas E3 is less granular.We also investigate the low agreement for Co-herence and discover that it’s due to a strong im-balance of the three classes (Coherent, Minor Er-rors, and Major Errors) where Coherent appears91.67% of cases, Minor Errors 6.67% and MajorErrors 1.67%. While this causes a low Krippen-dorff Alpha, we count the number of times all three60E Count Selected FactsE1 2- 83-year-old diabetic female- would like diabetic foot careE2 5- 83-year-old- diabetic- female- presents today- would like diabetic foot careE3 3- 83-year-old female- diabetic- would like diabetic foot careTable 3: Example of evaluators disagreement in factselection.evaluators agree on a generated description beingCoherent and find it to be 82.5%.Finally, for all derived metrics the agreementscores are very high. This shows a robustness ofthese metrics even with different granularity in factselection, and that the three evaluators agree onthe quality of a given generated description. Inother words, the evaluators agree on the quality ofthe generated descriptions even though they don’tagree on the way of selecting medical facts.7 Future WorkIn this paper we presented an evaluation of the qual-ity of medical summaries using fact counting. Theresults of this study help us to identify a number ofinsights to guide our future work:• We could work on better defining a medicalfact (as in Dušek and Kasner (2020)) and toprompt agreement on the level of granularity,for instance by instructing evaluators to splita description into the highest number of factsthat are meaningful;• Our evaluation focused on the quality of thegenerated descriptions and did not evaluatetheir usefulness in the medical setting. Suchextrinsic evaluation would be very valuable;• We could compare our approach of fact count-ing with the more common Likert scales.ReferencesAsli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao.2020. Evaluation of text generation: A survey.arXiv preprint arXiv:2006.14799.Elizabeth Chen, Sharad Manaktala, Indra Sarkar, andGenevieve Melton. 2011. A multi-site content anal-ysis of social history information in clinical notes.AMIA Annual Symposium Proceedings, 2011:227–36.Zhiyu Chen, Harini Eavani, Wenhu Chen, Yinyin Liu,and William Yang Wang. 2020. Few-shot NLG withpre-trained language model. In Proceedings of the58th Annual Meeting of the Association for Compu-tational Linguistics, pages 183–190, Online. Associ-ation for Computational Linguistics.Ondřej Dušek and Zdeněk Kasner. 2020. Evaluating se-mantic accuracy of data-to-text generation with nat-ural language inference. In Proceedings of the 13thInternational Conference on Natural Language Gen-eration, pages 131–137.Tobias Falke, Leonardo FR Ribeiro, Prasetya AjieUtama, Ido Dagan, and Iryna Gurevych. 2019.Ranking generated summaries by correctness: An in-teresting but challenging application for natural lan-guage inference. In Proceedings of the 57th AnnualMeeting of the Association for Computational Lin-guistics, pages 2214–2220.Ayelet Goldstein, Yuval Shahar, Efrat Orenbuch, andMatan J Cohen. 2017. Evaluation of an automatedknowledge-based textual summarization system forlongitudinal clinical data, in the intensive care do-main. Artificial intelligence in medicine, 82:20–33.Andrew F Hayes and Klaus Krippendorff. 2007. An-swering the call for a standard reliability measurefor coding data. Communication methods and mea-sures, 1(1):77–89.Juraj Juraska, Kevin Bowden, and Marilyn Walker.2019. ViGGO: A video game corpus for data-to-text generation in open-domain conversation. InProceedings of the 12th International Conferenceon Natural Language Generation, pages 164–172,Tokyo, Japan. Association for Computational Lin-guistics.Wojciech Kryscinski, Bryan McCann, Caiming Xiong,and Richard Socher. 2020. Evaluating the factualconsistency of abstractive text summarization. InProceedings of the 2020 Conference on EmpiricalMethods in Natural Language Processing (EMNLP),pages 9332–9346.Mike Lewis, Yinhan Liu, Naman Goyal, Mar-jan Ghazvininejad, Abdelrahman Mohamed, OmerLevy, Veselin Stoyanov, and Luke Zettlemoyer.2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation,and comprehension. In Proceedings of the 58th An-nual Meeting of the Association for ComputationalLinguistics, pages 7871–7880, Online. Associationfor Computational Linguistics.Neal Lewis, Daniel Gruhl, and Hui Yang. 2011. Ex-tracting family history diagnosis from clinical texts.pages 128–133.Chin-Yew Lin. 2004. Rouge: A package for automaticevaluation of summaries. In Text summarizationbranches out, pages 74–81.61Derek Miller. 2019. Leveraging bert for extractivetext summarization on lectures. arXiv preprintarXiv:1906.04165.Hans Moen, Juho Heimonen, Laura-Maria Murtola,Antti Airola, Tapio Pahikkala, Virpi Terävä, Ri-itta Danielsson-Ojala, Tapio Salakoski, and SannaSalanterä. 2014. On evaluation of automatically gen-erated clinical discharge summaries. In PAHI, pages101–114.Hans Moen, Laura-Maria Peltonen, Juho Heimonen,Antti Airola, Tapio Pahikkala, Tapio Salakoski, andSanna Salanterä. 2016. Comparison of automaticsummarisation methods for clinical free text notes.Artificial Intelligence in Medicine, 67:25 – 37.Ani Nenkova, Rebecca Passonneau, and KathleenMcKeown. 2007. The pyramid method: Incorporat-ing human content selection variation in summariza-tion evaluation. TSLP, 4.Ergin Soysal, Jingqi Wang, Min Jiang, Yonghui Wu,Serguei Pakhomov, Hongfang Liu, and Hua Xu.2017. CLAMP – a toolkit for efficiently build-ing customized clinical natural language processingpipelines. Journal of the American Medical Infor-matics Association, 25(3):331–336.Craig Alexander Thomson and Ehud Reiter. 2020. Agold standard methodology for evaluating accuracyin data-to-text systems. In Proceedings of the 13thInternational Conference on Natural Language Gen-eration.Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe-ter J. Liu. 2019. Pegasus: Pre-training with ex-tracted gap-sentences for abstractive summarization.Xingxing Zhang, Mirella Lapata, Furu Wei, and MingZhou. 2018. Neural latent extractive document sum-marization. In Proceedings of the 2018 Conferenceon Empirical Methods in Natural Language Process-ing, pages 779–784.

Towards objectively evaluating the quality of generated medical summaries

https://aura.abdn.ac.uk/bitstream/2164/18343/1/Moramarco_etal_ACL_Towards_Objectively_Evaluating_VoR.pdf

Towards objectively evaluating the quality of generated medical summaries

Abstract

Similar works

Full text

Available Versions

Aberdeen University Research