The ArisToCAT project aims to assess the comprehensibility of ‘raw’ (unedited) MT output for readers who can only rely on the MT output. In this project description, we summarize the main results of the project and present future work

Daems, Joke

Fonteyne, Margot

Macken, Lieve

Tezcan, Arda

English

Ghent University Academic Bibliography

Assessing the Comprehensibility of Automatic Translations (ArisToCAT)Lieve Macken, Margot Fonteyne, Arda Tezcan and Joke DaemsLT3, Language and Translation Technology TeamGhent UniversityBelgiumLieve.Macken@ugent.beAbstractThe ArisToCAT project aims to assess thecomprehensibility of ‘raw’ (unedited) MToutput for readers who can only rely on theMT output. In this project description, wesummarize the main results of the projectand present future work.1 IntroductionMachine translation (MT) systems cannot guaran-tee that the text they produce will be fluent andcoherent in both syntax and semantics. Errorsoccur frequently in machine-translated text, leav-ing the reader to guess parts of the intended mes-sage. With the arrival of neural machine transla-tion (NMT), however, the quality of machine trans-lation has increased significantly. As such, ma-chine translation is becoming an attractive solu-tion to deal with the increased need for translatedcontent. This could mean that, in the near future,readers will be more often confronted with ‘raw’(unedited) MT output.2 Quality of MT outputTo assess the quality improvements in MT, wecompared the quality of three different MT sys-tems for English–Dutch: a commercial neural sys-tem, a phrase-based system and a predominantlyrule-based system. We used Web-Anno1 as anno-tation tool and adopted a two-step approach to an-notate all errors in the MT output. In a first step,only the target text was visible and we marked allfluency errors; in a second step all accuracy errorsc  2020 The authors. This article is licensed under a CreativeCommons 3.0 licence, no derivative works, attribution, CC-BY-ND.1https://webanno.github.io/webanno/were labelled in both source and target text andwere linked. Van Brussel et al. (2018) found thatthe neural system, in general, outperformed thephrase-based and rule-based systems when con-sidering fluency. The output of the neural systemcontained fewer grammatical errors and hardly anyspelling mistakes. For accuracy, the improvementsof NMT are less apparent. The target sentencedoes not always contain traces of the errors or cluesof omissions, which might have an impact on thecomprehension.3 Reading comprehension testsIn a pilot study, Macken and Ghyselen (2018) se-lected three texts of the English MT Evaluationversion of the Corpus of Reading ComprehensionExercises (Scarton and Specia, 2016) and set up areading comprehension test for both human trans-lated and raw MT texts. Ninety-nine participantswere asked to read the translation very carefully af-ter which they had to answer comprehension ques-tions without having access to the translated text.Human translations received the best overall clar-ity scores, but the reading comprehension testsprovided much less unequivocal results.4 Comprehensibility of newly inventedwords in NMT outputNMT systems occasionally generate non-existingwords, i.e words that are not part of the vocabu-lary of the target language and were thus inventedby the NMT system. In cases in which readersonly have access to the MT output without thesource text, such non-existing words can affectcomprehension. There are several reasons whyan NMT system creates new non-existing words.One reason is that, although NMT systems haveMartins, Moniz, Fumega, Martins, Batista, Coheur, Parra, Trancoso, Turchi, Bisazza, Moorkens, Guerberof, Nurminen, Marg, Forcada (eds.)Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, p. 485–486Lisboa, Portugal, November 2020.made huge progress, they sometimes still generatea too literal translation for different types of multi-word expressions such as compounds, another rea-son, specific for NMT systems, is that they oper-ate at sub-word level to reduce vocabulary size.Macken et al. (2019) set up an experiment in whicheighty-six participants were given 15 non-existingwords (5 single words and 10 noun compounds)and were either asked to describe the meaning ofthese words or to select the correct meaning froma predefined list. The words were presented eitherin isolation or in sentence context. Non-existingwords indeed impaired comprehension as on av-erage in 60% of the cases the participants gave awrong answer. Sentence context, however, made iteasier for the participants to determine the mean-ing of the non-existing word as the percentage ofwrong answers is much higher (77%) when thewords were presented in isolation.5 NMT for literary translationTo assess whether, with the improved quality,NMT systems are able to produce high-qualitytranslations for more creative text types such as lit-erature, we translated Agatha Christie’s novel TheMysterious Affair at Styles with Google’s NMTsystem into Dutch and applied the two-step errorannotation. Fonteyne et al. (2020) found that 44%of the MT sentences did not contain any errors.The accuracy subcategory mistranslation was themost frequent error type encountered in the novel,followed by the fluency subcategories coherenceand style & register. Tezcan et al. (2019) furtherinvestigated how the MT version differs from thepublished professionally human-translated (HT)Dutch version of the book. Measures of lexi-cal richness (type-token ratio and mean segmentaltype-token ratio) gave inconclusive results. Theyalso looked at word translation entropy (Carl et al.,2016), which indicates the degree of uncertaintyto choose a particular translation from a set of tar-get words based on the number and distribution ofdifferent translations that are available for a givenword in a given context and found that the averageword translation entropy scores were higher in HTthan in MT, meaning that there was more varietyin the translations in HT. At the syntactic level, theMT generally follows more closely the structure ofthe source sentence compared to the HT version.6 Future workIn future work, we will set up eye-tracking exper-iments and expand the Ghent Eye-Tracking Cor-pus (Cop et al., 2017) with eye-tracking data forthe NMT version of the novel. This will allow usto analyse to what extent MT impacts the readingprocess, and which errors impact this reading pro-cess the most.Acknowledgements: The ArisToCAT project isa four-year research project (2017–2021) fundedby the Research Foundation – Flanders (FWO) –grant number G.0064.17N assigned to prof. dr.Lieve Macken. All research is carried out at theUniversity of Ghent (Belgium).ReferencesCarl, M., M. Schaeffer and S. Bangalore 2016.The CRITT Translation Process Research Database.In New Directions in Empirical Translation Pro-cess Research: Exploring the CRITT TPR-DB.,Cham, Heidelberg, New York, Dordrecht, London:Springer, 13–54.Cop, U., N. Dirix, D. Drieghe and W. Duyck 2017.Presenting GECO: An eyetracking corpus of mono-lingual and bilingual sentence reading. Behavior Re-search Methods, 49:602–615.Fonteyne, M., A. Tezcan, and L. Macken. Accepted.Literary MT under the Magnifying Glass: Assessingthe Quality of an NMT-Translated Agatha ChristieNovel. Proceedings of LREC 2020Macken, L. and I. Ghyselen. 2018. Measuring compre-hension and acceptability of neural machine trans-lated texts: a pilot study. Proceedings of the 40thConference Translating and the Computer, London,UK, pp. 120–126.Macken, L., L. Van Brussel and J. Daems 2019.NMT’s wonderland where people turn into rabbits.A study on the comprehensibility of newly inventedwords in NMT output. Computational Linguistics inthe Netherlands Journal, 9:67–80.Scarton, C. and L. Specia. 2016. A reading compre-hension corpus for machine translation evaluation.Proceedings of LREC 2016, Portorozˇ (Slovenia).Tezcan, A., J. Daems, and L. Macken. 2019. Whena ‘sport’ is a person and other issues for NMT ofnovels. Proceedings of the Qualities of Literary Ma-chine Translation, Dublin, Ireland: European Asso-ciation for Machine Translation, pp. 40–49Van Brussel, L., A. Tezcan and L. Macken. 2018. Afine-grained error analysis of NMT, SMT and RBMToutput for English-to-Dutch. Proceedings of LREC2018, Miyazaki, Japan.486

Assessing the Comprehensibility of Automatic Translations (ArisToCAT)

https://biblio.ugent.be/publication/8665530/file/8665538.pdf

Assessing the Comprehensibility of Automatic Translations (ArisToCAT)

Abstract

Similar works

Full text

Available Versions

Ghent University Academic Bibliography