The annual WMT Conference in Machine Translation has been running shared tasks since 2006. It started with a translation task based on Europarl, and has grown to include tasks on all aspects of MT corpus preparation, training and evaluation, including the flagship task on news translation. I will review the history of the task, lessons learnt, and plans for future tasks

Haddow, Barry

Repositorio Institucional de la Universidad de Alicante

Proceedings of the21st Annual Conference ofthe European Associationfor Machine Translation28–30 May 2018Universitat d’AlacantAlacant, SpainEdited byJuan Antonio Pe´rez-OrtizFelipe Sa´nchez-Mart´ınezMiquel Espla`-GomisMaja Popovic´Celia RicoAndre´ MartinsJoachim Van den BogaertMikel L. ForcadaOrganised byresearch groupThe papers published in this proceedings are —unless indicated otherwise— covered by theCreative Commons Attribution-NonCommercial-NoDerivatives 3.0 International (CC-BY-ND3.0). You may copy, distribute, and transmit the work, provided that you attribute it (au-thorship, proceedings, publisher) in the manner specified by the author(s) or licensor(s), andthat you do not use it for commercial purposes. The full text of the licence may be found athttps://creativecommons.org/licenses/by-nc-nd/3.0/deed.en.c© 2018 The authorsISBN: 978-84-09-01901-4The WMT Shared TasksBarry HaddowSchool of InformaticsUniversity of EdinburghEdinburghScotlandbhaddow@staffmail.ed.ac.ukAbstractThe annual WMT Conference in MachineTranslation has been running shared taskssince 2006. It started with a translationtask based on Europarl, and has grown toinclude tasks on all aspects of MT cor-pus preparation, training and evaluation,including the flagship task on news trans-lation. I will review the history of the task,lessons learnt, and plans for future tasks.1 IntroductionWe began organising shared tasks in machinetranslation at the Workshop in Machine Trans-lation (WMT) in 2006, initially with a transla-tion task based on Europarl. In later years, fund-ing from the EU projects EuroMatrixPlus andMosesCore (FP7) and QT21 and Cracker (H2020),plus commercial sponsorship, enabled us to in-crease the number of tasks and to produce pro-fessionally translated, unseen test sets drawn fromnews texts for the translation task. In 2016 WMTbecame a conference (retaining the acronym) andin the last three years the number of shared taskshas varied between 7 and 10.The shared tasks have covered translation(mainly news, but also other domains such as ITand biomedical and also more specialized taskssuch as pronoun and multimodal), training (bothtuning of SMT and training of NMT), reference-based evaluation, quality estimation, corpus prepa-ration (document alignment and corpus cleaning)as well as automatic post-editing. The quality esti-mation task has included different subtasks on es-c© 2018 The authors. This article is licensed under a CreativeCommons 3.0 licence, no derivative works, attribution, CC-BY-ND.timating the quality of MT output at word, sen-tence and document level, as well as trying to pre-dict the post-editing effort required for a given MToutput. The data from all the WMT tasks, includ-ing the training data, test data and task submissionsis made available for future research and has beenheavily used in academic publications.In the news translation task we have tried toinnovate in MT evaluation, whilst still providingfor comparison with previous years. After sev-eral years using a relative ranking approach, whereevaluators compare output from different systems,we switched to direct assessment (DA) in 2017. InDA, evaluators provide an assessment of adequacyon a scale from 0 to 100, which we find offers areliable system ranking and a more interpretableand comparable final score. The news task cov-ers a variety of languages, mainly European, withEnglish–German and English–Czech as our “core”languages. We have included both low-resource(e.g. Estonian–English and Hindi–English) andhigh-resource (e.g. French–English) pairs, and werelease our own parallel and monolingual data sets,as well as using standard sets like Europarl.In this talk I will review the history of the tasks,the lessons learnt and plans for future tasks, focus-ing on the news translation task. I will explain howthis task provides a common benchmark for com-paring different MT systems, which helps to driveMT research. I will also show how running thetask reveals difficulties and pitfalls in comparativeevaluation of MT systems.2 WebsiteThe URL for the latest conference/task is www.statmt.org/wmt18, where you will find linksto all previous conferences/workshops, tasks andpapers.Pe´rez-Ortiz, Sa´nchez-Mart´ınez, Espla`-Gomis, Popovic´, Rico, Martins, Van den Bogaert, Forcada (eds.)Proceedings of the 21st Annual Conference of the European Association for Machine Translation, p. 9Alacant, Spain, May 2018.

The WMT Shared Tasks

http://rua.ua.es/dspace/bitstream/10045/76015/1/EAMT2018-Proceedings_02.pdf

The WMT Shared Tasks

Abstract

Similar works

Full text

Available Versions

Repositorio Institucional de la Universidad de Alicante