25 research outputs found
Findings of the 2019 Conference on Machine Translation (WMT19)
This paper presents the results of the premier shared task organized alongside the Conference on Machine Translation (WMT) 2019.
Participants were asked to build machine translation systems for any of 18 language pairs, to be evaluated on a test set of news stories. The main metric for this task is human judgment of translation quality. The task was also opened up to additional test suites to probe specific aspects of translation
Are ambiguous conjunctions problematic for machine translation?
The translation of ambiguous words still poses challenges for machine translation.
In this work, we carry out a systematic quantitative analysis regarding the ability of different machine translation systems to disambiguate the source language conjunctions âbutâ and âandâ. We evaluate specialised test sets focused on the translation of these two conjunctions. The test sets contain source languages that do not distinguish different variants of the given conjunction, whereas the target languages do. In total, we evaluate the conjunction âbutâ on 20 translation outputs, and the conjunction âandâ on 10. All machine translation systems almost perfectly recognise one variant of the target conjunction, especially for the source conjunction
âbutâ. The other target variant, however, represents a challenge for machine translation systems, with accuracy varying from 50% to 95% for âbutâ and from 20% to 57% for âandâ. The major error for all systems is replacing the correct target variant with the opposite one
Linguistic evaluation of German-English Machine Translation using a Test Suite
We present the results of the application of a grammatical test suite for
GermanEnglish MT on the systems submitted at WMT19, with a
detailed analysis for 107 phenomena organized in 14 categories. The systems
still translate wrong one out of four test items in average. Low performance is
indicated for idioms, modals, pseudo-clefts, multi-word expressions and verb
valency. When compared to last year, there has been a improvement of function
words, non-verbal agreement and punctuation. More detailed conclusions about
particular systems and phenomena are also presented
Findings of the 2018 Conference on Machine Translation (WMT18)
This paper presents the results of the premier
shared task organized alongside the Confer-
ence on Machine Translation (WMT) 2018.
Participants were asked to build machine
translation systems for any of 7 language pairs
in both directions, to be evaluated on a test set
of news stories. The main metric for this task
is human judgment of translation quality. This
year, we also opened up the task to additional
test suites to probe specific aspects of transla-
tion
Fine-grained Human Evaluation of Transformer and Recurrent Approaches to Neural Machine Translation for English-to-Chinese
This research presents a fine-grained human evaluation to compare the
Transformer and recurrent approaches to neural machine translation (MT), on the
translation direction English-to-Chinese. To this end, we develop an error
taxonomy compliant with the Multidimensional Quality Metrics (MQM) framework
that is customised to the relevant phenomena of this translation direction. We
then conduct an error annotation using this customised error taxonomy on the
output of state-of-the-art recurrent- and Transformer-based MT systems on a
subset of WMT2019's news test set. The resulting annotation shows that,
compared to the best recurrent system, the best Transformer system results in a
31% reduction of the total number of errors and it produced significantly less
errors in 10 out of 22 error categories. We also note that two of the systems
evaluated do not produce any error for a category that was relevant for this
translation direction prior to the advent of NMT systems: Chinese classifiers.Comment: Accepted at the 22nd Annual Conference of the European Association
for Machine Translation (EAMT 2020