66 research outputs found
Results of the WMT19 metrics shared task: segment-level and strong MT systems pose big challenges
This paper presents the results of the WMT19 Metrics Shared Task. Participants were asked to score the outputs of the translations systems competing in the WMT19 News Translation Task with automatic metrics. 13 research groups submitted 24 metrics, 10 of which are reference-less "metrics" and constitute submissions to the joint task with WMT19 Quality Estimation Task, "QE as a Metric". In addition, we computed 11 baseline metrics, with 8 commonly applied baselines (BLEU, SentBLEU, NIST, WER, PER, TER, CDER, and chrF) and 3 reimplementations (chrF+, sacreBLEU-BLEU, and sacreBLEU-chrF). Metrics were evaluated on the system level, how well a given metric correlates with the WMT19 official manual ranking, and segment level, how well the metric correlates with human judgements of segment quality. This year, we use direct assessment (DA) as our only form of manual evaluation
Findings of the 2019 Conference on Machine Translation (WMT19)
This paper presents the results of the premier shared task organized alongside the Conference on Machine Translation (WMT) 2019.
Participants were asked to build machine translation systems for any of 18 language pairs, to be evaluated on a test set of news stories. The main metric for this task is human judgment of translation quality. The task was also opened up to additional test suites to probe specific aspects of translation
A Challenge Set Approach to Evaluating Machine Translation
Neural machine translation represents an exciting leap forward in translation
quality. But what longstanding weaknesses does it resolve, and which remain? We
address these questions with a challenge set approach to translation evaluation
and error analysis. A challenge set consists of a small set of sentences, each
hand-designed to probe a system's capacity to bridge a particular structural
divergence between languages. To exemplify this approach, we present an
English-French challenge set, and use it to analyze phrase-based and neural
systems. The resulting analysis provides not only a more fine-grained picture
of the strengths of neural systems, but also insight into which linguistic
phenomena remain out of reach.Comment: EMNLP 2017. 28 pages, including appendix. Machine readable data
included in a separate file. This version corrects typos in the challenge se
Native Language Identification on Text and Speech
This paper presents an ensemble system combining the output of multiple SVM
classifiers to native language identification (NLI). The system was submitted
to the NLI Shared Task 2017 fusion track which featured students essays and
spoken responses in form of audio transcriptions and iVectors by non-native
English speakers of eleven native languages. Our system competed in the
challenge under the team name ZCD and was based on an ensemble of SVM
classifiers trained on character n-grams achieving 83.58% accuracy and ranking
3rd in the shared task.Comment: Proceedings of the Workshop on Innovative Use of NLP for Building
Educational Applications (BEA
- …