1 research outputs found
Otem&Utem: Over- and Under-Translation Evaluation Metric for NMT
Although neural machine translation(NMT) yields promising translation
performance, it unfortunately suffers from over- and under-translation is- sues
[Tu et al., 2016], of which studies have become research hotspots in NMT. At
present, these studies mainly apply the dominant automatic evaluation metrics,
such as BLEU, to evaluate the overall translation quality with respect to both
adequacy and uency. However, they are unable to accurately measure the ability
of NMT systems in dealing with the above-mentioned issues. In this paper, we
propose two quantitative metrics, the Otem and Utem, to automatically evaluate
the system perfor- mance in terms of over- and under-translation respectively.
Both metrics are based on the proportion of mismatched n-grams between gold
ref- erence and system translation. We evaluate both metrics by comparing their
scores with human evaluations, where the values of Pearson Cor- relation
Coefficient reveal their strong correlation. Moreover, in-depth analyses on
various translation systems indicate some inconsistency be- tween BLEU and our
proposed metrics, highlighting the necessity and significance of our metrics