45,283 research outputs found
Non-Autoregressive Machine Translation with Auxiliary Regularization
As a new neural machine translation approach, Non-Autoregressive machine
Translation (NAT) has attracted attention recently due to its high efficiency
in inference. However, the high efficiency has come at the cost of not
capturing the sequential dependency on the target side of translation, which
causes NAT to suffer from two kinds of translation errors: 1) repeated
translations (due to indistinguishable adjacent decoder hidden states), and 2)
incomplete translations (due to incomplete transfer of source side information
via the decoder hidden states).
In this paper, we propose to address these two problems by improving the
quality of decoder hidden representations via two auxiliary regularization
terms in the training process of an NAT model. First, to make the hidden states
more distinguishable, we regularize the similarity between consecutive hidden
states based on the corresponding target tokens. Second, to force the hidden
states to contain all the information in the source sentence, we leverage the
dual nature of translation tasks (e.g., English to German and German to
English) and minimize a backward reconstruction error to ensure that the hidden
states of the NAT decoder are able to recover the source side sentence.
Extensive experiments conducted on several benchmark datasets show that both
regularization strategies are effective and can alleviate the issues of
repeated translations and incomplete translations in NAT models. The accuracy
of NAT models is therefore improved significantly over the state-of-the-art NAT
models with even better efficiency for inference.Comment: AAAI 201
Translating pro-drop languages with reconstruction models
Pronouns are frequently omitted in pro-drop languages, such as Chinese, generally leading to significant challenges with respect to the production of complete translations. To date, very little attention has been paid to the dropped pronoun (DP) problem within neural machine translation (NMT). In this work, we propose a novel reconstruction-based approach to alleviating DP translation problems for NMT models. Firstly, DPs within all source sentences are automatically annotated with parallel information extracted from the bilingual training corpus. Next, the annotated source sentence is reconstructed from hidden representations in the NMT model. With auxiliary training objectives, in the terms of reconstruction scores, the parameters associated with the NMT model are guided to produce enhanced hidden representations that are encouraged as much as possible to embed annotated DP information. Experimental results on both Chinese-English and Japanese-English dialogue translation tasks show that the proposed approach significantly and consistently improves translation performance over a strong NMT baseline, which is directly built on the training data annotated with DPs
- …