In recent years, neural machine translation (NMT) has become the dominant
approach in automated translation. However, like many other deep learning
approaches, NMT suffers from overfitting when the amount of training data is
limited. This is a serious issue for low-resource language pairs and many
specialized translation domains that are inherently limited in the amount of
available supervised data. For this reason, in this paper we propose regressing
word (ReWE) and sentence (ReSE) embeddings at training time as a way to
regularize NMT models and improve their generalization. During training, our
models are trained to jointly predict categorical (words in the vocabulary) and
continuous (word and sentence embeddings) outputs. An extensive set of
experiments over four language pairs of variable training set size has showed
that ReWE and ReSE can outperform strong state-of-the-art baseline models, with
an improvement that is larger for smaller training sets (e.g., up to +5:15 BLEU
points in Basque-English translation). Visualizations of the decoder's output
space show that the proposed regularizers improve the clustering of unique
words, facilitating correct predictions. In a final experiment on unsupervised
NMT, we show that ReWE and ReSE are also able to improve the quality of machine
translation when no parallel data are available
