3 research outputs found
Natural Language Processing with Small Feed-Forward Networks
We show that small and shallow feed-forward neural networks can achieve near
state-of-the-art results on a range of unstructured and structured language
processing tasks while being considerably cheaper in memory and computational
requirements than deep recurrent models. Motivated by resource-constrained
environments like mobile phones, we showcase simple techniques for obtaining
such small neural network models, and investigate different tradeoffs when
deciding how to allocate a small memory budget.Comment: EMNLP 2017 short pape
Pre-reordering for neural machine translation: helpful or harmful?
Pre-reordering, a preprocessing to make the source-side word orders close to those of the
target side, has been proven very helpful for statistical machine translation (SMT) in improving
translation quality. However, is it the case in neural machine translation (NMT)? In this paper,
we firstly investigate the impact of pre-reordered source-side data on NMT, and then propose to
incorporate features for the pre-reordering model in SMT as input factors into NMT (factored
NMT). The features, namely parts-of-speech (POS), word class and reordered index, are encoded as feature vectors and concatenated to the word embeddings to provide extra knowledge
for NMT. Pre-reordering experiments conducted on Japanese↔English and Chinese↔English
show that pre-reordering the source-side data for NMT is redundant and NMT models trained
on pre-reordered data deteriorate translation performance. However, factored NMT using
SMT-based pre-reordering features on Japanese→English and Chinese→English is beneficial
and can further improve by 4.48 and 5.89 relative BLEU points, respectively, compared to the
baseline NMT system
Neural pre-translation for hybrid machine translation
Hybrid machine translation (HMT) takes advantage of different types of machine translation
(MT) systems to improve translation performance. Neural machine translation (NMT) can
produce more fluent translations while phrase-based statistical machine translation (PB-SMT)
can produce adequate results primarily due to the contribution of the translation model. In
this paper, we propose a cascaded hybrid framework to combine NMT and PB-SMT to improve translation quality. Specifically, we first use the trained NMT system to pre-translate
the training data, and then employ the pre-translated training data to build an SMT system and
tune parameters using the pre-translated development set. Finally, the SMT system is utilised
as a post-processing step to re-decode the pre-translated test set and produce the final result.
Experiments conducted on Japanese!English and Chinese!English show that the proposed
cascaded hybrid framework can significantly improve performance by 2.38 BLEU points and
4.22 BLEU points, respectively, compared to the baseline NMT system