9,972 research outputs found
Back-Translation Sampling by Targeting Difficult Words in Neural Machine Translation
Neural Machine Translation has achieved state-of-the-art performance for
several language pairs using a combination of parallel and synthetic data.
Synthetic data is often generated by back-translating sentences randomly
sampled from monolingual data using a reverse translation model. While
back-translation has been shown to be very effective in many cases, it is not
entirely clear why. In this work, we explore different aspects of
back-translation, and show that words with high prediction loss during training
benefit most from the addition of synthetic data. We introduce several
variations of sampling strategies targeting difficult-to-predict words using
prediction losses and frequencies of words. In addition, we also target the
contexts of difficult words and sample sentences that are similar in context.
Experimental results for the WMT news translation task show that our method
improves translation quality by up to 1.7 and 1.2 Bleu points over
back-translation using random sampling for German-English and English-German,
respectively.Comment: 11 pages, 2 figures. Accepted at EMNLP 201
Data Augmentation for Low-Resource Neural Machine Translation
The quality of a Neural Machine Translation system depends substantially on
the availability of sizable parallel corpora. For low-resource language pairs
this is not the case, resulting in poor translation quality. Inspired by work
in computer vision, we propose a novel data augmentation approach that targets
low-frequency words by generating new sentence pairs containing rare words in
new, synthetically created contexts. Experimental results on simulated
low-resource settings show that our method improves translation quality by up
to 2.9 BLEU points over the baseline and up to 3.2 BLEU over back-translation.Comment: 5 pages, 2 figures, Accepted at ACL 201
Learning to Translate in Real-time with Neural Machine Translation
Translating in real-time, a.k.a. simultaneous translation, outputs
translation words before the input sentence ends, which is a challenging
problem for conventional machine translation methods. We propose a neural
machine translation (NMT) framework for simultaneous translation in which an
agent learns to make decisions on when to translate from the interaction with a
pre-trained NMT environment. To trade off quality and delay, we extensively
explore various targets for delay and design a method for beam-search
applicable in the simultaneous MT setting. Experiments against state-of-the-art
baselines on two language pairs demonstrate the efficacy of the proposed
framework both quantitatively and qualitatively.Comment: 10 pages, camera read
Understanding and Enhancing the Use of Context for Machine Translation
To understand and infer meaning in language, neural models have to learn
complicated nuances. Discovering distinctive linguistic phenomena from data is
not an easy task. For instance, lexical ambiguity is a fundamental feature of
language which is challenging to learn. Even more prominently, inferring the
meaning of rare and unseen lexical units is difficult with neural networks.
Meaning is often determined from context. With context, languages allow meaning
to be conveyed even when the specific words used are not known by the reader.
To model this learning process, a system has to learn from a few instances in
context and be able to generalize well to unseen cases. The learning process is
hindered when training data is scarce for a task. Even with sufficient data,
learning patterns for the long tail of the lexical distribution is challenging.
In this thesis, we focus on understanding certain potentials of contexts in
neural models and design augmentation models to benefit from them. We focus on
machine translation as an important instance of the more general language
understanding problem. To translate from a source language to a target
language, a neural model has to understand the meaning of constituents in the
provided context and generate constituents with the same meanings in the target
language. This task accentuates the value of capturing nuances of language and
the necessity of generalization from few observations. The main problem we
study in this thesis is what neural machine translation models learn from data
and how we can devise more focused contexts to enhance this learning. Looking
more in-depth into the role of context and the impact of data on learning
models is essential to advance the NLP field. Moreover, it helps highlight the
vulnerabilities of current neural networks and provides insights into designing
more robust models.Comment: PhD dissertation defended on November 10th, 202
Neural Machine Translation with Word Predictions
In the encoder-decoder architecture for neural machine translation (NMT), the
hidden states of the recurrent structures in the encoder and decoder carry the
crucial information about the sentence.These vectors are generated by
parameters which are updated by back-propagation of translation errors through
time. We argue that propagating errors through the end-to-end recurrent
structures are not a direct way of control the hidden vectors. In this paper,
we propose to use word predictions as a mechanism for direct supervision. More
specifically, we require these vectors to be able to predict the vocabulary in
target sentence. Our simple mechanism ensures better representations in the
encoder and decoder without using any extra data or annotation. It is also
helpful in reducing the target side vocabulary and improving the decoding
efficiency. Experiments on Chinese-English and German-English machine
translation tasks show BLEU improvements by 4.53 and 1.3, respectivelyComment: Accepted at EMNLP201
Adapting Sequence to Sequence models for Text Normalization in Social Media
Social media offer an abundant source of valuable raw data, however informal
writing can quickly become a bottleneck for many natural language processing
(NLP) tasks. Off-the-shelf tools are usually trained on formal text and cannot
explicitly handle noise found in short online posts. Moreover, the variety of
frequently occurring linguistic variations presents several challenges, even
for humans who might not be able to comprehend the meaning of such posts,
especially when they contain slang and abbreviations. Text Normalization aims
to transform online user-generated text to a canonical form. Current text
normalization systems rely on string or phonetic similarity and classification
models that work on a local fashion. We argue that processing contextual
information is crucial for this task and introduce a social media text
normalization hybrid word-character attention-based encoder-decoder model that
can serve as a pre-processing step for NLP applications to adapt to noisy text
in social media. Our character-based component is trained on synthetic
adversarial examples that are designed to capture errors commonly found in
online user-generated text. Experiments show that our model surpasses neural
architectures designed for text normalization and achieves comparable
performance with state-of-the-art related work.Comment: Accepted at the 13th International AAAI Conference on Web and Social
Media (ICWSM 2019
Exploring Diversity in Back Translation for Low-Resource Machine Translation
Back translation is one of the most widely used methods for improving the
performance of neural machine translation systems. Recent research has sought
to enhance the effectiveness of this method by increasing the 'diversity' of
the generated translations. We argue that the definitions and metrics used to
quantify 'diversity' in previous work have been insufficient. This work puts
forward a more nuanced framework for understanding diversity in training data,
splitting it into lexical diversity and syntactic diversity. We present novel
metrics for measuring these different aspects of diversity and carry out
empirical analysis into the effect of these types of diversity on final neural
machine translation model performance for low-resource
EnglishTurkish and mid-resource
EnglishIcelandic. Our findings show that generating back
translation using nucleus sampling results in higher final model performance,
and that this method of generation has high levels of both lexical and
syntactic diversity. We also find evidence that lexical diversity is more
important than syntactic for back translation performance
- …