21 research outputs found
Machine translation of morphologically rich languages using deep neural networks
This thesis addresses some of the challenges of translating morphologically rich languages (MRLs). Words in MRLs have more complex structures than those in other languages, so that a word can be viewed as a hierarchical structure with several internal subunits. Accordingly, word-based models in which words are treated as atomic units are not suitable for this set of languages. As a commonly used and eff ective solution, morphological decomposition is applied to segment words into atomic and meaning-preserving units, but this raises other types of problems some of which we study here. We mainly use neural networks (NNs) to perform machine translation (MT) in our research and study their diff erent properties. However, our research is not limited to neural models alone as we also consider some of the difficulties of conventional MT methods.
First we try to model morphologically complex words (MCWs) and provide better word-level representations. Words are symbolic concepts which are represented numerically in order to be used in NNs. Our first goal is to tackle this problem and find the best representation for MCWs. In the next step we focus on language modeling (LM) and work at the sentence level. We propose new morpheme-segmentation models by which we finetune existing LMs for MRLs. In this part of our research we try to find the most efficient neural language model for MRLs. After providing word- and sentence-level neural information in the first two steps, we try to use such information to enhance the translation quality in the statistical machine translation (SMT) pipeline using several diff erent models. Accordingly, the main goal in this part is to find methods by which deep neural networks (DNNs) can improve SMT.
One of the main interests of the thesis is to study neural machine translation (NMT) engines from diff erent perspectives, and finetune them to work with MRLs. In the last step we target this problem and perform end-to-end sequence modeling via NN-based models. NMT engines have recently improved significantly and perform as well as state-of-the-art systems, but still have serious problems with morphologically complex constituents. This shortcoming of NMT is studied in two separate chapters in the thesis, where in one chapter we investigate the impact of diff erent non-linguistic morpheme-segmentation models on the NMT pipeline, and in the other one we benefit from a linguistically motivated morphological analyzer and propose a novel neural architecture particularly for translating from MRLs. Our overall goal for this part of the research is to find the most suitable neural architecture to translate MRLs.
We evaluated our models on diff erent MRLs such as Czech, Farsi, German, Russian, and Turkish, and observed significant improvements. The main goal targeted in this research was to incorporate morphological information into MT and define architectures which are able to model the complex nature of MRLs. The results obtained from our experimental studies confirm that we were able to achieve our goal
Tailoring neural architectures for translating from morphologically rich languages
A morphologically complex word (MCW) is a hierarchical constituent with meaning-preserving
subunits, so word-based models which rely on surface forms might not be powerful enough
to translate such structures. When translating from morphologically rich languages (MRLs), a
source word could be mapped to several words or even a full sentence on the target side, which
means an MCW should not be treated as an atomic unit. In order to provide better translations
for MRLs, we boost the existing neural machine translation (NMT) architecture with a doublechannel encoder and a double-attentive decoder. The main goal targeted in this research is to
provide richer information on the encoder side and redesign the decoder accordingly to benefit
from such information. Our experimental results demonstrate that we could achieve our goal as
the proposed model outperforms existing subword- and character-based architectures and showed
significant improvements on translating from German, Russian, and Turkish into English
Enriching phrase tables for statistical machine translation using mixed embeddings
The phrase table is considered to be the main bilingual resource for the phrase-based statistical machine translation (PBSMT) model. During translation, a source sentence is decomposed
into several phrases. The best match of each source phrase is selected among several target-side
counterparts within the phrase table, and processed by the decoder to generate a sentence-level
translation. The best match is chosen according to several factors, including a set of bilingual
features. PBSMT engines by default provide four probability scores in phrase tables which are
considered as the main set of bilingual features. Our goal is to enrich that set of features, as a
better feature set should yield better translations. We propose new scores generated by a Convolutional Neural Network (CNN) which indicate the semantic relatedness of phrase pairs. We
evaluate our model in different experimental settings with different language pairs. We observe
significant improvements when the proposed features are incorporated into the PBSMT pipeline
Benchmarking SMT performance for Farsi using the TEP++ Corpus
Statistical machine translation (SMT) suffers from various problems which are exacerbated where training data is in short
supply. In this paper we address the data
sparsity problem in the Farsi (Persian) language and introduce a new parallel corpus, TEP++. Compared to previous results the new dataset is more efficient for
Farsi SMT engines and yields better output. In our experiments using TEP++ as
bilingual training data and BLEU as a metric, we achieved improvements of +11.17
(60%) and +7.76 (63.92%) in the Farsi–
English and English–Farsi directions, respectively. Furthermore we describe an
engine (SF2FF) to translate between formal and informal Farsi which in terms of
syntax and terminology can be seen as
different languages. The SF2FF engine
also works as an intelligent normalizer for
Farsi texts. To demonstrate its use, SF2FF
was used to clean the IWSLT–2013 dataset
to produce normalized data, which gave
improvements in translation quality over
FBK’s Farsi engine when used as training
dat
Improving character-based decoding using target-side morphological information for neural machine translation
Recently, neural machine translation
(NMT) has emerged as a powerful alternative to conventional statistical approaches.
However, its performance drops considerably in the presence of morphologically
rich languages (MRLs). Neural engines
usually fail to tackle the large vocabulary
and high out-of-vocabulary (OOV) word
rate of MRLs. Therefore, it is not suitable
to exploit existing word-based models
to translate this set of languages. In this
paper, we propose an extension to the
state-of-the-art model of Chung et al.
(2016), which works at the character level
and boosts the decoder with target-side
morphological information. In our architecture, an additional morphology table
is plugged into the model. Each time the
decoder samples from a target vocabulary,
the table sends auxiliary signals from the
most relevant affixes in order to enrich the
decoder’s current state and constrain it to
provide better predictions. We evaluated
our model to translate English into German, Russian, and Turkish as three MRLs
and observed significant improvements
Providing morphological information for SMT using neural networks
Treating morphologically complex words (MCWs) as atomic units in translation would not
yield a desirable result. Such words are complicated constituents with meaningful subunits. A
complex word in a morphologically rich language (MRL) could be associated with a number of
words or even a full sentence in a simpler language, which means the surface form of complex
words should be accompanied with auxiliary morphological information in order to provide a
precise translation and a better alignment. In this paper we follow this idea and propose two
different methods to convey such information for statistical machine translation (SMT) models. In the first model we enrich factored SMT engines by introducing a new morphological
factor which relies on subword-aware word embeddings. In the second model we focus on the
language-modeling component. We explore a subword-level neural language model (NLM) to
capture sequence-, word- and subword-level dependencies. Our NLM is able to approximate
better scores for conditional word probabilities, so the decoder generates more fluent translations. We studied two languages Farsi and German in our experiments and observed significant
improvements for both of them
Dynamic Position Encoding for Transformers
Recurrent models have been dominating the field of neural machine translation
(NMT) for the past few years. Transformers \citep{vaswani2017attention}, have
radically changed it by proposing a novel architecture that relies on a
feed-forward backbone and self-attention mechanism. Although Transformers are
powerful, they could fail to properly encode sequential/positional information
due to their non-recurrent nature. To solve this problem, position embeddings
are defined exclusively for each time step to enrich word information. However,
such embeddings are fixed after training regardless of the task and the word
ordering system of the source or target language.
In this paper, we propose a novel architecture with new position embeddings
depending on the input text to address this shortcoming by taking the order of
target words into consideration. Instead of using predefined position
embeddings, our solution \textit{generates} new embeddings to refine each
word's position information. Since we do not dictate the position of source
tokens and learn them in an end-to-end fashion, we refer to our method as
\textit{dynamic} position encoding (DPE). We evaluated the impact of our model
on multiple datasets to translate from English into German, French, and Italian
and observed meaningful improvements in comparison to the original Transformer
ALP-KD: Attention-Based Layer Projection for Knowledge Distillation
Knowledge distillation is considered as a training and compression strategy
in which two neural networks, namely a teacher and a student, are coupled
together during training. The teacher network is supposed to be a trustworthy
predictor and the student tries to mimic its predictions. Usually, a student
with a lighter architecture is selected so we can achieve compression and yet
deliver high-quality results. In such a setting, distillation only happens for
final predictions whereas the student could also benefit from teacher's
supervision for internal components.
Motivated by this, we studied the problem of distillation for intermediate
layers. Since there might not be a one-to-one alignment between student and
teacher layers, existing techniques skip some teacher layers and only distill
from a subset of them. This shortcoming directly impacts quality, so we instead
propose a combinatorial technique which relies on attention. Our model fuses
teacher-side information and takes each layer's significance into
consideration, then performs distillation between combined teacher layers and
those of the student. Using our technique, we distilled a 12-layer BERT (Devlin
et al. 2019) into 6-, 4-, and 2-layer counterparts and evaluated them on GLUE
tasks (Wang et al. 2018). Experimental results show that our combinatorial
approach is able to outperform other existing techniques.Comment: AAAI 2021. This work has been done while Peyman Passban was at Huawe