115 research outputs found
Proceedings
Proceedings of the Workshop on Annotation and
Exploitation of Parallel Corpora AEPC 2010.
Editors: Lars Ahrenberg, Jörg Tiedemann and Martin Volk.
NEALT Proceedings Series, Vol. 10 (2010), 98 pages.
© 2010 The editors and contributors.
Published by
Northern European Association for Language
Technology (NEALT)
http://omilia.uio.no/nealt .
Electronically published at
Tartu University Library (Estonia)
http://hdl.handle.net/10062/15893
What does Attention in Neural Machine Translation Pay Attention to?
Attention in neural machine translation provides the possibility to encode
relevant parts of the source sentence at each translation step. As a result,
attention is considered to be an alignment model as well. However, there is no
work that specifically studies attention and provides analysis of what is being
learned by attention models. Thus, the question still remains that how
attention is similar or different from the traditional alignment. In this
paper, we provide detailed analysis of attention and compare it to traditional
alignment. We answer the question of whether attention is only capable of
modelling translational equivalent or it captures more information. We show
that attention is different from alignment in some cases and is capturing
useful information other than alignments.Comment: To appear in IJCNLP 201
Recurrent Memory Networks for Language Modeling
Recurrent Neural Networks (RNN) have obtained excellent result in many
natural language processing (NLP) tasks. However, understanding and
interpreting the source of this success remains a challenge. In this paper, we
propose Recurrent Memory Network (RMN), a novel RNN architecture, that not only
amplifies the power of RNN but also facilitates our understanding of its
internal functioning and allows us to discover underlying patterns in data. We
demonstrate the power of RMN on language modeling and sentence completion
tasks. On language modeling, RMN outperforms Long Short-Term Memory (LSTM)
network on three large German, Italian, and English dataset. Additionally we
perform in-depth analysis of various linguistic dimensions that RMN captures.
On Sentence Completion Challenge, for which it is essential to capture sentence
coherence, our RMN obtains 69.2% accuracy, surpassing the previous
state-of-the-art by a large margin.Comment: 8 pages, 6 figures. Accepted at NAACL 201
Linguistic knowledge-based vocabularies for Neural Machine Translation
This article has been published in a revised form in Natural Language Engineering https://doi.org/10.1017/S1351324920000364. This version is free to view and download for private research and study only. Not for re-distribution, re-sale or use in derivative works. © Cambridge University PressNeural Networks applied to Machine Translation need a finite vocabulary to express textual information as a sequence of discrete tokens. The currently dominant subword vocabularies exploit statistically-discovered common parts of words to achieve the flexibility of character-based vocabularies without delegating the whole learning of word formation to the neural network. However, they trade this for the inability to apply word-level token associations, which limits their use in semantically-rich areas and prevents some transfer learning approaches e.g. cross-lingual pretrained embeddings, and reduces their interpretability. In this work, we propose new hybrid linguistically-grounded vocabulary definition strategies that keep both the advantages of subword vocabularies and the word-level associations, enabling neural networks to profit from the derived benefits. We test the proposed approaches in both morphologically rich and poor languages, showing that, for the former, the quality in the translation of out-of-domain texts is improved with respect to a strong subword baseline.This work is partially supported by Lucy Software / United Language Group (ULG) and the Catalan Agency for Management of University and Research Grants (AGAUR) through an Industrial PhD Grant. This work is also supported in part by the Spanish Ministerio de Economa y Competitividad, the European Regional Development Fund and the Agencia Estatal de Investigacin, through the postdoctoral senior grant Ramn y Cajal, contract TEC2015-69266-P (MINECO/FEDER,EU) and contract PCIN-2017-079 (AEI/MINECO).Peer ReviewedPostprint (author's final draft
Linguistic Input Features Improve Neural Machine Translation
Neural machine translation has recently achieved impressive results, while
using little in the way of external linguistic information. In this paper we
show that the strong learning capability of neural MT models does not make
linguistic features redundant; they can be easily incorporated to provide
further improvements in performance. We generalize the embedding layer of the
encoder in the attentional encoder--decoder architecture to support the
inclusion of arbitrary features, in addition to the baseline word feature. We
add morphological features, part-of-speech tags, and syntactic dependency
labels as input features to EnglishGerman, and English->Romanian neural
machine translation systems. In experiments on WMT16 training and test sets, we
find that linguistic input features improve model quality according to three
metrics: perplexity, BLEU and CHRF3. An open-source implementation of our
neural MT system is available, as are sample files and configurations.Comment: WMT16 final version; new EN-RO result
GI-tutor: grammar-checking for (Italian) students of German
Computers have played a significant role in second language learning as they can offer different types of language activities and provide learners with immediate and appropriate feedbacks. In ICALL systems, learning is activated by focusing the learner's attention to the correct form and comparing it to the wrong one. Feedback offers an explicit explanation of the mistakes made by the student.
The focus of this paper is a grammar checker designed for Italian native speakers learning German - GI-tutor. The system's structure was taken from a previous study [28] and enforced to analyse a conspicuous number of sentences, with different sentence and phrase structures. The lexicon used has been manually organized at the beginning, with some 8,000 entries overall; then, an enlargement has been obtained through an adaptation of the lexicon made available by Hamburg University Constraint Dependency Grammar (JWCDG) and downloaded from their website . A corpus containing wrong sentences was expanded by extracting data from exams written by first-year students of the German course at the University Ca' Foscari. The errors were then classified in order to obtain a general statistical analysis of the main problems encountered when learning German.
Attention was given also to parsers and their use and functionality in language learning. Furthermore, the performance of the constituency grammar checker was evaluated to determine the types and frequencies of errors it can successfully diagnose. This was done by comparing it to ParZu - a generic German dependency parser developed at the University of Zürich and to Stanford Parser for German
A tree does not make a well-formed sentence: Improving syntactic string-to-tree statistical machine translation with more linguistic knowledge
AbstractSynchronous context-free grammars (SCFGs) can be learned from parallel texts that are annotated with target-side syntax, and can produce translations by building target-side syntactic trees from source strings. Ideally, producing syntactic trees would entail that the translation is grammatically well-formed, but in reality, this is often not the case. Focusing on translation into German, we discuss various ways in which string-to-tree translation models over- or undergeneralise. We show how these problems can be addressed by choosing a suitable parser and modifying its output, by introducing linguistic constraints that enforce morphological agreement and constrain subcategorisation, and by modelling the productive generation of German compounds
Neural versus Phrase-Based Machine Translation Quality: a Case Study
Within the field of Statistical Machine Translation (SMT), the neural approach (NMT) has recently emerged as the first technology able to challenge the long-standing dominance of phrase-based approaches (PBMT). In particular, at the IWSLT 2015 evaluation campaign, NMT outperformed well established state-of-the-art PBMT systems on English-German, a language pair known to be particularly hard because of morphology and syntactic differences. To understand in what respects NMT provides better translation quality than PBMT, we perform a detailed analysis of neural versus phrase-based SMT outputs, leveraging high quality post-edits performed by professional translators on the IWSLT data. For the first time, our analysis provides useful insights on what linguistic phenomena are best modeled by neural models -- such as the reordering of verbs -- while pointing out other aspects that remain to be improved
Edinburgh's Statistical Machine Translation Systems for WMT16
This paper describes the University of Edinburgh’s
phrase-based and syntax-based
submissions to the shared translation tasks
of the ACL 2016 First Conference on Machine
Translation (WMT16). We submitted
five phrase-based and five syntaxbased
systems for the news task, plus one
phrase-based system for the biomedical
task
- …