1,175 research outputs found
Cross-lingual syntactically informed distributed word representations
We develop a novel cross-lingual word representation model which injects syntactic information through dependency-based contexts into a shared cross-lingual word vector space. The model, termed CL-DepEmb, is based on the following assumptions: (1) dependency relations are largely language-independent, at least for related languages and prominent dependency links such as direct objects, as evidenced by the Universal Dependencies project; (2) word translation equivalents take similar grammatical roles in a sentence and are therefore substitutable within their syntactic contexts. Experiments with several language pairs on word similarity and bilingual lexicon induction, two fundamental semantic tasks emphasising semantic similarity, suggest the usefulness of the proposed syntactically informed cross-lingual word vector spaces. Improvements are observed in both tasks over standard cross-lingual "offline mapping" baselines trained using the same setup and an equal level of bilingual supervision
Improving the translation environment for professional translators
When using computer-aided translation systems in a typical, professional translation workflow, there are several stages at which there is room for improvement. The SCATE (Smart Computer-Aided Translation Environment) project investigated several of these aspects, both from a human-computer interaction point of view, as well as from a purely technological side.
This paper describes the SCATE research with respect to improved fuzzy matching, parallel treebanks, the integration of translation memories with machine translation, quality estimation, terminology extraction from comparable texts, the use of speech recognition in the translation process, and human computer interaction and interface design for the professional translation environment. For each of these topics, we describe the experiments we performed and the conclusions drawn, providing an overview of the highlights of the entire SCATE project
Is Supervised Syntactic Parsing Beneficial for Language Understanding? An Empirical Investigation
Traditional NLP has long held (supervised) syntactic parsing necessary for
successful higher-level language understanding. The recent advent of end-to-end
neural language learning, self-supervised via language modeling (LM), and its
success on a wide range of language understanding tasks, however, questions
this belief. In this work, we empirically investigate the usefulness of
supervised parsing for semantic language understanding in the context of
LM-pretrained transformer networks. Relying on the established fine-tuning
paradigm, we first couple a pretrained transformer with a biaffine parsing
head, aiming to infuse explicit syntactic knowledge from Universal Dependencies
(UD) treebanks into the transformer. We then fine-tune the model for language
understanding (LU) tasks and measure the effect of the intermediate parsing
training (IPT) on downstream LU performance. Results from both monolingual
English and zero-shot language transfer experiments (with intermediate
target-language parsing) show that explicit formalized syntax, injected into
transformers through intermediate supervised parsing, has very limited and
inconsistent effect on downstream LU performance. Our results, coupled with our
analysis of transformers' representation spaces before and after intermediate
parsing, make a significant step towards providing answers to an essential
question: how (un)availing is supervised parsing for high-level semantic
language understanding in the era of large neural models
Cross-lingual Word Clusters for Direct Transfer of Linguistic Structure
It has been established that incorporating word cluster features derived from large unlabeled corpora can significantly improve prediction of linguistic structure. While previous work has focused primarily on English, we extend these results to other languages along two dimensions. First, we show that these results hold true for a number of languages across families. Second, and more interestingly, we provide an algorithm for inducing cross-lingual clusters and we show that features derived from these clusters significantly improve the accuracy of cross-lingual structure prediction. Specifically, we show that by augmenting direct-transfer systems with cross-lingual cluster features, the relative error of delexicalized dependency parsers, trained on English treebanks and transferred to foreign languages, can be reduced by up to 13%. When applying the same method to direct transfer of named-entity recognizers, we observe relative improvements of up to 26%
Revisiting the Context Window for Cross-lingual Word Embeddings
Existing approaches to mapping-based cross-lingual word embeddings are based
on the assumption that the source and target embedding spaces are structurally
similar. The structures of embedding spaces largely depend on the co-occurrence
statistics of each word, which the choice of context window determines. Despite
this obvious connection between the context window and mapping-based
cross-lingual embeddings, their relationship has been underexplored in prior
work. In this work, we provide a thorough evaluation, in various languages,
domains, and tasks, of bilingual embeddings trained with different context
windows. The highlight of our findings is that increasing the size of both the
source and target window sizes improves the performance of bilingual lexicon
induction, especially the performance on frequent nouns.Comment: ACL202
Meta-learning for fast cross-lingual adaptation in dependency parsing
Meta-learning, or learning to learn, is a technique that can help to overcome
resource scarcity in cross-lingual NLP problems, by enabling fast adaptation to
new tasks. We apply model-agnostic meta-learning (MAML) to the task of
cross-lingual dependency parsing. We train our model on a diverse set of
languages to learn a parameter initialization that can adapt quickly to new
languages. We find that meta-learning with pre-training can significantly
improve upon the performance of language transfer and standard supervised
learning baselines for a variety of unseen, typologically diverse, and
low-resource languages, in a few-shot learning setup
- …