Most cross-lingual embedding mapping algorithms assume the optimised
transformation functions to be linear. Recent studies showed that on some
occasions, learning a linear mapping does not work, indicating that the
commonly-used assumption may fail. However, it still remains unclear under
which conditions the linearity of cross-lingual embedding mappings holds. In
this paper, we rigorously explain that the linearity assumption relies on the
consistency of analogical relations encoded by multilingual embeddings. We did
extensive experiments to validate this claim. Empirical results based on the
analogy completion benchmark and the BLI task demonstrate a strong correlation
between whether mappings capture analogical information and are linear.Comment: Comments welcome

li, Chen

Lin, Chenghua

Peng, Xutan

Stevenson, Mark

English

arXiv

The technique of Cross-Lingual Word Embedding (CLWE) plays a fundamental role
in tackling Natural Language Processing challenges for low-resource languages.
Its dominant approaches assumed that the relationship between embeddings could
be represented by a linear mapping, but there has been no exploration of the
conditions under which this assumption holds. Such a research gap becomes very
critical recently, as it has been evidenced that relaxing mappings to be
non-linear can lead to better performance in some cases. We, for the first
time, present a theoretical analysis that identifies the preservation of
analogies encoded in monolingual word embeddings as a necessary and sufficient
condition for the ground-truth CLWE mapping between those embeddings to be
linear. On a novel cross-lingual analogy dataset that covers five
representative analogy categories for twelve distinct languages, we carry out
experiments which provide direct empirical support for our theoretical claim.
These results offer additional insight into the observations of other
researchers and contribute inspiration for the development of more effective
cross-lingual representation learning strategies

Li, Chen

arXiv.org e-Print Archive

Understanding Linearity of Cross-Lingual Word Embedding Mappings

The technique of Cross-Lingual Word Embedding (CLWE) plays a fundamental role in tackling Natural Language Processing challenges for low-resource languages. Its dominant approaches assumed that the relationship between embeddings could be represented by a linear mapping, but there has been no exploration of the conditions under which this assumption holds. Such a research gap becomes very critical recently, as it has been evidenced that relaxing mappings to be non-linear can lead to better performance in some cases. We, for the first time, present a theoretical analysis that identifies the preservation of analogies encoded in monolingual word embeddings as a *necessary and sufficient* condition for the ground-truth CLWE mapping between those embeddings to be linear. On a novel cross-lingual analogy dataset that covers five representative analogy categories for twelve distinct languages, we carry out experiments which provide direct empirical support for our theoretical claim. These results offer additional insight into the observations of other researchers and contribute inspiration for the development of more effective cross-lingual representation learning strategies

Peng, X.

Stevenson, R.

Lin, C.

Li, C.

White Rose Research Online

Understanding linearity of cross-lingual word embedding mappings

https://eprints.whiterose.ac.uk/188519/1/TMLR_analogy.pdf

Revisiting the linearity in cross-lingual embedding mappings: from a
  perspective of word analogies

Revisiting the linearity in cross-lingual embedding mappings: from a perspective of word analogies

Abstract

Similar works

Full text

Available Versions

arXiv.org e-Print Archive

White Rose Research Online