806 research outputs found
Revisiting the linearity in cross-lingual embedding mappings: from a perspective of word analogies
Most cross-lingual embedding mapping algorithms assume the optimised
transformation functions to be linear. Recent studies showed that on some
occasions, learning a linear mapping does not work, indicating that the
commonly-used assumption may fail. However, it still remains unclear under
which conditions the linearity of cross-lingual embedding mappings holds. In
this paper, we rigorously explain that the linearity assumption relies on the
consistency of analogical relations encoded by multilingual embeddings. We did
extensive experiments to validate this claim. Empirical results based on the
analogy completion benchmark and the BLI task demonstrate a strong correlation
between whether mappings capture analogical information and are linear.Comment: Comments welcome
Analogy Training Multilingual Encoders
Language encoders encode words and phrases in ways that capture their local semantic relatedness, but are known to be globally inconsistent. Global inconsistency can seemingly be corrected for, in part, by leveraging signals from knowledge bases, but previous results are partial and limited to monolingual English encoders. We extract a large-scale multilingual, multi-word analogy dataset from Wikidata for diagnosing and correcting for global inconsistencies and implement a four-way Siamese BERT architecture for grounding multilingual BERT (mBERT) in Wikidata through analogy training. We show that analogy training not only improves the global consistency of mBERT, as well as the isomorphism of language-specific subspaces, but also leads to significant gains on downstream tasks such as bilingual dictionary induction and sentence retrieval
DebIE: A Platform for Implicit and Explicit Debiasing of Word Embedding Spaces
Recent research efforts in NLP have demonstrated that distributional word
vector spaces often encode stereotypical human biases, such as racism and
sexism. With word representations ubiquitously used in NLP models and
pipelines, this raises ethical issues and jeopardizes the fairness of language
technologies. While there exists a large body of work on bias measures and
debiasing methods, to date, there is no platform that would unify these
research efforts and make bias measuring and debiasing of representation spaces
widely accessible. In this work, we present DebIE, the first integrated
platform for (1) measuring and (2) mitigating bias in word embeddings. Given an
(i) embedding space (users can choose between the predefined spaces or upload
their own) and (ii) a bias specification (users can choose between existing
bias specifications or create their own), DebIE can (1) compute several
measures of implicit and explicit bias and modify the embedding space by
executing two (mutually composable) debiasing models. DebIE's functionality can
be accessed through four different interfaces: (a) a web application, (b) a
desktop application, (c) a REST-ful API, and (d) as a command-line application.
DebIE is available at: debie.informatik.uni-mannheim.de.Comment: Accepted as EACL21 Dem
Revisiting the linearity in cross-lingual embedding mappings : from a perspective of word analogies
Most cross-lingual embedding mapping algorithms assume the optimised transformation functions to be linear. Recent studies showed that on some occasions, learning a linear mapping does not work, indicating that the commonly-used assumption may fail. However, it still remains unclear under which conditions the linearity of cross-lingual embedding mappings holds. In this paper, we rigorously explain that the linearity assumption relies on the consistency of analogical relations encoded by multilingual embeddings. We did extensive experiments to validate this claim. Empirical results based on the analogy completion benchmark and the BLI task demonstrate a strong correlation between whether mappings capture analogical information and are linear
Embedding Multilingual and Relational Data Using Linear Mappings
This thesis presents our research on the embedding method, a machine learning technique that encodes real-world signals into high-dimensional vectors. Specifically, we focus on a family of algorithms whose backbone is one simple yet elegant type of topological operation, the linear mapping, aka. linear transformation or vector space homomorphism. Past studies have shown the usefulness of these approaches for modelling complex data, such as lexicons from different languages and networks storing factual relations. However, they also exhibit crucial limitations, including a lack of theoretical justifications, precision drop in challenging setups, and considerable environmental impact during training, among others.
To bridge these gaps, we first identify the unnoticed link between the success of linear Cross-Lingual Word Embedding (CLWE) mappings and the preservation of the implicit analogy relation, using both theoretical and empirical evidence. Next, we propose a post-hoc L1-norm rotation step which substantially improves the performance of existing CLWE mappings. Then, beyond solving conventional questions where only modern languages are involved, we extend the application of CLWE mappings to summarising lengthy and opaque historical text. Finally, motivated by the learning procedure of CLWE models, we adopt linear mappings to optimise Knowledge Graph Embeddings (KGEs) iteratively, significantly reducing the carbon footprint required to train the algorithm
- …