13 research outputs found
Zero-shot Neural Transfer for Cross-lingual Entity Linking
Cross-lingual entity linking maps an entity mention in a source language to
its corresponding entry in a structured knowledge base that is in a different
(target) language. While previous work relies heavily on bilingual lexical
resources to bridge the gap between the source and the target languages, these
resources are scarce or unavailable for many low-resource languages. To address
this problem, we investigate zero-shot cross-lingual entity linking, in which
we assume no bilingual lexical resources are available in the source
low-resource language. Specifically, we propose pivot-based entity linking,
which leverages information from a high-resource "pivot" language to train
character-level neural entity linking models that are transferred to the source
low-resource language in a zero-shot manner. With experiments on 9 low-resource
languages and transfer through a total of 54 languages, we show that our
proposed pivot-based framework improves entity linking accuracy 17% (absolute)
on average over the baseline systems, for the zero-shot scenario. Further, we
also investigate the use of language-universal phonological representations
which improves average accuracy (absolute) by 36% when transferring between
languages that use different scripts.Comment: To appear in AAAI 201
Exploiting Cross-Lingual Subword Similarities in Low-Resource Document Classification
Text classification must sometimes be applied in a low-resource language with
no labeled training data. However, training data may be available in a related
language. We investigate whether character-level knowledge transfer from a
related language helps text classification. We present a cross-lingual document
classification framework (CACO) that exploits cross-lingual subword similarity
by jointly training a character-based embedder and a word-based classifier. The
embedder derives vector representations for input words from their written
forms, and the classifier makes predictions based on the word vectors. We use a
joint character representation for both the source language and the target
language, which allows the embedder to generalize knowledge about source
language words to target language words with similar forms. We propose a
multi-task objective that can further improve the model if additional
cross-lingual or monolingual resources are available. Experiments confirm that
character-level knowledge transfer is more data-efficient than word-level
transfer between related languages.Comment: AAAI 202
Towards zero-shot cross-lingual named entity disambiguation
[EN]In cross-Lingual Named Entity Disambiguation (XNED) the task is to link Named Entity mentions in text in some native language to English entities in a knowledge graph. XNED systems usually require training data for each native language, limiting their application for low resource languages with small amounts of training data. Prior work have proposed so-called zero-shot transfer systems which are only trained in English training data, but required native prior probabilities of entities with respect to mentions, which had to be estimated from native training examples, limiting their practical interest. In this work we present a zero-shot XNED architecture where, instead of a single disambiguation model, we have a model for each possible mention string, thus eliminating the need for native prior probabilities. Our system improves over prior work in XNED datasets in Spanish and Chinese by 32 and 27 points, and matches the systems which do require native prior information. We experiment with different multilingual transfer strategies, showing that better results are obtained with a purpose-built multilingual pre-training method compared to state-of-the-art generic multilingual models such as XLM-R. We also discovered, surprisingly, that English is not necessarily the most effective zero-shot training language for XNED into English. For instance, Spanish is more effective when training a zero-shot XNED system that dis-ambiguates Basque mentions with respect to an English knowledge graph.This work has been partially funded by the Basque Government (IXA excellence research group (IT1343-19) and DeepText project), Project BigKnowledge (Ayudas Fundacion BBVA a equipos de investigacion cientifica 2018) and via the IARPA BETTER Program contract 2019-19051600006 (ODNI, IARPA activity). Ander Barrena enjoys a post-doctoral grant ESPDOC18/101 from the UPV/EHU and also acknowledges the support of the NVIDIA Corporation with the donation of a Titan V GPU used for this research. The author thankfully acknowledges the computer resources at CTE-Power9 + V100 and technical support provided by Barcelona Supercomputing Center (RES-IM-2020-1-0020)
Parameter Space Factorization for Zero-Shot Learning across Tasks and Languages
Most combinations of NLP tasks and language varieties lack in-domain examples
for supervised training because of the paucity of annotated data. How can
neural models make sample-efficient generalizations from task-language
combinations with available data to low-resource ones? In this work, we propose
a Bayesian generative model for the space of neural parameters. We assume that
this space can be factorized into latent variables for each language and each
task. We infer the posteriors over such latent variables based on data from
seen task-language combinations through variational inference. This enables
zero-shot classification on unseen combinations at prediction time. For
instance, given training data for named entity recognition (NER) in Vietnamese
and for part-of-speech (POS) tagging in Wolof, our model can perform accurate
predictions for NER in Wolof. In particular, we experiment with a typologically
diverse sample of 33 languages from 4 continents and 11 families, and show that
our model yields comparable or better results than state-of-the-art, zero-shot
cross-lingual transfer methods. Moreover, we demonstrate that approximate
Bayesian model averaging results in smoother predictive distributions, whose
entropy inversely correlates with accuracy. Hence, the proposed framework also
offers robust estimates of prediction uncertainty. Our code is located at
github.com/cambridgeltl/parameter-factorizatio
Pivot-based candidate retrieval for cross-lingual entity linking
Entity candidate retrieval plays a critical role in cross-lingual entity linking (XEL). In XEL, entity candidate retrieval needs to retrieve a list of plausible candidate entities from a large knowledge graph in a target language given a piece of text in a sentence or question, namely a mention, in a source language. Existing works mainly fall into two categories: lexicon-based and semantic-based approaches. The lexicon-based approach usually creates cross-lingual and mention-entity lexicons, which is effective but relies heavily on bilingual resources (e.g. inter-language links in Wikipedia). The semantic-based approach maps mentions and entities in different languages to a unified embedding space, which reduces dependence on large-scale bilingual dictionaries. However, its effectiveness is limited by the representation capacity of fixed-length vectors. In this paper, we propose a pivot-based approach which inherits the advantages of the aforementioned two approaches while avoiding their limitations. It takes an intermediary set of plausible target-language mentions as pivots to bridge the two types of gaps: cross-lingual gap and mention-entity gap. Specifically, it first converts mentions in the source language into an intermediary set of plausible mentions in the target language by cross-lingual semantic retrieval and a selective mechanism, and then retrieves candidate entities based on the generated mentions by lexical retrieval. The proposed approach only relies on a small bilingual word dictionary, and fully exploits the benefits of both lexical and semantic matching. Experimental results on two challenging cross-lingual entity linking datasets spanning over 11 languages show that the pivot-based approach outperforms both the lexicon-based and semantic-based approach by a large margin