4,972 research outputs found
ProKD: An Unsupervised Prototypical Knowledge Distillation Network for Zero-Resource Cross-Lingual Named Entity Recognition
For named entity recognition (NER) in zero-resource languages, utilizing
knowledge distillation methods to transfer language-independent knowledge from
the rich-resource source languages to zero-resource languages is an effective
means. Typically, these approaches adopt a teacher-student architecture, where
the teacher network is trained in the source language, and the student network
seeks to learn knowledge from the teacher network and is expected to perform
well in the target language. Despite the impressive performance achieved by
these methods, we argue that they have two limitations. Firstly, the teacher
network fails to effectively learn language-independent knowledge shared across
languages due to the differences in the feature distribution between the source
and target languages. Secondly, the student network acquires all of its
knowledge from the teacher network and ignores the learning of target
language-specific knowledge. Undesirably, these limitations would hinder the
model's performance in the target language. This paper proposes an unsupervised
prototype knowledge distillation network (ProKD) to address these issues.
Specifically, ProKD presents a contrastive learning-based prototype alignment
method to achieve class feature alignment by adjusting the distance among
prototypes in the source and target languages, boosting the teacher network's
capacity to acquire language-independent knowledge. In addition, ProKD
introduces a prototypical self-training method to learn the intrinsic structure
of the language by retraining the student network on the target data using
samples' distance information from prototypes, thereby enhancing the student
network's ability to acquire language-specific knowledge. Extensive experiments
on three benchmark cross-lingual NER datasets demonstrate the effectiveness of
our approach.Comment: AAAI 202
Transfer to a Low-Resource Language via Close Relatives: The Case Study on Faroese
Multilingual language models have pushed state-of-the-art in cross-lingual
NLP transfer. The majority of zero-shot cross-lingual transfer, however, use
one and the same massively multilingual transformer (e.g., mBERT or XLM-R) to
transfer to all target languages, irrespective of their typological,
etymological, and phylogenetic relations to other languages. In particular,
readily available data and models of resource-rich sibling languages are often
ignored. In this work, we empirically show, in a case study for Faroese -- a
low-resource language from a high-resource language family -- that by
leveraging the phylogenetic information and departing from the
'one-size-fits-all' paradigm, one can improve cross-lingual transfer to
low-resource languages. In particular, we leverage abundant resources of other
Scandinavian languages (i.e., Danish, Norwegian, Swedish, and Icelandic) for
the benefit of Faroese. Our evaluation results show that we can substantially
improve the transfer performance to Faroese by exploiting data and models of
closely-related high-resource languages. Further, we release a new web corpus
of Faroese and Faroese datasets for named entity recognition (NER), semantic
text similarity (STS), and new language models trained on all Scandinavian
languages
Zero-shot Neural Transfer for Cross-lingual Entity Linking
Cross-lingual entity linking maps an entity mention in a source language to
its corresponding entry in a structured knowledge base that is in a different
(target) language. While previous work relies heavily on bilingual lexical
resources to bridge the gap between the source and the target languages, these
resources are scarce or unavailable for many low-resource languages. To address
this problem, we investigate zero-shot cross-lingual entity linking, in which
we assume no bilingual lexical resources are available in the source
low-resource language. Specifically, we propose pivot-based entity linking,
which leverages information from a high-resource "pivot" language to train
character-level neural entity linking models that are transferred to the source
low-resource language in a zero-shot manner. With experiments on 9 low-resource
languages and transfer through a total of 54 languages, we show that our
proposed pivot-based framework improves entity linking accuracy 17% (absolute)
on average over the baseline systems, for the zero-shot scenario. Further, we
also investigate the use of language-universal phonological representations
which improves average accuracy (absolute) by 36% when transferring between
languages that use different scripts.Comment: To appear in AAAI 201
Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT
Pretrained contextual representation models (Peters et al., 2018; Devlin et
al., 2018) have pushed forward the state-of-the-art on many NLP tasks. A new
release of BERT (Devlin, 2018) includes a model simultaneously pretrained on
104 languages with impressive performance for zero-shot cross-lingual transfer
on a natural language inference task. This paper explores the broader
cross-lingual potential of mBERT (multilingual) as a zero shot language
transfer model on 5 NLP tasks covering a total of 39 languages from various
language families: NLI, document classification, NER, POS tagging, and
dependency parsing. We compare mBERT with the best-published methods for
zero-shot cross-lingual transfer and find mBERT competitive on each task.
Additionally, we investigate the most effective strategy for utilizing mBERT in
this manner, determine to what extent mBERT generalizes away from language
specific features, and measure factors that influence cross-lingual transfer.Comment: EMNLP 2019 Camera Read
- …