54 research outputs found
FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding
Large-scale cross-lingual language models (LM), such as mBERT, Unicoder and
XLM, have achieved great success in cross-lingual representation learning.
However, when applied to zero-shot cross-lingual transfer tasks, most existing
methods use only single-language input for LM finetuning, without leveraging
the intrinsic cross-lingual alignment between different languages that proves
essential for multilingual tasks. In this paper, we propose FILTER, an enhanced
fusion method that takes cross-lingual data as input for XLM finetuning.
Specifically, FILTER first encodes text input in the source language and its
translation in the target language independently in the shallow layers, then
performs cross-language fusion to extract multilingual knowledge in the
intermediate layers, and finally performs further language-specific encoding.
During inference, the model makes predictions based on the text input in the
target language and its translation in the source language. For simple tasks
such as classification, translated text in the target language shares the same
label as the source language. However, this shared label becomes less accurate
or even unavailable for more complex tasks such as question answering, NER and
POS tagging. To tackle this issue, we further propose an additional
KL-divergence self-teaching loss for model training, based on auto-generated
soft pseudo-labels for translated text in the target language. Extensive
experiments demonstrate that FILTER achieves new state of the art on two
challenging multilingual multi-task benchmarks, XTREME and XGLUE.Comment: Accepted to AAAI 2021; Top-1 Performance on XTREME
(https://sites.research.google/xtreme, September 8, 2020) and XGLUE
(https://microsoft.github.io/XGLUE, September 14, 2020) benchmar
Zero-shot Neural Transfer for Cross-lingual Entity Linking
Cross-lingual entity linking maps an entity mention in a source language to
its corresponding entry in a structured knowledge base that is in a different
(target) language. While previous work relies heavily on bilingual lexical
resources to bridge the gap between the source and the target languages, these
resources are scarce or unavailable for many low-resource languages. To address
this problem, we investigate zero-shot cross-lingual entity linking, in which
we assume no bilingual lexical resources are available in the source
low-resource language. Specifically, we propose pivot-based entity linking,
which leverages information from a high-resource "pivot" language to train
character-level neural entity linking models that are transferred to the source
low-resource language in a zero-shot manner. With experiments on 9 low-resource
languages and transfer through a total of 54 languages, we show that our
proposed pivot-based framework improves entity linking accuracy 17% (absolute)
on average over the baseline systems, for the zero-shot scenario. Further, we
also investigate the use of language-universal phonological representations
which improves average accuracy (absolute) by 36% when transferring between
languages that use different scripts.Comment: To appear in AAAI 201
Neural Cross-Lingual Entity Linking
A major challenge in Entity Linking (EL) is making effective use of
contextual information to disambiguate mentions to Wikipedia that might refer
to different entities in different contexts. The problem exacerbates with
cross-lingual EL which involves linking mentions written in non-English
documents to entries in the English Wikipedia: to compare textual clues across
languages we need to compute similarity between textual fragments across
languages. In this paper, we propose a neural EL model that trains fine-grained
similarities and dissimilarities between the query and candidate document from
multiple perspectives, combined with convolution and tensor networks. Further,
we show that this English-trained system can be applied, in zero-shot learning,
to other languages by making surprisingly effective use of multi-lingual
embeddings. The proposed system has strong empirical evidence yielding
state-of-the-art results in English as well as cross-lingual: Spanish and
Chinese TAC 2015 datasets.Comment: Association for the Advancement of Artificial Intelligence (AAAI),
201
What Matters for Neural Cross-Lingual Named Entity Recognition: An Empirical Analysis
Building named entity recognition (NER) models for languages that do not have
much training data is a challenging task. While recent work has shown promising
results on cross-lingual transfer from high-resource languages to low-resource
languages, it is unclear what knowledge is transferred. In this paper, we first
propose a simple and efficient neural architecture for cross-lingual NER.
Experiments show that our model achieves competitive performance with the
state-of-the-art. We further analyze how transfer learning works for
cross-lingual NER on two transferable factors: sequential order and
multilingual embeddings, and investigate how model performance varies across
entity lengths. Finally, we conduct a case-study on a non-Latin language,
Bengali, which suggests that leveraging knowledge from Wikipedia will be a
promising direction to further improve the model performances. Our results can
shed light on future research for improving cross-lingual NER.Comment: 7 page
XL-NBT: A Cross-lingual Neural Belief Tracking Framework
Task-oriented dialog systems are becoming pervasive, and many companies
heavily rely on them to complement human agents for customer service in call
centers. With globalization, the need for providing cross-lingual customer
support becomes more urgent than ever. However, cross-lingual support poses
great challenges---it requires a large amount of additional annotated data from
native speakers. In order to bypass the expensive human annotation and achieve
the first step towards the ultimate goal of building a universal dialog system,
we set out to build a cross-lingual state tracking framework. Specifically, we
assume that there exists a source language with dialog belief tracking
annotations while the target languages have no annotated dialog data of any
form. Then, we pre-train a state tracker for the source language as a teacher,
which is able to exploit easy-to-access parallel data. We then distill and
transfer its own knowledge to the student state tracker in target languages. We
specifically discuss two types of common parallel resources: bilingual corpus
and bilingual dictionary, and design different transfer learning strategies
accordingly. Experimentally, we successfully use English state tracker as the
teacher to transfer its knowledge to both Italian and German trackers and
achieve promising results.Comment: 13 pages, 5 figures, 3 tables, accepted to EMNLP 2018 conferenc
- …