7 research outputs found
From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding
The lack of publicly available evaluation data for low-resource languages limits progress in Spoken Language Understanding (SLU). As key tasks like intent classification and slot filling require abundant training data, it is desirable to reuse existing data in high-resource languages to develop models for low-resource scenarios. We introduce xSID, a new benchmark for cross-lingual (x) Slot and Intent Detection in 13 languages from 6 language families, including a very low-resource dialect. To tackle the challenge, we propose a joint learning approach, with English SLU training data and non-English auxiliary tasks from raw text, syntax and translation for transfer. We study two setups which differ by type and language coverage of the pre-trained embeddings. Our results show that jointly learning the main tasks with masked language modeling is effective for slots, while machine translation transfer works best for intent classification
Gender Bias in Masked Language Models for Multiple Languages
Masked Language Models (MLMs) pre-trained by predicting masked tokens on
large corpora have been used successfully in natural language processing tasks
for a variety of languages. Unfortunately, it was reported that MLMs also learn
discriminative biases regarding attributes such as gender and race. Because
most studies have focused on MLMs in English, the bias of MLMs in other
languages has rarely been investigated. Manual annotation of evaluation data
for languages other than English has been challenging due to the cost and
difficulty in recruiting annotators. Moreover, the existing bias evaluation
methods require the stereotypical sentence pairs consisting of the same context
with attribute words (e.g. He/She is a nurse). We propose Multilingual Bias
Evaluation (MBE) score, to evaluate bias in various languages using only
English attribute word lists and parallel corpora between the target language
and English without requiring manually annotated data. We evaluated MLMs in
eight languages using the MBE and confirmed that gender-related biases are
encoded in MLMs for all those languages. We manually created datasets for
gender bias in Japanese and Russian to evaluate the validity of the MBE. The
results show that the bias scores reported by the MBE significantly correlates
with that computed from the above manually created datasets and the existing
English datasets for gender bias.Comment: NAACL 202