24 research outputs found
Octopus: A Multitask Model and Toolkit for Arabic Natural Language Generation
Understanding Arabic text and generating human-like responses is a
challenging endeavor. While many researchers have proposed models and solutions
for individual problems, there is an acute shortage of a comprehensive Arabic
natural language generation toolkit that is capable of handling a wide range of
tasks. In this work, we present a novel Arabic text-to-text Transformer model,
namely AraT5v2. Our new model is methodically trained on extensive and diverse
data, utilizing an extended sequence length of 2,048 tokens. We explore various
pretraining strategies including unsupervised, supervised, and joint
pertaining, under both single and multitask settings. Our models outperform
competitive baselines with large margins. We take our work one step further by
developing and publicly releasing Octopus, a Python-based package and
command-line toolkit tailored for eight Arabic generation tasks all exploiting
a single model. We release the models and the toolkit on our public repository
ORCA: A Challenging Benchmark for Arabic Language Understanding
Due to their crucial role in all NLP, several benchmarks have been proposed
to evaluate pretrained language models. In spite of these efforts, no public
benchmark of diverse nature currently exists for evaluation of Arabic. This
makes it challenging to measure progress for both Arabic and multilingual
language models. This challenge is compounded by the fact that any benchmark
targeting Arabic needs to take into account the fact that Arabic is not a
single language but rather a collection of languages and varieties. In this
work, we introduce ORCA, a publicly available benchmark for Arabic language
understanding evaluation. ORCA is carefully constructed to cover diverse Arabic
varieties and a wide range of challenging Arabic understanding tasks exploiting
60 different datasets across seven NLU task clusters. To measure current
progress in Arabic NLU, we use ORCA to offer a comprehensive comparison between
18 multilingual and Arabic language models. We also provide a public
leaderboard with a unified single-number evaluation metric (ORCA score) to
facilitate future research.Comment: All authors contributed equally. Accepted at ACL 2023, Toronto,
Canad
Dolphin: A Challenging and Diverse Benchmark for Arabic NLG
We present Dolphin, a novel benchmark that addresses the need for an
evaluation framework for the wide collection of Arabic languages and varieties.
The proposed benchmark encompasses a broad range of 13 different NLG tasks,
including text summarization, machine translation, question answering, and
dialogue generation, among others. Dolphin comprises a substantial corpus of 40
diverse and representative public datasets across 50 test splits, carefully
curated to reflect real-world scenarios and the linguistic richness of Arabic.
It sets a new standard for evaluating the performance and generalization
capabilities of Arabic and multilingual models, promising to enable researchers
to push the boundaries of current methodologies. We provide an extensive
analysis of Dolphin, highlighting its diversity and identifying gaps in current
Arabic NLG research. We also evaluate several Arabic and multilingual models on
our benchmark, allowing us to set strong baselines against which researchers
can compare
GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP
The recent emergence of ChatGPT has brought a revolutionary change in the
landscape of NLP. Although ChatGPT has consistently shown impressive
performance on English benchmarks, its exact capabilities on most other
languages remain largely unknown. To better understand ChatGPT's capabilities
on Arabic, we present a large-scale evaluation of the model on a broad range of
Arabic NLP tasks. Namely, we evaluate ChatGPT on 32 diverse natural language
understanding and generation tasks on over 60 different datasets. To the best
of our knowledge, our work offers the first performance analysis of ChatGPT on
Arabic NLP at such a massive scale. Our results show that, despite its success
on English benchmarks, ChatGPT trained in-context (few-shot) is consistently
outperformed by much smaller dedicated models finetuned on Arabic. These
results suggest that there is significant place for improvement for
instruction-tuned LLMs such as ChatGPT.Comment: Work in progres
Zero-Shot Slot and Intent Detection in Low-Resource Languages
Intent detection and slot filling are critical tasks in spoken and natural
language understanding for task-oriented dialog systems. In this work we
describe our participation in the slot and intent detection for low-resource
language varieties (SID4LR; Aepli et al. (2023)). We investigate the slot and
intent detection (SID) tasks using a wide range of models and settings. Given
the recent success of multitask-prompted finetuning of large language models,
we also test the generalization capability of the recent encoder-decoder model
mT0 (Muennighoff et al., 2022) on new tasks (i.e., SID) in languages they have
never intentionally seen. We show that our best model outperforms the baseline
by a large margin (up to +30 F1 points) in both SID tasksComment: VarDial @ EAC