4 research outputs found
LM-CPPF: Paraphrasing-Guided Data Augmentation for Contrastive Prompt-Based Few-Shot Fine-Tuning
In recent years, there has been significant progress in developing
pre-trained language models for NLP. However, these models often struggle when
fine-tuned on small datasets. To address this issue, researchers have proposed
various adaptation approaches. Prompt-based tuning is arguably the most common
way, especially for larger models. Previous research shows that adding
contrastive learning to prompt-based fine-tuning is effective as it helps the
model generate embeddings that are more distinguishable between classes, and it
can also be more sample-efficient as the model learns from positive and
negative examples simultaneously. One of the most important components of
contrastive learning is data augmentation, but unlike computer vision,
effective data augmentation for NLP is still challenging. This paper proposes
LM-CPPF, Contrastive Paraphrasing-guided Prompt-based Fine-tuning of Language
Models, which leverages prompt-based few-shot paraphrasing using generative
language models, especially large language models such as GPT-3 and OPT-175B,
for data augmentation. Our experiments on multiple text classification
benchmarks show that this augmentation method outperforms other methods, such
as easy data augmentation, back translation, and multiple templates.Comment: 10 pages, 1 figure, 8 tables, 1 algorithm Proceedings of the 61st
Annual Meeting of the Association for Computational Linguistic
Automatic Speech Recognition for Speech Assessment of Persian Preschool Children
Preschool evaluation is crucial because it gives teachers and parents
influential knowledge about children's growth and development. The COVID-19
pandemic has highlighted the necessity of online assessment for preschool
children. One of the areas that should be tested is their ability to speak.
Employing an Automatic Speech Recognition(ASR) system is useless since they are
pre-trained on voices that are different from children's voices in terms of
frequency and amplitude. We constructed an ASR for our cognitive test system to
solve this issue using the Wav2Vec 2.0 model with a new pre-training objective
called Random Frequency Pitch(RFP). In addition, we used our new dataset to
fine-tune our model for Meaningless Words(MW) and Rapid Automatic Naming(RAN)
tests. Our new approach reaches a Word Error Rate(WER) of 6.45 on the Persian
section of the CommonVoice dataset. Furthermore, our novel methodology produces
positive outcomes in zero- and few-shot scenarios.Comment: 8 pages, 5 figures, 4 tables, 1 algorith
PEACH: Pre-Training Sequence-to-Sequence Multilingual Models for Translation with Semi-Supervised Pseudo-Parallel Document Generation
Multilingual pre-training significantly improves many multilingual NLP tasks,
including machine translation. Most existing methods are based on some variants
of masked language modeling and text-denoising objectives on monolingual data.
Multilingual pre-training on monolingual data ignores the availability of
parallel data in many language pairs. Also, some other works integrate the
available human-generated parallel translation data in their pre-training. This
kind of parallel data is definitely helpful, but it is limited even in
high-resource language pairs. This paper introduces a novel semi-supervised
method, SPDG, that generates high-quality pseudo-parallel data for multilingual
pre-training. First, a denoising model is pre-trained on monolingual data to
reorder, add, remove, and substitute words, enhancing the pre-training
documents' quality. Then, we generate different pseudo-translations for each
pre-training document using dictionaries for word-by-word translation and
applying the pre-trained denoising model. The resulting pseudo-parallel data is
then used to pre-train our multilingual sequence-to-sequence model, PEACH. Our
experiments show that PEACH outperforms existing approaches used in training
mT5 and mBART on various translation tasks, including supervised, zero- and
few-shot scenarios. Moreover, PEACH's ability to transfer knowledge between
similar languages makes it particularly useful for low-resource languages. Our
results demonstrate that with high-quality dictionaries for generating accurate
pseudo-parallel, PEACH can be valuable for low-resource languages.Comment: 15 pages, 5 figures, 16 tables, 1 algorithm, LoResMT@EACL 202
UTNLP at SemEval-2022 Task 6: A Comparative Analysis of Sarcasm Detection Using Generative-based and Mutation-based Data Augmentation
Sarcasm is a term that refers to the use of words to mock, irritate, or amuse
someone. It is commonly used on social media. The metaphorical and creative
nature of sarcasm presents a significant difficulty for sentiment analysis
systems based on affective computing. The methodology and results of our team,
UTNLP, in the SemEval-2022 shared task 6 on sarcasm detection are presented in
this paper. We put different models, and data augmentation approaches to the
test and report on which one works best. The tests begin with traditional
machine learning models and progress to transformer-based and attention-based
models. We employed data augmentation based on data mutation and data
generation. Using RoBERTa and mutation-based data augmentation, our best
approach achieved an F1-sarcastic of 0.38 in the competition's evaluation
phase. After the competition, we fixed our model's flaws and achieved an
F1-sarcastic of 0.414.Comment: 6 pages, 2 figures, International Workshop on Semantic Evaluation
co-located with NAAC