English and Chinese, known as resource-rich languages, have witnessed the
strong development of transformer-based language models for natural language
processing tasks. Although Vietnam has approximately 100M people speaking
Vietnamese, several pre-trained models, e.g., PhoBERT, ViBERT, and vELECTRA,
performed well on general Vietnamese NLP tasks, including POS tagging and named
entity recognition. These pre-trained language models are still limited to
Vietnamese social media tasks. In this paper, we present the first monolingual
pre-trained language model for Vietnamese social media texts, ViSoBERT, which
is pre-trained on a large-scale corpus of high-quality and diverse Vietnamese
social media texts using XLM-R architecture. Moreover, we explored our
pre-trained model on five important natural language downstream tasks on
Vietnamese social media texts: emotion recognition, hate speech detection,
sentiment analysis, spam reviews detection, and hate speech spans detection.
Our experiments demonstrate that ViSoBERT, with far fewer parameters, surpasses
the previous state-of-the-art models on multiple Vietnamese social media tasks.
Our ViSoBERT model is available only for research purposes.Comment: Accepted at EMNLP'2023 Main Conferenc