9 research outputs found
A Trip Towards Fairness: Bias and De-Biasing in Large Language Models
An outbreak in the popularity of transformer-based Language Models (such as
GPT (Brown et al., 2020) and PaLM (Chowdhery et al., 2022)) has opened the
doors to new Machine Learning applications. In particular, in Natural Language
Processing and how pre-training from large text, corpora is essential in
achieving remarkable results in downstream tasks. However, these Language
Models seem to have inherent biases toward certain demographics reflected in
their training data. While research has attempted to mitigate this problem,
existing methods either fail to remove bias altogether, degrade performance, or
are expensive. This paper examines the bias produced by promising Language
Models when varying parameters and pre-training data. Finally, we propose a
de-biasing technique that produces robust de-bias models that maintain
performance on downstream tasks
PreCog: Exploring the Relation between Memorization and Performance in Pre-trained Language Models
Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languages
The Dark Side of the Language: Pre-trained Transformers in the DarkNet
Pre-trained Transformers are challenging human performances in many natural
language processing tasks. The gigantic datasets used for pre-training seem to
be the key for their success on existing tasks. In this paper, we explore how a
range of pre-trained natural language understanding models perform on truly
novel and unexplored data, provided by classification tasks over a DarkNet
corpus. Surprisingly, results show that syntactic and lexical neural networks
largely outperform pre-trained Transformers. This seems to suggest that
pre-trained Transformers have serious difficulties in adapting to radically
novel texts