Women Are Beautiful, Men Are Leaders: Gender Stereotypes in Machine Translation and Language Modeling

We present GEST -- a new dataset for measuring gender-stereotypical reasoning in masked LMs and English-to-X machine translation systems. GEST contains samples that are compatible with 9 Slavic languages and English for 16 gender stereotypes about men and women (e.g., Women are beautiful, Men are leaders). The definition of said stereotypes was informed by gender experts. We used GEST to evaluate 11 masked LMs and 4 machine translation systems. We discovered significant and consistent amounts of stereotypical reasoning in almost all the evaluated models and languages

Disinformation Capabilities of Large Language Models

Automated disinformation generation is often listed as one of the risks of large language models (LLMs). The theoretical ability to flood the information space with disinformation content might have dramatic consequences for democratic societies around the world. This paper presents a comprehensive study of the disinformation capabilities of the current generation of LLMs to generate false news articles in English language. In our study, we evaluated the capabilities of 10 LLMs using 20 disinformation narratives. We evaluated several aspects of the LLMs: how well they are at generating news articles, how strongly they tend to agree or disagree with the disinformation narratives, how often they generate safety warnings, etc. We also evaluated the abilities of detection models to detect these articles as LLM-generated. We conclude that LLMs are able to generate convincing news articles that agree with dangerous disinformation narratives

Cross-Lingual Learning With Distributed Representations

Cross-lingual Learning can help to bring state-of-the-art deep learning solutions to smaller languages. These languages in general lack resource for training advanced neural networks. With transfer of knowledge across languages we can improve the results for various NLP tasks

Učenie s prenosom medzi prirodzenými jazykmi

Hlboké učenie sa aktuálne javí ako veľmi perspektívny prístup k mnohým úlohám spracovania prirodzeného jazyka. Tento úspech sa však za-tiaľ prejavuje najmä pri jazykoch, ktoré majú dostatočné množstvo zdrojov na natrénovanie komplexných neurónových sietí. Menšie jazyky s menším množ-stvom zdrojov majú problém tieto nové techniky využiť a priepasť medzi nimi a “bohatými“ jazykmi sa tak prehlbuje. V našej práci sa venujeme tomu, ako túto priepasť zmenšiť pomocou prenosu naučenej informácie z jazyka do jazyka. Hlavnou myšlienkou je trénovanie hlbokých neurónových sietí v multilingvál-nom režime tak, aby sa model naučil využívať znalosti z jedného jazyka aj pre vstupy z iných. Navrhli a vykonali sme experiment s prenosom informácie o sentimente slov pomocou zdieľaného priestoru distribučných vektorov. V ex-perimente sme dosiahli výsledky porovnateľné s nemeckými manuálne vytvo-renými lexikónmi sentimentu, pričom sme však nepoužili žiadne nemecké dáta týkajúce sa sentimentu

Report on the Current State of Societal Biases in Slovak AI

<p>This report serves as a concluding report of our project <i>Societal Biases in Slovak AI r</i>unning from November 2022 to October 2023. <strong>The goal of our project was to understand how gender biases in particular affect Slovak AI systems</strong>, but also to increase the awareness of this issue in both the expert community and the general public, as this is an issue that affects virtually everyone using modern communication technologies. Our aim was to approach this issue from an interdisciplinary point of view. The team included people with technical expertise in AI and natural language processing; but also AI ethics experts, social scientists, translators, and gender experts.</p><p>The main contribution of our project is the evaluation of multiple types of AI systems, namely English-to-Slovak machine translation systems, Slovak masked language models, and Slovak speech-to-text systems. We have proposed and implemented an evaluation methodology and observed whether these systems exhibit biased behavior with respect to gender. Each experiment was aimed at a specific type of biased behavior, such as male as norm behavior, stereotypical thinking, or non-equitable performance. <strong>Worryingly, we were able to discover some sort of problematic behavior in all the models taken into consideration</strong></p><h3>Contents of the report</h3><ul><li><strong>Chapter 1</strong> introduces the topic, our project and the role of this report.</li><li><strong>Chapter 2 </strong>provides an executive summary of this report and presents the main findings and conclusions of the project. It can be read to ascertain what the main outcomes of our project are at glance.</li><li><strong>Chapter 3</strong> introduces the issue of fairness in AI and provides a brief theoretical background for our research. It introduces and discusses terms such as fairness, bias, equality, algorithmic harms and others.</li><li><strong>Chapter 4 </strong>is an invited chapter written by gender experts Mariana Szapuová and Janka Kottulová. The chapter discusses what gender stereotypes are, how they impact our society and language.</li><li><strong>Chapter 5</strong> describes the experimental research we conducted. It documents how we evaluated the presence of gender biases in various Slovak AI systems, introduces the results, and discusses the implications of what we have found.</li><li><strong>Chapter 6 </strong>documents a data ethics assessment performed during the project to ensure that our research on sensitive issues is as ethical as possible, taking into consideration the needs and points of view of all the stakeholders.</li></ul><p><i>The project is co-funded by the U.S. Embassy Bratislava</i></p><p> </p&gt

Multilingual Previously Fact-Checked Claim Retrieval

Fact-checkers are often hampered by the sheer amount of online content that needs to be fact-checked. NLP can help them by retrieving already existing fact-checks relevant to the content being investigated. This paper introduces a new multilingual dataset -- MultiClaim -- for previously fact-checked claim retrieval. We collected 28k posts in 27 languages from social media, 206k fact-checks in 39 languages written by professional fact-checkers, as well as 31k connections between these two groups. This is the most extensive and the most linguistically diverse dataset of this kind to date. We evaluated how different unsupervised methods fare on this dataset and its various dimensions. We show that evaluating such a diverse dataset has its complexities and proper care needs to be taken before interpreting the results. We also evaluated a supervised fine-tuning approach, improving upon the unsupervised method significantly

Hate Speech Operationalization: A Preliminary Examination of Hate Speech Indicators and Their Structure

Hate speech should be tackled and prosecuted based on how it is operationalized. However, the existing theoretical definitions of hate speech are not sufficiently fleshed out or easily operable. In order to overcome this inadequacy, and with the help of interdisciplinary experts, we propose an empirical definition of hate speech by providing a list of 10 hate speech indicators and the rationale behind them (the indicators refer to specific, observable, and measurable characteristics that offer a practical definition of hate speech). A preliminary exploratory examination of the structure of hate speech, with the focus on comments related to migrants (one of the most reported grounds of hate speech), revealed that two indicators in particular, denial of human rights and promoting violent behavior, occupy a central role in the network of indicators. Furthermore, we discuss the practical implications of the proposed hate speech indicators – especially (semi-)automatic detection using the latest natural language processing (NLP) and machine learning (ML) methods. Having a set of quantifiable indicators could benefit researchers, human right activists, educators, analysts, and regulators by providing them with a pragmatic approach to hate speech assessment and detection

MultiClaim: Multilingual Previously Fact-Checked Claim Retrieval

MultiClaim: Multilingual Previously Fact-Checked Claim Retrieval - is a dataset that can be used to train a test models used for disinformation combatting. The dataset consists of 206k claims fact-checked by professional fact-checkers and 28k social media posts gathered from the wild. Each social media post has at least on claim assigned. The main idea is to develop information retrieval models that will assign appropriate claims to all the posts. GitHub repository: https://github.com/kinit-sk/multiclaim Contents fact_check_post_mapping.csv - Mapping between fact checks and social media posts: fact_check_id post_id fact_checks.csv - Data about fact-checks: fact_check_id claim - This is the translated text (see below) of the fact-check claim instances - Instances of the fact-check – a list of timestamps and URLs. title - This is the translated text (see below) of the fact-check title posts.csv - Data about social media posts: post_id instances - Instances of the fact-check – a list of timestamps and what were the social media platforms. ocr - This is a list of translated texts (see below) of the OCR transcripts based on the images attached to the post. verdicts - This is a list of verdicts attached by Meta (e.g., False information) text - This is the translated text (see below) of the text written by the user. What is a translated text? A tuple of text, its translation to English and detected languages, e.g., in the sample below we have an original Croatian text, its translation to English and finally the predicted language composition (hbs = Serbo-Croatian): ( '"...bolnice su pune ? ti ina, muk...upravo sada, bolnica Rebro..tragi no sme no', '"...hospitals are full? silence, silence... right now, Rebro hospital... tragically funny', [('hbs', 1.0)] ) More details TB

MULTITuDE

<p>MULTITuDE is a dataset for multilingual machine-generated text detection benchmark, described in the <a href="https://arxiv.org/abs/2310.13606">EMNLP 2023 conference paper</a>. It consists of 7992 human-written news texts in 11 languages subsampled from <a href="https://github.com/danielvarab/massive-summ">MassiveSumm</a>, accompanied by 66089 texts generated by 8 large language models (by using headlines of news articles). The creation process and scripts for replication/extension are located in a <a href="https://github.com/kinit-sk/mgt-detection-benchmark">GitHub repository</a>.</p><p>If you use this dataset in any publication, project, tool or in any other form, please, cite the <a href="https://arxiv.org/abs/2310.13606">paper</a>.</p><h2>Fields</h2><p>The dataset has the following fields:</p><ul><li>'text' - a text sample,</li><li>'label' - 0 for human-written text, 1 for machine-generated text,</li><li>'multi_label' - a string representing a large language model that generated the text or the string "human" representing a human-written text,</li><li>'split' - a string identifying train or test split of the dataset for the purpose of training and evaluation respectively,</li><li>'language' - the ISO 639-1 language code identifying the language of the given text,</li><li>'length' - word count of the given text,</li><li>'source' - a string identifying the source dataset / news medium of the given text.</li></ul><h2>Statistics (the number of samples)</h2><p><strong>Splits:</strong></p><ul><li>train - 44786</li><li>test - 29295</li></ul><p><strong>Binary labels:</strong></p><ul><li>0 - 7992</li><li>1 - 66089</li></ul><p><strong>Multiclass labels:</strong></p><ul><li>gpt-3.5-turbo -       8300</li><li>gpt-4 -                    8300</li><li>text-davinci-003 -   8297</li><li>alpaca-lora-30b -   8290</li><li>vicuna-13b -          8287</li><li>opt-66b -                8229</li><li>llama-65b -            8229</li><li>opt-iml-max-1.3b - 8157</li><li>human -                 7992</li></ul><p><strong>Languages:</strong></p><ul><li>English (en) - 29460 (train + test)</li><li>Spanish (es) - 11586 (train + test)</li><li>Russian (ru) - 11578 (train + test)</li><li>Dutch (nl) - 2695 (test)</li><li>Catalan (ca) - 2691 (test)</li><li>Czech (cs) - 2689 (test)</li><li>German (de) - 2685 (test)</li><li>Chinese (zh) - 2683 (test)</li><li>Portuguese (pt) - 2673 (test)</li><li>Arabic (ar) - 2673 (test)</li><li>Ukrainian (uk) - 2668 (test)</li></ul&gt