124 research outputs found
Unlearn What You Want to Forget: Efficient Unlearning for LLMs
Large language models (LLMs) have achieved significant progress from
pre-training on and memorizing a wide range of textual data, however, this
process might suffer from privacy issues and violations of data protection
regulations. As a result, the ability to easily remove data related to
individual users from such models while not deteriorating their predictive
quality after the removal becomes increasingly important. To address these
issues, in this work, we propose an efficient unlearning framework that could
efficiently update LLMs without having to retrain the whole model after data
removals, by introducing lightweight unlearning layers learned with a selective
teacher-student objective into the transformers. In addition, we introduce a
fusion mechanism to effectively combine different unlearning layers that learns
to forget different sets of data to handle a sequence of forgetting operations.
Experiments on classification and generation tasks demonstrate the
effectiveness of our proposed methods compared to the state-of-the-art
baselines.Comment: EMNLP 202
Tuiteamos o pongamos un tuit? Investigating the Social Constraints of Loanword Integration in Spanish Social Media
Speakers of non-English languages often adopt loanwords from English to express new or unusual concepts. While these loanwords may be borrowed unchanged, speakers may also integrate the words to fit the constraints of their native language, e.g. creating Spanish tuitear from English tweet. Linguists have often considered the process of loanword integration to be more dependent on language-internal constraints, but sociolinguistic constraints such as speaker background remain only qualitatively understood. We investigate the role of social context and speaker background in Spanish speakers\u27 use of integrated loanwords on social media. We find first that newspaper authors use the integrated forms of loanwords and native words more often than social media authors, showing that integration is associated with formal domains. In social media, we find that speaker background and expectations of formality explain loanword and native word integration, such that authors who use more Spanish and who write to a wider audience tend to use integrated verb forms more often. This study shows that loanword integration reflects not only language-internal constraints but also social expectations that vary by conversation and speaker
CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations
Recent work has aimed to capture nuances of human behavior by using LLMs to
simulate responses from particular demographics in settings like social science
experiments and public opinion surveys. However, there are currently no
established ways to discuss or evaluate the quality of such LLM simulations.
Moreover, there is growing concern that these LLM simulations are flattened
caricatures of the personas that they aim to simulate, failing to capture the
multidimensionality of people and perpetuating stereotypes. To bridge these
gaps, we present CoMPosT, a framework to characterize LLM simulations using
four dimensions: Context, Model, Persona, and Topic. We use this framework to
measure open-ended LLM simulations' susceptibility to caricature, defined via
two criteria: individuation and exaggeration. We evaluate the level of
caricature in scenarios from existing work on LLM simulations. We find that for
GPT-4, simulations of certain demographics (political and marginalized groups)
and topics (general, uncontroversial) are highly susceptible to caricature.Comment: To appear at EMNLP 2023 (Main
Impressions: Understanding Visual Semiotics and Aesthetic Impact
Is aesthetic impact different from beauty? Is visual salience a reflection of
its capacity for effective communication? We present Impressions, a novel
dataset through which to investigate the semiotics of images, and how specific
visual features and design choices can elicit specific emotions, thoughts and
beliefs. We posit that the impactfulness of an image extends beyond formal
definitions of aesthetics, to its success as a communicative act, where style
contributes as much to meaning formation as the subject matter. However, prior
image captioning datasets are not designed to empower state-of-the-art
architectures to model potential human impressions or interpretations of
images. To fill this gap, we design an annotation task heavily inspired by
image analysis techniques in the Visual Arts to collect 1,440 image-caption
pairs and 4,320 unique annotations exploring impact, pragmatic image
description, impressions, and aesthetic design choices. We show that existing
multimodal image captioning and conditional generation models struggle to
simulate plausible human responses to images. However, this dataset
significantly improves their ability to model impressions and aesthetic
evaluations of images through fine-tuning and few-shot adaptation.Comment: To be published in EMNLP 202
TADA: Task-Agnostic Dialect Adapters for English
Large Language Models, the dominant starting point for Natural Language
Processing (NLP) applications, fail at a higher rate for speakers of English
dialects other than Standard American English (SAE). Prior work addresses this
using task-specific data or synthetic data augmentation, both of which require
intervention for each dialect and task pair. This poses a scalability issue
that prevents the broad adoption of robust dialectal English NLP. We introduce
a simple yet effective method for task-agnostic dialect adaptation by aligning
non-SAE dialects using adapters and composing them with task-specific adapters
from SAE. Task-Agnostic Dialect Adapters (TADA) improve dialectal robustness on
4 dialectal variants of the GLUE benchmark without task-specific supervision.Comment: 5 Pages; ACL Findings Paper 202
DADA: Dialect Adaptation via Dynamic Aggregation of Linguistic Rules
Existing large language models (LLMs) that mainly focus on Standard American
English (SAE) often lead to significantly worse performance when being applied
to other English dialects. While existing mitigations tackle discrepancies for
individual target dialects, they assume access to high-accuracy dialect
identification systems. The boundaries between dialects are inherently
flexible, making it difficult to categorize language into discrete predefined
categories. In this paper, we propose DADA (Dialect Adaptation via Dynamic
Aggregation), a modular approach to imbue SAE-trained models with
multi-dialectal robustness by composing adapters which handle specific
linguistic features. The compositional architecture of DADA allows for both
targeted adaptation to specific dialect variants and simultaneous adaptation to
various dialects. We show that DADA is effective for both single task and
instruction finetuned language models, offering an extensible and interpretable
framework for adapting existing LLMs to different English dialects
Characterizing Collective Attention via Descriptor Context: A Case Study of Public Discussions of Crisis Events
Social media datasets make it possible to rapidly quantify collective
attention to emerging topics and breaking news, such as crisis events.
Collective attention is typically measured by aggregate counts, such as the
number of posts that mention a name or hashtag. But according to rationalist
models of natural language communication, the collective salience of each
entity will be expressed not only in how often it is mentioned, but in the form
that those mentions take. This is because natural language communication is
premised on (and customized to) the expectations that speakers and writers have
about how their messages will be interpreted by the intended audience. We test
this idea by conducting a large-scale analysis of public online discussions of
breaking news events on Facebook and Twitter, focusing on five recent crisis
events. We examine how people refer to locations, focusing specifically on
contextual descriptors, such as "San Juan" versus "San Juan, Puerto Rico."
Rationalist accounts of natural language communication predict that such
descriptors will be unnecessary (and therefore omitted) when the named entity
is expected to have high prior salience to the reader. We find that the use of
contextual descriptors is indeed associated with proxies for social and
informational expectations, including macro-level factors like the location's
global salience and micro-level factors like audience engagement. We also find
a consistent decrease in descriptor context use over the lifespan of each
crisis event. These findings provide evidence about how social media users
communicate with their audiences, and point towards more fine-grained models of
collective attention that may help researchers and crisis response
organizations to better understand public perception of unfolding crisis
events.Comment: ICWSM 202
- …