22 research outputs found
Dictionary-based Phrase-level Prompting of Large Language Models for Machine Translation
Large language models (LLMs) demonstrate remarkable machine translation (MT)
abilities via prompting, even though they were not explicitly trained for this
task. However, even given the incredible quantities of data they are trained
on, LLMs can struggle to translate inputs with rare words, which are common in
low resource or domain transfer scenarios. We show that LLM prompting can
provide an effective solution for rare words as well, by using prior knowledge
from bilingual dictionaries to provide control hints in the prompts. We propose
a novel method, DiPMT, that provides a set of possible translations for a
subset of the input words, thereby enabling fine-grained phrase-level prompted
control of the LLM. Extensive experiments show that DiPMT outperforms the
baseline both in low-resource MT, as well as for out-of-domain MT. We further
provide a qualitative analysis of the benefits and limitations of this
approach, including the overall level of controllability that is achieved
That was the last straw, we need more: Are Translation Systems Sensitive to Disambiguating Context?
The translation of ambiguous text presents a challenge for translation
systems, as it requires using the surrounding context to disambiguate the
intended meaning as much as possible. While prior work has studied ambiguities
that result from different grammatical features of the source and target
language, we study semantic ambiguities that exist in the source (English in
this work) itself. In particular, we focus on idioms that are open to both
literal and figurative interpretations (e.g., goose egg), and collect TIDE, a
dataset of 512 pairs of English sentences containing idioms with disambiguating
context such that one is literal (it laid a goose egg) and another is
figurative (they scored a goose egg, as in a score of zero). In experiments, we
compare MT-specific models and language models for (i) their preference when
given an ambiguous subsentence, (ii) their sensitivity to disambiguating
context, and (iii) the performance disparity between figurative and literal
source sentences. We find that current MT models consistently translate English
idioms literally, even when the context suggests a figurative interpretation.
On the other hand, LMs are far more context-aware, although there remain
disparities across target languages. Our findings underline the potential of
LMs as a strong backbone for context-aware translation.Comment: EMNLP 2023 Finding
Simple, Interpretable and Stable Method for Detecting Words with Usage Change across Corpora
International audienceThe problem of comparing two bodies of text and searching for words that differ in their usage between them arises often in digital humanities and computational social science. This is commonly approached by training word embeddings on each corpus, aligning the vector spaces, and looking for words whose cosine distance in the aligned space is large. However, these methods often require extensive filtering of the vocabulary to perform well, and-as we show in this work-result in unstable, and hence less reliable, results. We propose an alternative approach that does not use vector space alignment, and instead considers the neighbors of each word. The method is simple, interpretable and stable. We demonstrate its effectiveness in 9 different setups, considering different corpus splitting criteria (age, gender and profession of tweet authors, time of tweet) and different languages (English, French and Hebrew)
Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models
Despite their popularity in non-English NLP, multilingual language models
often underperform monolingual ones due to inter-language competition for model
parameters. We propose Cross-lingual Expert Language Models (X-ELM), which
mitigate this competition by independently training language models on subsets
of the multilingual corpus. This process specializes X-ELMs to different
languages while remaining effective as a multilingual ensemble. Our experiments
show that when given the same compute budget, X-ELM outperforms jointly trained
multilingual models across all considered languages and that these gains
transfer to downstream tasks. X-ELM provides additional benefits over
performance improvements: new experts can be iteratively added, adapting X-ELM
to new languages without catastrophic forgetting. Furthermore, training is
asynchronous, reducing the hardware requirements for multilingual training and
democratizing multilingual modeling
BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfer
Despite remarkable advancements in few-shot generalization in natural
language processing, most models are developed and evaluated primarily in
English. To facilitate research on few-shot cross-lingual transfer, we
introduce a new benchmark, called BUFFET, which unifies 15 diverse tasks across
54 languages in a sequence-to-sequence format and provides a fixed set of
few-shot examples and instructions. BUFFET is designed to establish a rigorous
and equitable evaluation framework for few-shot cross-lingual transfer across a
broad range of tasks and languages. Using BUFFET, we perform thorough
evaluations of state-of-the-art multilingual large language models with
different transfer methods, namely in-context learning and fine-tuning. Our
findings reveal significant room for improvement in few-shot in-context
cross-lingual transfer. In particular, ChatGPT with in-context learning often
performs worse than much smaller mT5-base models fine-tuned on English task
data and few-shot in-language examples. Our analysis suggests various avenues
for future research in few-shot cross-lingual transfer, such as improved
pretraining, understanding, and future evaluations.Comment: The data and code is available at https://buffetfs.github.io