23 research outputs found
Analysing Lexical Semantic Change with Contextualised Word Representations
This paper presents the first unsupervised approach to lexical semantic
change that makes use of contextualised word representations. We propose a
novel method that exploits the BERT neural language model to obtain
representations of word usages, clusters these representations into usage
types, and measures change along time with three proposed metrics. We create a
new evaluation dataset and show that the model representations and the detected
semantic shifts are positively correlated with human judgements. Our extensive
qualitative analysis demonstrates that our method captures a variety of
synchronic and diachronic linguistic phenomena. We expect our work to inspire
further research in this direction.Comment: To appear in Proceedings of the 58th Annual Meeting of the
Association for Computational Linguistics (ACL-2020
UiO-UvA at SemEval-2020 Task 1: Contextualised Embeddings for Lexical Semantic Change Detection
We apply contextualised word embeddings to lexical semantic change detection
in the SemEval-2020 Shared Task 1. This paper focuses on Subtask 2, ranking
words by the degree of their semantic drift over time. We analyse the
performance of two contextualising architectures (BERT and ELMo) and three
change detection algorithms. We find that the most effective algorithms rely on
the cosine similarity between averaged token embeddings and the pairwise
distances between token embeddings. They outperform strong baselines by a large
margin (in the post-evaluation phase, we have the best Subtask 2 submission for
SemEval-2020 Task 1), but interestingly, the choice of a particular algorithm
depends on the distribution of gold scores in the test set.Comment: To appear in Proceedings of the 14th International Workshop on
Semantic Evaluation (SemEval-2020
Do Not Fire the Linguist : Grammatical Profiles Help Language Models Detect Semantic Change
Morphological and syntactic changes in word usage-as captured, e.g., by grammatical profiles-have been shown to be good predictors of a word's meaning change. In this work, we explore whether large pre-trained contextualised language models, a common tool for lexical semantic change detection, are sensitive to such morphosyntactic changes. To this end, we first compare the performance of grammatical profiles against that of a multilingual neural language model (XLM-R) on 10 datasets, covering 7 languages, and then combine the two approaches in ensembles to assess their complementarity. Our results show that ensembling grammatical profiles with XLM-R improves semantic change detection performance for most datasets and languages. This indicates that language models do not fully cover the fine-grained morphological and syntactic signals that are explicitly represented in grammatical profiles. An interesting exception are the test sets where the time spans under analysis are much longer than the time gap between them (for example, century-long spans with a one-year gap between them). Morphosyntactic change is slow so grammatical profiles do not detect in such cases. In contrast, language models, thanks to their access to lexical information, are able to detect fast topical changes.Peer reviewe
Interpretable Word Sense Representations via Definition Generation: The Case of Semantic Change Analysis
We propose using automatically generated natural language definitions of
contextualised word usages as interpretable word and word sense
representations. Given a collection of usage examples for a target word, and
the corresponding data-driven usage clusters (i.e., word senses), a definition
is generated for each usage with a specialised Flan-T5 language model, and the
most prototypical definition in a usage cluster is chosen as the sense label.
We demonstrate how the resulting sense labels can make existing approaches to
semantic change analysis more interpretable, and how they can allow users --
historical linguists, lexicographers, or social scientists -- to explore and
intuitively explain diachronic trajectories of word meaning. Semantic change
analysis is only one of many possible applications of the `definitions as
representations' paradigm. Beyond being human-readable, contextualised
definitions also outperform token or usage sentence embeddings in
word-in-context semantic similarity judgements, making them a new promising
type of lexical representation for NLP.Comment: ACL 202
Refer, Reuse, Reduce: Generating Subsequent References in Visual and Conversational Contexts
Dialogue participants often refer to entities or situations repeatedly within
a conversation, which contributes to its cohesiveness. Subsequent references
exploit the common ground accumulated by the interlocutors and hence have
several interesting properties, namely, they tend to be shorter and reuse
expressions that were effective in previous mentions. In this paper, we tackle
the generation of first and subsequent references in visually grounded
dialogue. We propose a generation model that produces referring utterances
grounded in both the visual and the conversational context. To assess the
referring effectiveness of its output, we also implement a reference resolution
system. Our experiments and analyses show that the model produces better, more
effective referring utterances than a model not grounded in the dialogue
context, and generates subsequent references that exhibit linguistic patterns
akin to humans.Comment: In Proceedings of the 2020 Conference on Empirical Methods in Natural
Language Processing (EMNLP 2020
State-of-the-art generalisation research in NLP: a taxonomy and review
The ability to generalise well is one of the primary desiderata of natural
language processing (NLP). Yet, what `good generalisation' entails and how it
should be evaluated is not well understood, nor are there any common standards
to evaluate it. In this paper, we aim to lay the ground-work to improve both of
these issues. We present a taxonomy for characterising and understanding
generalisation research in NLP, we use that taxonomy to present a comprehensive
map of published generalisation studies, and we make recommendations for which
areas might deserve attention in the future. Our taxonomy is based on an
extensive literature review of generalisation research, and contains five axes
along which studies can differ: their main motivation, the type of
generalisation they aim to solve, the type of data shift they consider, the
source by which this data shift is obtained, and the locus of the shift within
the modelling pipeline. We use our taxonomy to classify over 400 previous
papers that test generalisation, for a total of more than 600 individual
experiments. Considering the results of this review, we present an in-depth
analysis of the current state of generalisation research in NLP, and make
recommendations for the future. Along with this paper, we release a webpage
where the results of our review can be dynamically explored, and which we
intend to up-date as new NLP generalisation studies are published. With this
work, we aim to make steps towards making state-of-the-art generalisation
testing the new status quo in NLP.Comment: 35 pages of content + 53 pages of reference
AltGen: 1.3M Plausible Alternatives From Neural Text Generators
<h2>AltGen: 1.3M Plausible Alternatives From Neural Text Generators</h2><p>The AltGen dataset contains 1.3 million English texts generated by neural language generators conditioned on contexts from three corpora of acceptability judgements and two corpora of reading times. </p><p>For each corpus, each text generator, and each sampling algorithm,100 generations are sampled—for a total of 1,257,300 generations. Details about the language generators and the corpora are presented in a paper published at EMNLP 2023 (in particular, Section 4). Please cite this paper if you use any version of the dataset in your work:</p><blockquote><p>Mario Giulianelli, Sarenne Wallbridge, and Raquel Fernández. 2023. <strong>Information Value: Measuring Utterance Predictability as Distance from Plausible Alternatives</strong>. In <i>Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing</i>. Association for Computational Linguistics.</p></blockquote><p>The files are in jsonl format and include a <i>context_id</i> field, which allows retrieving the relevant entry from the original corpus, and the <i>alternatives</i> field, which contains the language model generations. Please note that the alternatives are not post-processed (see code and footnote 2 in the paper for further details). Filenames are built as follows: <i>DecodingAlgorithm</i>_<i>DecodingParameter</i>-n<i>NumAlternatives</i>-maxlen_<i>MaxGenerationLength</i>-sep_<i>Separator.</i>jsonl.</p>